Backup

Backup on Beamlines

It is very easy to collect several hundreds of Gigabytes worth of data, maybe even 1 Terabyte. It is therefore important that you are careful about backing up your data on these beamlines at the start of your experiment in synchronise mode and that you bring adequate media for this.

Data Directories on NICE

Data are written directly to the NICE directories from mxCuBE (e.g. use the directory to save images as: /data/visitor/mx###/beamline/date/mysubdirectory).

Account and data are deleted 30 days after the end of the experience and data for around six months on tape. Your home directory on NICE has a quota of 100MB: if this is full you may have problems logging in. The beamline data directories listed in the home directories are in fact links to the main storage data disks which have no quotas and where data should be stored from the beamlines.

Data Security

Users concerned about the security of their data can follow the instructions below to protect images from being viewed by other ESRF users:

1. As the beamline generic account (opid14, opid23 or opid29) go to the data directory where your images are stored. Type nice_chown mxxxxx . which will change the ownership of all the files in that directory and below to files to mxxxxx (or ixxxxx) - would be asked make sure it is in your directory!
2. Login to your user account on NICE and go to the same data directory and type chmod -R go-rwx .. This will make all the files only visible to the mxxxxx (or inxxxx) account.

a) Backup PCs in the beamline Control Cabins

Each MX station has backup PCs available in the control cabin. The bolded windows PC in this list are fully dedicated to backups with a Giga link.

• from a Linux machine :

Please note that we recommend you bring FAT32 formatted USB hard disks with no data on them (or data which you are prepared to delete). The supported filesystem at the ESRF is FAT32 (for cross-platform compatibility). If you bring any other format it will need to be formatted at the beamline. This can take a while with larger disks.

You can use the command : rsync -r -t -v source/ destination/

• from a Windows PC :

You can drag & drop folders from the nice directory to your disk or synchronise your backup while you perform your experiment using syncback (available on all dedicated backup windows PCs)

 

a.1) ID29 Pilatus Data Backup

Due the the huge amount of data that can be collected on ID29, the linux machine lid29io is equipped with a 10Gbit connection to NICE and has support for USB3.0 external disks. We recommend to use this machine for main backup. You can run rsync as explained above or use the GUI Grsync which can be started from the Desktop.

b) Bring Your Own Laptop

Please note that our local contacts are NOT trained to deal with the wide variety of laptops, operating systems and keyboards the users bring on-site. We therefore advise you to:
Set up your laptop as early as possible following this web page (link available on-site only). Expert help for Windows PCs only can be provided through Computing Services on working days from 9.00am to 5.00pm! Users with Linux and Mac systems are on your own...

Please contact your Local Contact, who will call 24-24 (telephone hotline operates on normal working days from 9.00am to 11.30am and from 12.30pm to 5.00pm) in case you have problem setting up your laptop.

WiFi is available on the beamlines for general net access (this is a guest network outside the firewall for web browsing, etc). Look here (link available on-site only) for information on how to use it.

To mount a disk under Linux you could try the following once you have connected to the network:

On a beamline machine go to the data collection directory, e.g. "cd /data/visitor/mx123/id14eh4/mysubdirectory/", then type "df ."
You will then see the server name required for mounting the disk, e.g.

gx.esrf.fr:/data/visitor
                     212600881312 133543415168 79057466144  63% /mntdirect/_data_visitor

On your laptop do the following:

mkdir -p /data/visitor/
mount  gx.esrf.fr:/data/visitor /data/visitor/

Please note that if when you type df you see for example

gxge1.esrf.fr:/data/visitor

remove the ge1 part of the path to give the command

mount  gx.esrf.fr:/data/visitor /data/visitor/

The ge1 is part of the private gigabit network on the beamline and is inaccessible from NICE.

UNIX data compression, compress myimages.mccd, will compress image data down to around 70% of the original size if you are limited in disk space.

When connecting your laptop, use one of the IP addresses below corresponding to the beamline you are using. Make sure that the temporary address you choose is not already used by another computer. To do this, open a terminal window and type

ping -w 1 <ipaddress>

(e.g. ping -w 1 160.103.64.50)

If you have 0% packet loss it means the address is already in use. If the packet loss is 100% then it is OK to use this address.

Beamline	Name/IP Available for Users		Gateway out of Beamline Network
	Name	IP
ID14-1	id14tmp0.esrf.fr id14tmp1.esrf.fr	160.103.64.50 160.103.64.51	160.103.64.99
ID14-2	id14tmp2.esrf.fr id14tmp3.esrf.fr	160.103.64.52 160.103.64.53	160.103.64.99
ID14-3	id14tmp4.esrf.fr id14tmp5.esrf.fr	160.103.64.54 160.103.64.55	160.103.64.99
ID14-4	id14tmp6.esrf.fr id14tmp7.esrf.fr	160.103.64.56 160.103.64.57	160.103.64.99
ID23-1	id23tmp0.esrf.fr id23tmp1.esrf.fr id23tmp2.esrf.fr	160.103.43.50 160.103.43.51 160.103.43.52	160.103.43.99
ID23-2	id23tmp3.esrf.fr id23tmp4.esrf.fr id23tmp5.esrf.fr	160.103.43.53 160.103.43.54 160.103.43.55	160.103.43.99
ID29	id29tmp0.esrf.fr id29tmp1.esrf.fr id29tmp2.esrf.fr	160.103.49.50 160.103.49.51 160.103.49.52	160.103.49.99

 

c) sFTP Backup

Instructions for sFTP are only visible from SFTP Procedure.

rsync  -avztu --modify-window=5  -e ssh imagedirectory user@home.lab:/destination