THIS PAGE HAS BEEN ARCHIVED TO LABARCHIVES!

Please refer to this site/make edits here for the most updated information: https://mynotebook.labarchives.com/user_edit

Offline Recon Processing without transferring data to the cluster using sshfs

NOTE: these instructions are not necessary if streaming directly to oribi

It is possible to process raw data directly from the RAID without first transferring it to cluster space using sshfs. This program allows one to mount a directory on a host computer through ssh, so its contents are visible and usable to the host but only take up disk space in the remote directory. sshfs requires special permissions, and as of August 2015 can only be used on nike by those given access to a special sudo rule by the Help Desk.

Before you can do the below, the lab manager must email help and request your user account be added to pdc:/etc/SUDOers.ldif for cn: lcn_fuse. After that is approved, the necessary steps are:

ssh nike

2. run the following command. this is a special sudo rule that will only work on nike, and gives access to the fuse group which is necessary for sshfs. **You will have to request access to launchpad first to execute this command, if you do not already have access.

sudo -u your_username -g fuse /bin/tcsh

3. mount the RAID directory containing raw data onto nike using sshfs:

sshfs streaming@suez.partners.org:/Volumes/path/to/data/dir /mnt/offline_recon -o idmap=user

The suez address may need to be adjusted if it isn't on partners.org. user is the word 'user', not your username. Also, when prompted for a password, this is the password for the streaming account on suez, not your NMR or Partners account.

4. link /mnt/offline_recon to the folder where you are processing. For example:

     cd /cluster/exvivo/I40_rh_32chexvivo_150um_sef
     ln -s /mnt/offline_recon raw

NOTE: At this time, use the offline_recon_dev version!!! This can be used be creating the following alias:

alias offline_recon_dev "/autofs/cluster/exvivo/rock/4/users/hires/scripts/offline_recon_dev/setup.csh"

5. Run offline_recon as usual by following instructions on the OfflineRecon page. Make sure to use the nike queue. Also, you MUST be on nike when running offline_recon, or else the link to /mnt/offline_recon won't work. pbsubmit works on nike, so you can submit jobs from there to the nike queue instead of logging on to launchpad.

6. Sometime after the strip_process jobs are finished (so when the .dat files are no longer needed), run the following command to unmount the RAID directory:

 fusermount -u /mnt/offline_recon

NOTE: By using sshfs, you aren't backing up the raw data from the RAID (which normally happens by transferring to the cluster). This factor needs to be taken into account, especially for extra special data.