• XgridNotes

Using Freesurfer on Apple Xgrid

Freesurfer jobs can be run on the XGrid. The advantage of this approach is that you can submit hundreds of jobs to run and the controller will automatically allocate each freesurfer instance to an available processor. This will maximize processing power and get the total number of jobs completed in the least amount of time. If a node is restarted in the middle or processing, the progress will be lost and the process with restart on another available node.

Setting up Xgrid

If you have Mac OS X Server 10.5 or higher, then you already have a computer setup for hosting the Xgrid controller. If you only have machines running the "client" (standard) version of Mac OS X, you can get a copy of XgridLite. XgridLite will enable any machine to be the Xgrid controller.

If you only have one computer, you can use a single Mac as both a controller and a client (submit jobs to the same machine they are running on). For each computer, you will need to enable Xgrid Sharing in the System Preferences --> Sharing Panel.

You must have Freesurfer installed (including license file) on all nodes of the grid.

For more background on Xgrid see TenGrid

Organizing Your Data

In order to use these script, you first need to organize your data into folders. Each folder should be named for the participant that you wish to analyze (e.g. Subj001). Inside each folder, place the NIFTI or MGZ file for their T1-weighted high-res anatomical image, also named after the subject (e.g. Subj001.nii). Your folder structure should look something like this:

-Subj001
---Subj001.nii
-Subj002
---Subj002.nii
-Subj003
---Subj003.nii

Generic Script

To begin, copy the script example below and save the file alongside your participant folders as xgrid_freesurfer.sh. Once saved, chmod +x the file to make it executable. Be sure to change the path to your Freesurfer installation (usually /Applications/freesurfer). Also if you use MGZ files replace the extension in the sed statement to include .mgz instead of .nii. You can also modify the recon-all arguments. It may be useful to add -qcache to the recon-all statement to generate cached smoothing levels.

remove the space between the # and the !

# !/bin/bash

export NO_FSFAST=1

#setup Freesurfer
export FREESURFER_HOME=/usr/local/freesurfer
#source $FREESURFER_HOME/SetUpFreeSurfer.sh
. $FREESURFER_HOME/SetUpFreeSurfer.sh

SUBJECTS_DIR=$(pwd)

for FS in $*
do
   subjectID=$(/bin/echo $FS | /usr/bin/sed 's/.nii//')
   recon-all -s $subjectID -i $FS -all
done

Submitting Jobs

At the most basic level, you can submit each folder individually using the following syntax:

xgrid -h <your xgrid controller> -p <your xgrid password> -job submit -in Subj001 xgrid_freesurfer.sh Subj001.nii

The options work as follows:

-h is to specify the hostname (or IP address) of your controller
-p is to specify your password for submitting jobs
-job submit - tells xgrid that you are submitting a job
-in identifies the folder (in this case Subj001) that will be copied
xgrid_freesurfer.sh tells the job to run the script (above) in the folder copied
Subj001.nii is the single argument passed to the xgrid_freesurfer.sh script.  

Each submitted job will supply you with a job number. You can also grab the Apple Server Admin Tools, which include an Xgrid Admin GUI application for monitoring your Xgrid, including computers included in the grid, job numbers, and status of jobs on the grid.

However you can also automate this process using a script, a Ruby script below can do this. I suggest saving this script (xgrid_submit.rb) alongside the xgrid_freesurfer.sh script. Remember to chmod +x the script to make it executable.

remove the space between the # and the !

# !/usr/bin/ruby

puts "Xgrid submission application for Freesurfer";

ARGV.each{|x|
    puts "xgrid -h <your xgrid controller> -p <password> -job submit -in #{x} xgrid_freesurfer.sh #{x}.nii"
    system("xgrid -h <your xgrid controller> -p <password> -job submit -in #{x} xgrid_freesurfer.sh #{x}.nii");
}

Retrieving Results

In order to get the results, you can run the following command for each job on the cluster.

xgrid -h <your xgrid controller> -p <password> -job results -id <job id> -out <output folder>

This process, like submitting a job, can be automated. Below is an example ruby script for automatically retrieving jobs 1 through 20 from the grid.

remove the space between the # and the !

# !/usr/bin/ruby

puts 'Get xgrid results...'

1.upto(20) {|a|
    puts "xgrid -h <your xgrid controller> -p <password> -job results -id #{a} -out .";
        %x[xgrid -h <your xgrid controller> -p <password> -job results -id #{a} -out .];
}

The options here work as follows:

-h is the host of your controller
-p is your Xgrid password
-job results - specifies that you want the results for a job
-id tells Xgrid what the id number of the job you wish to retrieve is
-out . tells Xgrid to save the resulting files to the current directory

The benefit of using these scripts is that Xgrid will save the processed subject folders that can then be transplanted into your SUBJECTS_DIR.

Questions

pmolfese (at) haskins (dot) yale (dot) edu