Buckner40 and Adni60 Testing

see also: TestPlan, and Buckner40Notes

Introduction

The 'Buckner40' is a name given to a small subset of subjects originating from Washington University, collected by Randy Buckner and colleagues. It consists of a group of 40 nondemented subjects, ranging in age from 18 to 93 (with no subjects between 30 to 65 years of age), 21 women and 19 men. These data are a subset of a larger open-access structural data set that can be obtained from the fMRIDC at: http://www.fmridc.org/f/fmridc, accession number 2-2004-1168X.

For Freesurfer, this dataset is especially nice as a test-set because of the wide subject age range, and the quality of the scans, which, while good, are not great, but are typical of what Freesurfer has to handle.

The Buckner40Notes wiki page documents the work of examining these brains, slice by slice, to verify their correctness, in terms of how well Freesurfer has reconstructed the white and pial surfaces, and segmented the subcortical structures.

It is important that Freesurfer's recon-all processing stream continue to process all of the subjects successfully, where that means the results are as good as or better than prior 'known-good' results (the reference set). (Note, if Freesurfer improves the results, then a new reference set is created, aka 'reset the standard').

Historical tests

The location of Buckner40 reference datasets with respect to previously released versions, and the development version, can be found at the following locations:

/space/freesurfer/subjects/test/buckner40/stable5_1_0
/space/freesurfer/subjects/test/buckner40/stable5_3_0
/space/freesurfer/subjects/test/buckner40/stable6
/space/freesurfer/subjects/test/buckner40/dev

And Adni60:

/space/freesurfer/subjects/test/adni60/stable5_1_0
/space/freesurfer/subjects/test/adni60/stable5_3_0
/space/freesurfer/subjects/test/adni60/stable6
/space/freesurfer/subjects/test/adni60/dev

Running the test

The 2 main scripts for running the Buckner40 and Adni60 tests are located here:

/space/freesurfer/subjects/test/scripts/comparison_test.sh
/space/freesurfer/subjects/test/scripts/comparison_utils.sh

To run the Buckner40 and generate groupstats, ssh to launchpad and run the comparison_test.sh as follows:

$> ssh launchpad
$> /space/freesurfer/subjects/test/scripts/comparison_test.sh \
     -fs /usr/local/freesurfer/dev \
     -subjects_dir /space/freesurfer/subjects/test/buckner40/dev \
     -generate_groupstats /space/freesurfer/subjects/test/buckner40/dev/groupstats_2016_07_30

To run the Adni60 and generate groupstats, call the scrpts the exact same way, just replace "Buckner40" in the path with "Adni60":

$> ssh launchpad
$> /space/freesurfer/subjects/test/scripts/comparison_test.sh \
     -fs /usr/local/freesurfer/dev \
     -subjects_dir /space/freesurfer/subjects/test/adni60/dev \
     -generate_groupstats /space/freesurfer/subjects/test/adni60/dev/groupstats_2016_07_30

Running tests as a cronjob

The Buckner40 and Adni60 datasets should be run on a weekly, monthly basis via cron, and should always be run (manually) as a precursor to a merge of the development branch with the master branch. To run these tests as a cronjob every week on Friday at midnight, the crontab file on launchpad should look as follows:

##########################################
## Run Buckner40 once per week (Friday, 11pm)
##########################################

FREESURFER_DEV=/usr/local/freesurfer/dev
BUCKNER40_DEV=/space/freesurfer/subjects/test/buckner40/dev
BUCKNER40_DEV_GROUPSTATS=$BUCKNER40_DEV/groupstats_$(date +"%Y-%m-%d")

00 23 * * 5 pbsubmit -f -m zkaufman -c "/space/freesurfer/subjects/test/scripts/comparison_test.sh -fs $FREESURFER_DEV -subjects_dir $BUCKNER40_DEV -generate_groupstats $BUCKNER40_DEV_GROUPSTATS"

Prelease procedure

In preparation for a release, a new folder should be created named for the expected freesurfer version number. For example, inpreparation of the release of freesurfer v7 we would create a "stable7" folder:

$> mkdir /space/freesurfer/subjects/test/buckner40/stable7

In that folder should be a group.fsgd, raw_origs.tar.gz, and subjects.txt files. These files already exist in the parent directory (/space/freesurfer/subjects/test/buckner40) and thus can be symlinked at follows:

$> cd /space/freesurfer/subjects/test/buckner40/stable7
$> ln -s ../buckner40.fsgd group.fsgd
$> ln -s ../buckner40_origs.tar.gz raw_origs.tar.gz
$> ln -s ../subjects.txt

The subjects data and groupstats can then be run continually using the freesurfer release candidates until it is finally run WITH THE ACTUAL FREESURFER VERSION TO BE RELEASED. The subject data and groupstats should be preserved for posterity with the exact version of freesurfer that is to be released:

$> ssh launchpad
$> /space/freesurfer/subjects/test/scripts/comparison_test.sh \
     -fs /usr/local/freesurfer/stable7 \
     -subjects_dir /space/freesurfer/subjects/test/buckner40/stable \
     -generate_groupstats /space/freesurfer/subjects/test/buckner40/stable7/groupstats_2016_07_30