Buckner40 and Adni60 Testing
see also: TestPlan, and Buckner40Notes
Introduction
The 'Buckner40' is a name given to a small subset of subjects originating from Washington University, collected by Randy Buckner and colleagues. It consists of a group of 40 nondemented subjects, ranging in age from 18 to 93 (with no subjects between 30 to 65 years of age), 21 women and 19 men. These data are a subset of a larger open-access structural data set created by the Buckner lab. That data may (or may not) be available at the time of this writing, but you can try the link https://cnl.rc.fas.harvard.edu/data-tools. Previously, it was referenced by accession number 2-2004-1168X.
For Freesurfer, this dataset is especially nice as a test-set because of the wide subject age range, and the quality of the scans, which, while good, are not great, but are typical of what Freesurfer has to handle.
The Buckner40Notes wiki page documents the work of examining these brains, slice by slice, to verify their correctness, in terms of how well Freesurfer has reconstructed the white and pial surfaces, and segmented the subcortical structures.
It is important that Freesurfer's recon-all processing stream continue to process all of the subjects successfully, where that means the results are as good as or better than prior 'known-good' results (the reference set). (Note, if Freesurfer improves the results, then a new reference set is created, aka 'reset the standard').
Historical tests
The location of Buckner40 reference datasets with respect to previously released versions, and the development version, can be found at the following locations:
/space/freesurfer/subjects/test/buckner40/stable5_1_0 /space/freesurfer/subjects/test/buckner40/stable5_3_0 /space/freesurfer/subjects/test/buckner40/stable6 /space/freesurfer/subjects/test/buckner40/dev
And Adni60:
/space/freesurfer/subjects/test/adni60/stable5_1_0 /space/freesurfer/subjects/test/adni60/stable5_3_0 /space/freesurfer/subjects/test/adni60/stable6 /space/freesurfer/subjects/test/adni60/dev
Running the test
The 2 main scripts for running the Buckner40 and Adni60 tests are located here:
/space/freesurfer/subjects/test/scripts/comparison_test.sh /space/freesurfer/subjects/test/scripts/comparison_utils.sh
To run the Buckner40 and generate groupstats, ssh to launchpad and run the comparison_test.sh as follows:
$> ssh launchpad
$> /space/freesurfer/subjects/test/scripts/comparison_test.sh \
-fs /usr/local/freesurfer/dev \
-subjects_dir /space/freesurfer/subjects/test/buckner40/dev \
-generate_groupstats /space/freesurfer/subjects/test/buckner40/dev/groupstats_2016_07_30To run the Adni60 and generate groupstats, call the scrpts the exact same way, just replace "Buckner40" in the path with "Adni60":
$> ssh launchpad
$> /space/freesurfer/subjects/test/scripts/comparison_test.sh \
-fs /usr/local/freesurfer/dev \
-subjects_dir /space/freesurfer/subjects/test/adni60/dev \
-generate_groupstats /space/freesurfer/subjects/test/adni60/dev/groupstats_2016_07_30
Running tests as a cronjob
The Buckner40 and Adni60 datasets should be run on a weekly, monthly basis via cron, and should always be run (manually) as a precursor to a merge of the development branch with the master branch. To run these tests as a cronjob every week on Friday at midnight, the crontab file on launchpad should look as follows:
########################################## ## Run Buckner40 once per week (Friday, 11pm) ########################################## FREESURFER_DEV=/usr/local/freesurfer/dev BUCKNER40_DEV=/space/freesurfer/subjects/test/buckner40/dev BUCKNER40_DEV_GROUPSTATS=$BUCKNER40_DEV/groupstats_$(date +"%Y-%m-%d") 00 23 * * 5 pbsubmit -f -m zkaufman -c "/space/freesurfer/subjects/test/scripts/comparison_test.sh -fs $FREESURFER_DEV -subjects_dir $BUCKNER40_DEV -generate_groupstats $BUCKNER40_DEV_GROUPSTATS"
Prelease procedure
In preparation for a release, a new folder should be created named for the expected freesurfer version number. For example, inpreparation of the release of freesurfer v7 we would create a "stable7" folder:
$> mkdir /space/freesurfer/subjects/test/buckner40/stable7
In that folder should be a group.fsgd, raw_origs.tar.gz, and subjects.txt files. These files already exist in the parent directory (/space/freesurfer/subjects/test/buckner40) and thus can be symlinked at follows:
$> cd /space/freesurfer/subjects/test/buckner40/stable7 $> ln -s ../buckner40.fsgd group.fsgd $> ln -s ../buckner40_origs.tar.gz raw_origs.tar.gz $> ln -s ../subjects.txt
The subjects data and groupstats can then be run continually using the freesurfer release candidates until it is finally run WITH THE ACTUAL FREESURFER VERSION TO BE RELEASED. The subject data and groupstats should be preserved for posterity with the exact version of freesurfer that is to be released:
$> ssh launchpad
$> /space/freesurfer/subjects/test/scripts/comparison_test.sh \
-fs /usr/local/freesurfer/stable7 \
-subjects_dir /space/freesurfer/subjects/test/buckner40/stable \
-generate_groupstats /space/freesurfer/subjects/test/buckner40/stable7/groupstats_2016_07_30