This page describes and provides access to a table which shows the relationships between steps in recon-all, and files that are read, created, written or deleted at each step. 

The main innovation is that this info has been derived from an actual run of Freesurfer, not just from documentation which can so easily be out of sync with the real product.

Applies to: Freesurfer version 3.0.5 2007-02-07

== Download Excel version ==

Though an image of the full chart appears at the bottom of this page, it's more tractable to inspect (or print) your own copy of the Excel version, which you can download from this link:

[[attachment:gw_FS_FileIO_20070618.xls]]

== Data notes ==
1. C = Create; I = In (read); O = Out (write); D = Delete.

2. You'll see that the most conspicuous feature of the table is a more-or-less diagonal line of "C"s (indicating "create"), as successive steps create new files based on Input files produced in previous steps. I listed the steps in execution order from top to bottom, and I have deliberately organized the files (ie: columns) in order of creation-step, so as to create this visually clarifying diagonal "cascade" effect.

3. To avoid clutter, I have omitted columns for most log files, and all "touch" files.

4. Unlike my previous effort of this type, in this chart I've retained steps and files from right hemi as well as left hemi processing. I did this to more clearly see steps that involve both left and right. Inspecting the chart, you'll see that each side starts with a "tessellate" operation, and the "Right/Left" column helps to keep you oriented.

== Method ==
In brief, this chart builds on directory timestamp data gathered during a run of recon-all on the "bert" sample subject. Gathering this data entails two parts:

 1. A "directory survey" script (FWIW in python) recursively navigates the Freesurfer directories, capturing read, modify and status-change date-time information for each file.

 2. Commands added to the recon-all script to run the directory survey script before and after each existing recon-all command. More precisely, recon-all's existing code builds a command string at each step in variable "cmd", and then runs it by invoking "$cmd".  I manually searched for each such location, and inserted my own command to invoke "before" and "after" versions of the directory survey script.

Once the run is complete, the directory-survey data is slurped into a database, cleaned up, and then a cross-tab query organizes the data in Files-vs-steps form.

=== Wrinkles ===
 1. The main wrinkle is that using the file system's read, modify and status-change date-time feature is a bit iffy.  First, this only works on some systems -- on others not all the data-time values work, or they may have granularity too coarse to be useful (despite being reported in seconds). I found that this did work OK on Centos 4 (RHEL 4, or Fedora Core something) with local files. It did not work on Centos 4 with the files on Mac OS X server accessed via NFS. Don't know whether the problems lies with Mac or NFS.

 2. Even on a system where directory timestamps work acceptably well, it's obviously necessary to make sure the successive steps in the recon-all process do not occur within one second of each other, otherwise the available timestamp resolution will be unable to report a difference in times. For that reason, the directory-survey script incorporates a delay of at least two seconds.

 3. Backup jobs running in the middle of the proceedings can log extra reads that are nothing to do with recon-all :-).

== Image of the Files vs Steps table ==
(As noted above, you're probably better off with the Excel version -- so consider this image mainly an advertisement :-).

{{attachment:gw_FS_FileIO_20070618.gif}}

== Example python script to capture directory timestamps ==
{{{
#!/usr/bin/python
import sys
import os
import time

# Arg1: label for this dir snapshot
# Arg2: dir to be ls'ed 
# Use > or >> to write or append to file

label       = sys.argv[1]
label       = '"' + label + '"'
startdir    = sys.argv[2]
# outfilepath = sys.argv[3]

# atime;    /* Time of last access */ 
# mtime;    /* Time of last data modification */ 
# ctime;    /* Time of last file status change */ 
# Supposedly...
# ctime includes things like chmod, chown, etc. which do not change the 
# contents of the file, but impact the file itself. 


#-----------------------
def secs2str(secs):
  tt   = time.localtime(secs)
  tstr = time.strftime('%Y-%m-%d %H:%M:%S', tt)
  return tstr

#-----------------------

# print "====================="
# print sys.argv[0] + ' ' + label + ' ' + startdir + ' ' + outfilepath
# print sys.argv

runstr = 'run_' + time.strftime('%Y-%m-%d_%H:%M:%S')
# timestr = time.strftime('%Y-%m-%d %H:%M:%S')

allfiles = []

for (dirname, dirshere, fileshere) in os.walk(startdir):
    for filename in fileshere:
        pathname = os.path.join(dirname, filename)
        statinfo = os.stat(pathname)
        statstr = ( 'gwdirdump.py ' + label + ' ' + runstr
                   + ' "' + secs2str(statinfo.st_atime) + '"'
                   + ' "' + secs2str(statinfo.st_mtime) + '"'
                   + ' "' + secs2str(statinfo.st_ctime) + '" ' + filename + ' ' + dirname)
        print statstr

# Sleep to prevent more file changes before time ticks over at least a second!
time.sleep(2)

}}}

= Document Author(s) =
Graham Wideman (GrahamWideman)