Differences between revisions 5 and 6

Co-registration of dissection photographs of coronal slabs into a 3D volume

For FreeSurfer 8.0 or development versions downloaded before May 2025, please see https://surfer.nmr.mgh.harvard.edu/fswiki/mri_3d_photo_recon_2024.

If you use this tools in your analysis, please cite:

* Machine learning of dissection photographs and surface scanning for quantitative 3D neuropathology. H Gazula, H Tregidgo, B Billot, Y Balbastre, J Williams Ramirez, R Herisse, LJ Deden-Binder, A Casamitjana, EJ Melief, CS Latimer, MD Kilgore, M Montine, E Robinson, E Blackburn, MS Marshall, TR Connors, DH Oakley, MP Frosh, SI Young, K Van Leemput, AV Dalca, B Fischl, CL Mac Donald, CD Keene, B Hyman, and JE Iglesias. Under revision.)

This is a newer version of our older tool with several key improvements:

More flexible framework, accepting a variable number of references (e.g., surface scan and/or MRI scan or MNI atlas etc).
Faster optimization
Easier configuration of parameters: it self configures depending on type(s) of input(s)
Supports slabs of variable thickness
New feature: machine learning imputation / interpolation to 1mm slab thickness (fun stuff! See pictures below)

The main options are (required arguments in bold): mri_3d_photo_recon

--input_photo_dir [DIR]	Directory with input pixel-corrected photos.
--input_segmentation_dir [DIR]	Directory with input slab masks/segmentations.
--hemisphere [HEMI]	Must be left, right, or both
--slice_thickness [THICKNESS]	Slice thickness in mm (will be finetured if possible); if unsure, give conservative (thinner) estimate.
--photo_resolution [RESO]	Resolution of the photos in mm; if using PhotoToools this would be 0.1.
--output_directory [DIR]	Output directory with reconstructed photo volume and reference.
--photos_of_posterior_side	Use this flag if photos are taken of posterior side of slabs (default is anterior side)
--order_posterior_to_anterior	Use this flag if photos are ordered from posterior to anterior (default is anterior to posterior)
--ref_mesh [FILE]	Reference surface mesh (if available)
--mesh_reorient_with_indices [INDICES]	triplet of vertex indices to reorient mesh; format is just i,j,k (see details below)
--ref_mri [FILE]	Reference MRI scan (if available)
--ref_mri_synthseg [FILE]	SynthSeg of reference MRI scan (will be computed if scan provided and SynthSeg does not exist)
--ref_mri_synthsr [FILE]	SynthSR of reference MRI; only needed if MRI is provided and is not a 1mm T1 (will be computed if it does not exist)
--low_field_synthsr [FILE]	Use low-field version of SynthSR (eg for Hyperfine scans)
--threads [THREADS]	Number of cores to be used. Default is 1. You can use -1 to use all available cores
--gpu [INDEX]	Index of GPU to use (default is None, i.e., CPU mode)

Some additional niche options:

--fresh_tissue	This flag makes the regularizers more lenient to accommodate fresh tissue (typically more deformed)
--initial_stretch_factor_lr [FACTOR]	Initialize stretch of photos in left-right direction by this factor (useful with squashed specimens)
--weights [FILE]	CSV file with slab weights (follows order of photos). Format is just: weight1,weight2,...,weightN
--thickness_cap [MAXIMUM]	when using weights, limit the maximum estimated thickness to this value (in mm). More details below.
--equalize_images	This flag activates histogram equalization of images (useful if they have low contrast, but can hurt the machine learning imputation)
--skip_bfgs	Flag to skip BFGS finetuning (i.e., do only Adam)
--deform_recon_dir [DIR]	Directory with FS dir of reference MRI scan, to deform surfaces etc (expects to find deform_recon_dir/surf)
--input_roi_dir [DIR]	Directory with ROI masks (typically in MNI space) to deform to photo reconstruction

There are also advanced options to control the exact control point spacing and regularization (run command with help flag -h), but these are self configured depending on the type of reference and the --fresh_tissue flag so in principle you should not need to modify them.

Vertex indices for reorienting reference meshes (--mesh_reorient_with_indices)

These are the vertex indices of the frontal pole, occipital pole, and top of the central sulcus, separated with commas (e.g., -–mesh_reorient_with_indices 1000,3000,2000). You can find these indices with Freeview.


Frontal Pole	Occipital Pole	Top of the central sulcus

Slab weights

If you cut slabs of uneven thickness, then the --thickness option expects the average thickness. However, you can also put the slabs on a scale, record the weights, and pass the to the code with the --weights option. The weights must be in a CSV file (comma separated values), following the order of photos. The format is just: weight1,weight2,...,weightN (in grams). The thicknesses get then estimated from the weights and the surface areas.

Now: the thickness estimation can sometimes be unrealistic due to funny combinations of weights and surface areas. The code tries to detect these situations automatically. If this happens, it will prompt you to use the --thickness_cap option, which limits the maximum thickness that a slab can have in this variable thickness mode. In principle, you should not need to use --thickness_cap unless you are proompted to do so.

Machine learning imputation

The new code uses machine learning to impute / interpolate slices every 1mm (see example below).

Example

mri_3d_photo_recon \
--input_photo_dir ./deformed \
--input_segmentation_dir ./connected_components \
--ref_surface ./mesh/case.stl \
--mesh_reorient_with_indices 999,1010,333333 \
--photos_of_posterior_side \
--slice_thickness 8 \
--photo_resolution 0.1
--hemisphere left

-Deletions are marked like this.
+Additions are marked like this.
 Line 14:
- * New feature: machine learning interpolation to 1mm slab thickness (fun stuff! See pictures below)
+ * New feature: machine learning imputation / interpolation to 1mm slab thickness (fun stuff! See pictures below)
 Line 16:
-The main options are:

mri_3d_photo_recon -h

||--input_photo_dir ||Directory with input pixel-corrected photos. ||

||--input_segmentation_dir ||Directory with input slab masks/segmentations ||

||--hemisphere ||Must be left, right, or both ||

||--slice_thickness ||Slice thickness in mm (if unsure, give conservative estimate, i.e., a bit lower value). ||

||--photo_resolution ||Resolution of the photos in mm (if using PhotoToools this would be 0.1). ||
+The main options are (required arguments in bold):

mri_3d_photo_recon 

||'''--input_photo_dir''' [DIR] ||Directory with input pixel-corrected photos. ||

||'''--input_segmentation_dir''' [DIR] ||Directory with input slab masks/segmentations. ||

||'''--hemisphere''' [HEMI] ||Must be left, right, or both ||

||'''--slice_thickness''' [THICKNESS] ||Slice thickness in mm (will be finetured if possible); if unsure, give conservative (thinner) estimate. ||

||'''--photo_resolution''' [RESO] ||Resolution of the photos in mm; if using PhotoToools this would be 0.1. ||

||'''--output_directory''' [DIR] ||Output directory with reconstructed photo volume and reference. ||

||--photos_of_posterior_side || Use this flag if photos are taken of posterior side of slabs (default is anterior side)||

||--order_posterior_to_anterior || Use this flag if photos are ordered from posterior to anterior (default is anterior to posterior) ||

||--ref_mesh [FILE] || Reference surface mesh (if available) ||

||--mesh_reorient_with_indices [INDICES] || triplet of vertex indices to reorient mesh; format is just i,j,k (see details below)||

||--ref_mri [FILE] || Reference MRI scan (if available)||

||--ref_mri_synthseg [FILE] || SynthSeg of reference MRI scan (will be computed if scan provided and SynthSeg does not exist) ||

||--ref_mri_synthsr [FILE] || SynthSR of reference MRI; only needed if MRI is provided and is not a 1mm T1 (will be computed if it does not exist) ||

||--low_field_synthsr [FILE] || Use low-field version of SynthSR (eg for Hyperfine scans) ||

||--threads [THREADS] || Number of cores to be used. Default is 1. You can use -1 to use all available cores ||

||--gpu [INDEX] || Index of GPU to use (default is None, i.e., CPU mode) ||
-Line 24:
+Line 35:
+Some additional niche options:

||--fresh_tissue  || This flag makes the regularizers more lenient to accommodate fresh tissue (typically more deformed)||

||--initial_stretch_factor_lr [FACTOR] || Initialize stretch of photos in left-right direction by this factor (useful with squashed specimens) ||

||--weights [FILE] || CSV file with slab weights (follows order of photos). Format is just: weight1,weight2,...,weightN ||

||--thickness_cap [MAXIMUM] || when using weights, limit the maximum estimated thickness to this value (in mm). More details below.||

||--equalize_images || This flag activates histogram equalization of images (useful if they have low contrast, but can hurt the machine learning imputation) ||

||--skip_bfgs || Flag to skip BFGS finetuning (i.e., do only Adam) ||

||--deform_recon_dir [DIR] || Directory with FS dir of reference MRI scan, to deform surfaces etc (expects to find deform_recon_dir/surf) ||

||--input_roi_dir [DIR] || Directory with ROI masks (typically in MNI space) to deform to photo reconstruction ||
-Line 25:
+Line 45:
+There are also advanced options to control the exact control point spacing and regularization (run command with help flag -h), but these are self configured depending on the type of reference and the --fresh_tissue flag so in principle you should not need to modify them.
-Line 26:
+Line 47:
-||--ref_mask ||When using a binary volume as a reference. ||

||--ref_surface ||Using a 3D surface scan as a reference. ||

||--ref_soft_mask ||Using the provided average atlas. Best for using retrospective processing on data without a better reference. ||

||--mesh_reorient_with_indices ||Vertex indices of the frontal pole, occipital pole, and top of the central sulcus, separated with commas, for mesh alignment. ||

||--photos_of_posterior_side ||Use when photos are taken of the posterior side of slabs (default is anterior side). ||

||--order_posterior_to_anterior ||Use when photos are ordered from posterior to anterior (default is anterior to posterior). ||

||--allow_z_stretch ||Use to adjust the slice thickness to best match the reference. You should probably '''*never*''' use this with soft references (ref_soft_mask). ||

||--rigid_only_for_photos ||Switch on if you want photos to deform only rigidly (not affine). ||
+<<BR>>
-Line 35:
+Line 49:
+==== Vertex indices for reorienting reference meshes (--mesh_reorient_with_indices) ====
-Line 36:
+Line 51:
-||--output_directory ||Output directory with reconstructed photo volume and reference || <<BR>>





  -h, --help            show help message





  --input_photo_dir [INPUT_PHOTO_DIR]

                        Directory with input photos (required)



  --input_segmentation_dir [INPUT_SEGMENTATION_DIR]

                        Directory with input slab masks / segmentations (required)



  --hemisphere HEMISPHERE

                        hemisphere; must be left, right, or both (required)



  --slice_thickness SLICE_THICKNESS

                        Slice thickness in mm (required); will be finetuned if possible



  --photo_resolution PHOTO_RESOLUTION

                        Resolution of the photos in mm (required)



  --output_directory OUTPUT_DIRECTORY

                        Output directory with reconstructed photo volume and reference (required)



  --ref_mri REF_MRI     

                        Reference MRI scan (if available)



  --ref_mri_synthseg REF_MRI_SYNTHSEG

                        SynthSeg of reference MRI scan (will be computed if it does not exist)



  --ref_mri_synthsr REF_MRI_SYNTHSR

                        SynthSR of reference MRI; only needed if MRI is not 1mm T1 (will be computed if it does not exist)



  --low_field_synthsr   

                        Use low-field version of SynthSR (eg for Hyperfine scans)



  --input_roi_dir INPUT_ROI_DIR

                        Directory with ROi masks to deform to photo reconstruction (optional)



  --ref_mesh REF_MESH   

                        Reference surface mesh (if available)



  --mesh_reorient_with_indices MESH_REORIENT_WITH_INDICES

                        Indices to reorient mesh (see PhotoTools page in FreeSurfer wiki)



  --fresh_tissue        

                        Uses more lenient regularizers to accommodate fresh tissue



  --photos_of_posterior_side

                        Use when photos are taken of posterior side of slabs (default is anterior side)



  --order_posterior_to_anterior

                        Use when photos are ordered from posterior to anterior (default is anterior to posterior)



  --initial_stretch_factor_lr INITIAL_STRETCH_FACTOR_LR

                        Initialize stretch of photos in left-right direction by this factor (useful with squashed brains)



  --weights WEIGHTS     

                        CSV file with slab weights (follows order of photos). Format is just: weight1,weight2,...,weightN



  --thickness_cap MAX

                        when using weights, limit the maximum estimated thickness to this value (in mm). 

                        This is useful when the geometric estimation of thicknesses is wrong.

                        The code (without this option) tries to detect these situations automatically and tells you when to enable this option



  --equalize_images     

                        Use to equalize images (useful if they have low contrast, but can hurt the machine learning imputation)



  --skip_bfgs           

                        Use to skip BFGS finetuning (i.e., do only Adam)



  --threads THREADS     

                        Number of cores to be used. Default is 1. You can use -1 to use all available cores



  --gpu GPU             

                        Index of GPU to use (default is None, i.e., CPU mode)



  --deform_recon_dir DEFORM_RECON_DIR

                        Directory with FS dir of reference, to deform surfaces etc (expects to find deform_recon_dir/surf) (optional)



  --cp_spacing_2d CP_SPACING_2D

                        (Advanced) Control point spacing for 2D deformation



  --cp_spacing_3d CP_SPACING_3D

                        (Advanced) Control point spacing for 3D deformation



  --k_lncc_mri K_LNCC_MRI

                        (Advanced) Weight of LNCC between reference MRI and reconstruction



  --k_dice_mri K_DICE_MRI

                        (Advanced) Weight of Dice between masks of reference and reconstruction



  --k_dif_slice_loss K_DIF_SLICE_LOSS

                        (Advanced) Weight of SSD between consecutive slices of reconstruction



  --k_mesh_loss K_MESH_LOSS

                        (Advanced) Weight of absolute distance to edge of masks from mesh vertices



  --k_regularizer K_REGULARIZER

                        (Advanced) Weight of regularizer of log_det(affine matrices)



  --k_regularizer_nonlin K_REGULARIZER_NONLIN

                        (Advanced) Weight of regularizer of 2D nonlinear deformation of photos



  --k_regularizer_nonlin3d K_REGULARIZER_NONLIN3D

                        (Advanced) Weight of regularizer of 3D nonlinear deformatin of reference







In Freeview, you can find the number corresponding to the vertices of each anatomical area to use in the --mesh_reorient_with_indices flag. These indices should be comma-separated in the order: frontal pole, occipital pole, and top of the central sulcus.<<BR>>
+These are the vertex indices of the frontal pole, occipital pole, and top of the central sulcus, separated with commas (e.g., -–mesh_reorient_with_indices 1000,3000,2000). You can find these indices with Freeview.
-Line 150:
+Line 55:
-For example, the --mesh_reorient_with_indices flag might look like this:

 . –mesh_reorient_with_indices 999999,999999,999999
+<<BR>>
-Line 153:
+Line 57:
-'''Example for running reconstruction''' <<BR>>
+==== Slab weights ====

                      

If you cut slabs of uneven thickness, then the --thickness option expects the average thickness. However, you can also put the slabs on a scale, record the weights, and pass the to the code with the --weights option. The weights must be in a CSV file (comma separated values), following the order of photos. The format is just: weight1,weight2,...,weightN (in grams). The thicknesses get then estimated from the weights and the surface areas.



Now: the thickness estimation can sometimes be unrealistic due to funny combinations of weights and surface areas. The code tries to detect these situations automatically. If this happens, it will prompt you to use the --thickness_cap option, which limits the maximum thickness that a slab can have in this variable thickness mode. In principle, you should not need to use --thickness_cap  unless you are proompted to do so.



<<BR>>



==== Machine learning imputation ====



The new code uses machine learning to impute / interpolate slices every 1mm (see example below).



|| {{attachment:imputation.png||wheight="254"}} ||



<<BR>>



==== Example ====
-Line 159:
+Line 80:
- . --photos_of_posterior_side --allow_z_stretch \
+ . --photos_of_posterior_side  \
-Line 162:
+Line 83:
+ . --hemisphere left