Unified CLI
clpipe provides a unifed command line interface under the “clpipe” command. This interface uses the same processing functions as clpipe’s traditional commands, except the are grouped together and given slightly different names. Unifying the commands makes them easier to find and conceptualize in relation to one another without having to rely too much on documentation.
Running this command on its own will show you a subset of clpipe’s processing commands, in the order that they are intended to be used:
> clpipe
Usage: clpipe [OPTIONS] COMMAND [ARGS]...
Welcome to clpipe. Please choose a processing command.
Options:
-v, --version Display clpipe's version.
--help Show this message and exit.
Commands:
setup Set up a clpipe project
dicom Raw DICOM Data Commands.
bids BIDS Commands
preprocess Submit BIDS-formatted images to fMRIPrep
postprocess Additional preprocessing for connectivity analysis
postprocess2 Additional preprocessing for GLM or connectivity analysis
glm GLM Commands
status Check the status of your project
NOTE - ROI extraction has yet to be added because one of its dependencies causes the CLI to output an un-hidable warning
To see the options for a particular sub-command, call this command after “clpipe” and ask for –help:
> clpipe setup --help
Usage: clpipe setup [OPTIONS]
Set up a clpipe project
Options:
-project_title TEXT [required]
-project_dir DIRECTORY Where the project will be located. [required]
-source_data DIRECTORY Where the raw data (usually DICOMs) are located.
-move_source_data Move source data into project/data_DICOMs folder.
USE WITH CAUTION.
-symlink_source_data Symlink the source data into project/data_dicoms.
Usually safe to do.
-debug Flag to enable detailed error messages and traceback
--help Show this message and exit.
This command can be used just like the original project_setup command in clpipe:
> clpipe setup -project_title "My Project" -project_dir . -debug
Some of the clpipe commands, like bids and glm, contain their own nested sub commands, which can be viewed by calling the top command:
> clpipe bids
Usage: clpipe bids [OPTIONS] COMMAND [ARGS]...
BIDS Commands
Options:
--help Show this message and exit.
Commands:
validate Check that the given directory conforms to the BIDS standard
These commands contain their own help dialouge as well:
> clpipe bids validate --help
Usage: clpipe bids validate [OPTIONS] [BIDS_DIR]
Check that the given directory conforms to the BIDS standard
Options:
-config_file FILE Uses a given configuration file
-log_dir FILE Where to put HPC output files (such as SLURM output
files)
-verbose Creates verbose validator output. Use if you want to see
ALL files with errors/warnings.
-submit Flag to submit commands to the HPC
-interactive Run in an interactive session. Only use in an interactive
compute session.
-debug Flag to enable detailed error messages and traceback
--help Show this message and exit.
Here we perform the command equivalent to “bids_validate”:
> clpipe bids validate -config_file path/to/my/config -submit
Finally, here is an equivalent command taking advantage of short option names:
> clpipe bids validate -c path/to/my/config -s
Here is a description of all available commands:
clpipe
Welcome to clpipe.
Please choose one of the commands below for more information.
If you’re not sure where to begin, please see the documentation at: https://clpipe.readthedocs.io/en/latest/index.html
clpipe [OPTIONS] COMMAND [ARGS]...
Options
- -version, -v
Display clpipe’s version.
bids
BIDS Commands.
Please choose one of the commands below for more information.
clpipe bids [OPTIONS] COMMAND [ARGS]...
bids_validate
Validate if a directory BIDS standard.
Validates the directory at BIDS_DIR, or at the BIDS directory in your config file’s DICOMToBIDSOptions if -config_file is given.
Results are viewable in logs/bids_validation_logs unless -interactive is used.
clpipe bids bids_validate [OPTIONS] [BIDS_DIR]
Options
- -config_file, -c <config_file>
Required The path to your clpipe configuration file.
- -log_dir <log_dir>
Where to put your HPC output files (such as SLURM output files).
- -verbose, -v
Creates verbose validator output. Use if you want to see ALL files with errors/warnings.
- -submit, -s
Flag to submit commands to the HPC.
- -interactive
Run in interactive mode. Only use in an interactive compute session.
- -debug, -d
Flag to enable detailed error messages and traceback.
Arguments
- BIDS_DIR
Optional argument
bids_validate
Validate if a directory BIDS standard.
Validates the directory at BIDS_DIR, or at the BIDS directory in your config file’s DICOMToBIDSOptions if -config_file is given.
Results are viewable in logs/bids_validation_logs unless -interactive is used.
clpipe bids_validate [OPTIONS] [BIDS_DIR]
Options
- -config_file, -c <config_file>
Required The path to your clpipe configuration file.
- -log_dir <log_dir>
Where to put your HPC output files (such as SLURM output files).
- -verbose, -v
Creates verbose validator output. Use if you want to see ALL files with errors/warnings.
- -submit, -s
Flag to submit commands to the HPC.
- -interactive
Run in interactive mode. Only use in an interactive compute session.
- -debug, -d
Flag to enable detailed error messages and traceback.
Arguments
- BIDS_DIR
Optional argument
config
Configuration-related commands.
clpipe config [OPTIONS] COMMAND [ARGS]...
get_default
Generates a default configuration file for your project.
clpipe config get_default [OPTIONS]
Options
- -outputFile, -o <outputfile>
Filepath for the outputted configuration file.
update
Updates an existing configuration file with any new fields. Does not modify existing fields.
clpipe config update [OPTIONS]
Options
- -config_file, -c <config_file>
Required Configuration file to update.
- -backup
Automatically backup the previous configuration file
convert2bids
Convert DICOM files to BIDS format.
Providing no SUBJECTS will default to all subjects. List subject IDs in SUBJECTS to process specific subjects:
> clpipe convert2bids 123 124 125 …
Available subject IDs are determined by the dicom_dir_format string.
clpipe convert2bids [OPTIONS] [SUBJECTS]...
Options
- -config_file, -c <config_file>
Required The path to your clpipe configuration file.
- -conv_config_file <conv_config_file>
A conversion definition file, either a dcm2bids conversion config .json file or a heudiconv heuristic .py file.
- -dicom_dir, -i <dicom_dir>
The folder where subject dicoms are located.
- -dicom_dir_format <dicom_dir_format>
Format string which specifies how subjects/sessions are organized within the dicom_dir. Example: {subject}_{session}. See https://clpipe.readthedocs.io/en/latest/bids_convert.html for more details.
- -BIDS_dir, -o <bids_dir>
The dicom info output file name.
- -overwrite
Overwrite existing BIDS data?
- -log_dir <log_dir>
Where to put your HPC output files (such as SLURM output files).
- -subject <subject>
DEPRECATED: specify one subject to process - can give an arbitrary number of subjects as arguments now.
- -session <session>
Specify the session to convert. Available sessions determined by the {session} placeholder given by dicom_dir_format.
- -longitudinal
Convert all subjects/sessions into individual pseudo-subjects. Use if you do not want T1w averaged across sessions during FMRIprep
- -submit, -s
Flag to submit commands to the HPC.
- -debug, -d
Flag to enable detailed error messages and traceback.
Arguments
- SUBJECTS
Optional argument(s)
dicom
Raw DICOM Data Commands.
Please choose one of the commands below for more information.
clpipe dicom [OPTIONS] COMMAND [ARGS]...
convert2bids
Convert DICOM files to BIDS format.
Providing no SUBJECTS will default to all subjects. List subject IDs in SUBJECTS to process specific subjects:
> clpipe convert2bids 123 124 125 …
Available subject IDs are determined by the dicom_dir_format string.
clpipe dicom convert2bids [OPTIONS] [SUBJECTS]...
Options
- -config_file, -c <config_file>
Required The path to your clpipe configuration file.
- -conv_config_file <conv_config_file>
A conversion definition file, either a dcm2bids conversion config .json file or a heudiconv heuristic .py file.
- -dicom_dir, -i <dicom_dir>
The folder where subject dicoms are located.
- -dicom_dir_format <dicom_dir_format>
Format string which specifies how subjects/sessions are organized within the dicom_dir. Example: {subject}_{session}. See https://clpipe.readthedocs.io/en/latest/bids_convert.html for more details.
- -BIDS_dir, -o <bids_dir>
The dicom info output file name.
- -overwrite
Overwrite existing BIDS data?
- -log_dir <log_dir>
Where to put your HPC output files (such as SLURM output files).
- -subject <subject>
DEPRECATED: specify one subject to process - can give an arbitrary number of subjects as arguments now.
- -session <session>
Specify the session to convert. Available sessions determined by the {session} placeholder given by dicom_dir_format.
- -longitudinal
Convert all subjects/sessions into individual pseudo-subjects. Use if you do not want T1w averaged across sessions during FMRIprep
- -submit, -s
Flag to submit commands to the HPC.
- -debug, -d
Flag to enable detailed error messages and traceback.
Arguments
- SUBJECTS
Optional argument(s)
flywheel_sync
Sync your DICOM data with Flywheel.
You must first login to Flywheel with ‘fw login’ to sync. See the clpipe documentation on flywheel_sync for further help.
clpipe dicom flywheel_sync [OPTIONS]
Options
- -config_file, -c <config_file>
The path to your clpipe configuration file.
- -source_url <source_url>
The path to your project in Flywheel. Starts with fw://. You can browse your available projects with “fw ls”
- -dropoff_dir <dropoff_dir>
Where to sync your files.
- -submit, -s
Flag to submit commands to the HPC.
- -debug, -d
Flag to enable detailed error messages and traceback.
flywheel_sync
Sync your DICOM data with Flywheel.
You must first login to Flywheel with ‘fw login’ to sync. See the clpipe documentation on flywheel_sync for further help.
clpipe flywheel_sync [OPTIONS]
Options
- -config_file, -c <config_file>
The path to your clpipe configuration file.
- -source_url <source_url>
The path to your project in Flywheel. Starts with fw://. You can browse your available projects with “fw ls”
- -dropoff_dir <dropoff_dir>
Where to sync your files.
- -submit, -s
Flag to submit commands to the HPC.
- -debug, -d
Flag to enable detailed error messages and traceback.
glm
General Linear Model (GLM) Commands.
Please choose one of the commands below for more information.
clpipe glm [OPTIONS] COMMAND [ARGS]...
apply_mumford
Apply the Mumford registration workaround to L1 FEAT folders.
Applied by default in glm-l2-preparefsf. This command is useful for applying the Mumford workaround to single-run subjects who skip L2, allowing you to still combine them with multiple-run subjects at L3.
Must provide GLM config file OR a path to your L1 FEAT folders.
clpipe glm apply_mumford [OPTIONS]
Options
- -glm_config_file, -g <glm_config_file>
Your GLM configuration file.
- -l1_feat_folders_path <l1_feat_folders_path>
Directory containing your L1 FEAT folders.
- -remove_reg_standard
Remove reg_standard folders (generated by L2) in addition to reg.
- -debug, -d
Flag to enable detailed error messages and traceback
fsl_onset_extract
Convert onset files to FSL’s 3 column format.
clpipe glm fsl_onset_extract [OPTIONS]
Options
- -config_file, -c <config_file>
Required Use a given configuration file.
- -glm_config_file, -g <glm_config_file>
Required Use a given GLM configuration file.
- -debug, -d
Print detailed processing information and traceback for errors.
get_default_config
Generates a default GLM configuration file for your project.
clpipe glm get_default_config [OPTIONS]
Options
- -outputFile, -o <outputfile>
Filepath for the outputted configuration file.
launch
Launch all prepared .fsf files for L1 or L2 GLM analysis.
LEVEL is the level of anlaysis, L1 or L2
MODEL must be a a corresponding L1 or L2 model from your GLM configuration file.
clpipe glm launch [OPTIONS] LEVEL MODEL
Options
- -glm_config_file, -g <glm_config_file>
Required The path to your clpipe configuration file.
- -test_one
Only submit one job for testing purposes.
- -submit, -s
Flag to submit commands to the HPC.
- -debug, -d
Flag to enable detailed error messages and traceback.
Arguments
- LEVEL
Required argument
- MODEL
Required argument
prepare
Propagate an .fsf file template for L1 or L2 GLM analysis.
LEVEL is the level of anlaysis, L1 or L2
MODEL must be a a corresponding L1 or L2 model from your GLM configuration file.
clpipe glm prepare [OPTIONS] LEVEL MODEL
Options
- -glm_config_file, -g <glm_config_file>
Required The path to your clpipe configuration file.
- -debug, -d
Flag to enable detailed error messages and traceback.
Arguments
- LEVEL
Required argument
- MODEL
Required argument
report_outliers
Generate a confound outliers report.
Must provide one of either –confounds_dir or –confounds_file.
clpipe glm report_outliers [OPTIONS]
Options
- --confounds_dir <confounds_dir>
Path to a directory containing subjects and confounds files.
- --confounds_file <confounds_file>
Path to confounds file
- --output_file <output_file>
Path to save outlier count results.
- --confound_suffix <confound_suffix>
Confound file to search for, like ‘confounds.tsv’
postprocess
Additional processing for GLM or connectivity analysis.
Providing no SUBJECTS will default to all subjects. List subject IDs in SUBJECTS to process specific subjects:
> clpipe postprocess2 123 124 125 …
clpipe postprocess [OPTIONS] [SUBJECTS]...
Options
- -config_file, -c <config_file>
Required The path to your clpipe configuration file.
- -fmriprep_dir, -i <fmriprep_dir>
Which fmriprep directory to process. If a configuration file is provided with a BIDS directory, this argument is not necessary. Note, must point to the
fmriprepdirectory, not its parent directory.
- -output_dir, -o <output_dir>
Where to put the postprocessed data. If a configuration file is provided with a output directory, this argument is not necessary.
- -processing_stream, -p <processing_stream>
Specify a processing stream to use defined in your configuration file.
- -log_dir <log_dir>
Where to put your HPC output files (such as SLURM output files).
- -index_dir <index_dir>
Give the path to an existing pybids index database.
- -refresh_index, -r
Refresh the pybids index database to reflect new fmriprep artifacts.
- -batch, -no-batch
Flag to create batch jobs without prompting.
- -cache, -no-cache
- -submit, -s
Flag to submit commands to the HPC.
- -debug, -d
Flag to enable detailed error messages and traceback.
Arguments
- SUBJECTS
Optional argument(s)
preprocess
Submit BIDS-formatted images to fMRIPrep.
Providing no SUBJECTS will default to all subjects. List subject IDs in SUBJECTS to process specific subjects:
> clpipe preprocess 123 124 125 …
clpipe preprocess [OPTIONS] [SUBJECTS]...
Options
- -config_file, -c <config_file>
Required The path to your clpipe configuration file.
- -bids_dir, -i <bids_dir>
The dicom info output file name.
- -working_dir <working_dir>
Where to generate the working directory.
- -output_dir, -o <output_dir>
Where to put the preprocessed data. If a configuration file is provided with a output directory, this argument is not necessary.
- -log_dir <log_dir>
Where to put your HPC output files (such as SLURM output files).
- -submit, -s
Flag to submit commands to the HPC.
- -debug, -d
Flag to enable detailed error messages and traceback.
Arguments
- SUBJECTS
Optional argument(s)
project_setup
Initialize a clpipe project.
clpipe project_setup [OPTIONS]
Options
- -project_title <project_title>
Choose a title for your project.
- -project_dir <project_dir>
Required Where the project will be located.
- -source_data <source_data>
Where the raw data (usually DICOMs) are located.
- -move_source_data
Move source data into project/data_DICOMs folder. USE WITH CAUTION.
- -symlink_source_data
Symlink the source data into project/data_dicoms. Usually safe to do.
- -profile <profile>
The batch profile you would like to use. Default is unc. The options include: “unc”, “duke”, “uva”, “pitt”
- -debug
Flag to enable detailed error messages and traceback.
reports
Generate reports for your project.
Please choose one of the commands below for more information.
clpipe reports [OPTIONS] COMMAND [ARGS]...
fmriprep
Create a .zip directory of all fMRIPrep reports.
clpipe reports fmriprep [OPTIONS]
Options
- -config_file, -c <config_file>
Required The configuration file for the current data processing setup.
- -output_name, -o <output_name>
Path and name of the output archive. Defaults to current working directory and “fMRIPrep_Reports.zip”
- -clear_temp, -keep_temp
Keep or clear the built temporary directory. Defaults to clear_temp.
- -debug, -d
Print traceback on errors.
roi
Region of Interest (ROI) Commands.
Please choose one of the commands below for more information.
clpipe roi [OPTIONS] COMMAND [ARGS]...
atlases
Display all available atlases.
clpipe roi atlases [OPTIONS]
extract
Extract ROIs with a given atlas.
clpipe roi extract [OPTIONS] [SUBJECTS]...
Options
- -config_file, -c <config_file>
Use a given configuration file. If left blank, uses the default config file, requiring definition of BIDS, working and output directories. This will extract all ROI sets specified in the configuration file.
- -target_dir, -i <target_dir>
Which postprocessed directory to process. If a configuration file is provided with a target directory, this argument is not necessary.
- -target_suffix <target_suffix>
Which target suffix to process. If a configuration file is provided with a target suffix, this argument is not necessary.
- -output_dir, -o <output_dir>
Where to put the ROI extracted data. If a configuration file is provided with a output directory, this argument is not necessary.
- -task <task>
Which task to process. If none, then all tasks are processed.
- -atlas_name <atlas_name>
What atlas to use. Use the command ‘clpipe roi atlases’ to see which are available. When specified for a custom atlas, this is what the output files will be named.
- -custom_atlas <custom_atlas>
A custom atlas image, in .nii or .nii.gz for label or maps, or a .txt tab delimited set of ROI coordinates if for a sphere atlas. Not needed if specified in config.
- -custom_label <custom_label>
A custom atlas label file. Not needed if specified in config.
- -custom_type <custom_type>
What type of atlas? (label, maps, or spheres). Not needed if specified in config.
- -sphere_radius <sphere_radius>
Sphere radius in mm. Only applies to sphere atlases.
- -overlap_ok
Are overlapping ROIs allowed?
- -overwrite
Overwrite existing ROI timeseries?
- -log_output_dir <log_output_dir>
Where to put HPC output files (such as SLURM output files). If not specified, defaults to <outputDir>/batchOutput.
- -submit, -s
Flag to submit commands to the HPC
- -single
Flag to directly run command. Used internally.
- -debug, -d
Flag to enable detailed error messages and traceback
Arguments
- SUBJECTS
Optional argument(s)
status
Check the status of your project.
clpipe status [OPTIONS]
Options
- -config_file, -c <config_file>
The path to your clpipe configuration file.
- -cache_file <cache_file>
Path to your status cache file.
templateflow_setup
Installs the templates for preprocessing listed in TemplateFlowTemplates. If you don’t run this, fMRIPrep defaults to MNI152NLin2009cAsym. If you do, fMRIPrep will create a copy of each image in every space listed.
clpipe templateflow_setup [OPTIONS]
Options
- -config_file <config_file>
Use a given configuration file. If left blank, uses the default config file, requiring definition of BIDS, working and output directories.
- -debug
Flag to enable detailed error messages and traceback