NV-Segment-CTMR Overview

Description:

NV-Segment-CTMR is a specialized foundation model for 3D medical image segmentation that excels at accurate, adaptable, automatic segmentation across anatomies and modalities, including computed tomography (CT) and magnetic resonance (MR) imaging. NV-Segment-CTMR adapts to varying conditions and anatomical regions, enabling comprehensive automated annotation workflows.

At the core of NV-Segment-CTMR are two automated workflows. Segment Everything enables whole-body exploration, which is crucial for understanding complex diseases affecting multiple organs and for holistic treatment planning. Segment by Class provides detailed sectional views based on specific classes, supporting targeted disease analysis or organ mapping, such as tumor identification in critical organs.

This model is for research and development only.

Github links:

https://github.com/NVIDIA-Medtech/NV-Segment-CTMR/tree/main

Run pipeline:

For running the pipeline, NV-Segment-CTMR requires at least one prompt for segmentation. It supports label prompt, which is the index of the class for automatic segmentation. NV-Segment-CTMR does not support point based interactive segmentation. For interactive model, please refer to VISTA3D

Here is a code snippet to showcase how to execute inference with this model.

import os
import tempfile

import torch
from hugging_face_pipeline import HuggingFacePipelineHelper


FILE_PATH = os.path.dirname(__file__)
with tempfile.TemporaryDirectory() as tmp_dir:
    output_dir = os.path.join(tmp_dir, "output_dir")
    pipeline_helper = HuggingFacePipelineHelper("vista3d")
    pipeline = pipeline_helper.init_pipeline(
        os.path.join(FILE_PATH, "vista3d_pretrained_model"),
        device=torch.device("cuda:0"),
    )
    inputs = [
        {
            "image": "/data/Task09_Spleen/imagesTs/spleen_1.nii.gz",
            "label_prompt": [3],
        },
        {
            "image": "/data/Task09_Spleen/imagesTs/spleen_11.nii.gz",
            "modality": 'CT_BODY',
        },
        {
            "image": "/data/Task09_Spleen/imagesTs/spleen_11.nii.gz",
            "modality": 'MRI_BODY'
        },
    ]
    pipeline(inputs, output_dir=output_dir)

The inputs defines the image to segment and the prompt for segmentation.

inputs = {'image': '/data/Task09_Spleen/imagesTs/spleen_15.nii.gz', 'label_prompt':[1]}

• The inputs must include the key image which contain the absolute path to the nii image file, and includes prompt keys of label_prompt.
• The label_prompt is a list of length B, which can perform B foreground objects segmentation, e.g. [2,3,4,5]. The full list of label definition is in metadata.json.
• If no prompt is provided, user can use modality to use predefined class indices. Supported modality includes CT_BODY, MRI_BODY, MRI_BRAIN.

Note: For brain structure segmentation, current model only support standard brain T1 images. The brain T1 images must be preprocessed with skull stripping and normalization. Follow https://github.com/junyuchen245/MIR/tree/main/tutorials/brain_MRI_preprocessing to process the brain images

License/Terms of Use:

NVIDIA OneWay Non-Commercial License for academic research purposes

Deployment Geography:

Global

Use Case:

Medical researchers, AI developers, and healthcare institutions are expected to use this system to perform automated medical image segmentation, conduct multi-organ analysis, and accelerate annotation workflows in research applications.

Release Date:

Huggingface: 10/27/2025 via https://huggingface.co/NVIDIA

Reference(s):

[1] He, Yufan, et al. "VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imaging." arXiv preprint arXiv:2406.05285. 2024. https://arxiv.org/abs/2406.05285

Model Architecture:

Architecture Type: Transformer Network Architecture: SAM-like architecture for 3D medical imaging segmentation

This model was developed from scratch using MONAI components. Number of model parameters: 218M

Input:

Input Type(s): Image Input Format(s): Neuroimaging Informatics Technology Initiative (NIfTI) Input Parameters: Three-Dimensional (3D) Other Properties Related to Input: Supports both computed tomography (CT) and magnetic resonance (MR) imaging modalities. It also supports optional class information for targeted segmentation workflows.

Input Modalities:

• CT Images: 3D computed tomography volumes
• MR Images: 3D magnetic resonance volumes
• Class Selection: Optional class indices for targeted segmentation workflows

Output:

Output Type(s): Image Output Format: Neuroimaging Informatics Technology Initiative (NIfTI) Output Parameters: Three-Dimensional (3D) Other Properties Related to Output: Segmentation masks with up to 345+ anatomical classes, providing comprehensive organ and tissue delineation for medical imaging analysis.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (GPU cores) and software frameworks (CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine(s):

• MONAI Core v.1.5.0

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
• NVIDIA Hopper

Supported Operating System(s):

• Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

0.1 - Initial release version for 3D medical imaging segmentation with multi-modality support

Training, Testing, and Evaluation Datasets:

Dataset Overview:

Total Size: ~31k Total Number of Datasets: 32 datasets

Public datasets from multiple scanner types were processed to create standardized 3D medical imaging volumes with expert-validated anatomical segmentation masks across diverse anatomical regions and pathological conditions. The data processing pipeline ensured consistent voxel spacing, standardized orientations, and validated anatomical segmentations.

Training Dataset:

Data Modality:

• Image

Image Training Data Size:

• Less than a Million Images

Data Collection Method by dataset:

• Hybrid: Human, Automatic/Sensors

Labeling Method by dataset:

• Hybrid: Human, Automatic/Sensors

Testing Dataset:

Data Collection Method by dataset:

• Hybrid: Human, Automatic/Sensors

Labeling Method by dataset:

• Hybrid: Human, Automatic/Sensors

Evaluation Dataset:

Data Collection Method by dataset:

• Hybrid: Human, Automatic/Sensors

Labeling Method by dataset:

• Hybrid: Human, Automatic/Sensors

Inference:

Acceleration Engine: PyTorch Test Hardware:

• A100
• H100

Additional Information:

Available Anatomical Classes (345+ total):

NV-Segment-CTMR supports comprehensive anatomical segmentation with the following categories:

Core Organs and Systems:

• Abdominal organs: liver (1), kidney (2), spleen (3), pancreas (4), gallbladder (10), stomach (12), bladder (15), colon (62)
• Cardiovascular: heart (115), aorta (6), inferior vena cava (7), superior vena cava (125), portal and splenic veins (17)
• Respiratory: lung (20), trachea (57), airway (132), individual lung lobes (28-32)
• Neurological: brain (22), spinal cord (121), complete brain structures (214-345)

Skeletal System:

• Spine: Complete vertebral column from C1-S1 (33-56, 127)
• Thoracic: Bilateral ribs 1-12 (63-86), sternum (122), costal cartilages (114)
• Appendicular: Bilateral long bones, joints, and extremities (87-96)

Detailed Brain Segmentation: Comprehensive brain parcellation including ventricles, cortical regions, subcortical structures, and specialized brain areas (214-345) based on neuroanatomical atlases.

Pathological Structures:

• Tumors: lung tumor (23), pancreatic tumor (24), hepatic tumor (26), brain tumor (176)
• Cancer: colon cancer primaries (27)
• Lesions: bone lesion (128)
• Cysts: bilateral kidney cysts (116-117)
• Note: We recommend the NV-Segment-CT model for better tumor performance.

Specialized Regions:

• Head and neck: detailed facial structures, sensory organs, and cranial anatomy (172-213)
• Cardiac: heart chambers, major vessels, and cardiac-specific structures (108, 149-155)
• Reproductive: prostate zones (118, 147-148), uterocervix (161), gonads (160)

Complete numerical mapping and deprecated classes available in model documentation.

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.

Please report model quality, risk, security vulnerabilities or concerns here.