Healthcare AI Models

Should you have any questions, please do not hesitate to reach out. Simply fill out the contact form: https://aka.ms/healthcare-ai-request-external (opens in new tab)

The Microsoft healthcare AI models are intended for research and model development exploration. The models are not designed or intended to be deployed in clinical settings as-is nor for use in the diagnosis or treatment of any health or medical condition, and the individual models’ performances for such purposes have not been established. Developers bear sole responsibility and liability for any use of the healthcare AI models, including verification of outputs and incorporation into any product or service intended for a medical purpose or to inform clinical decision-making, compliance with applicable healthcare laws and regulations, and obtaining any necessary clearances or approvals.

AI Foundry Model Catalog

Filtered to Health and Life Sciences: https://aka.ms/health-life-sciences (opens in new tab).

Along with deployment access, you can find details on Model Architecture, License, Training Information, Evaluation Results, Sample Inputs/Outputs, Data and Resource Specs for Deployment, and HW Requirements.

GitHub Samples Repository

Designed to help you get started with Microsoft’s healthcare AI models. Whether you are a researcher, data scientist, or developer, you will find a variety of examples and solution templates that showcase how to leverage these powerful models for different healthcare scenarios. From basic deployment and usage patterns to advanced solutions addressing real-world medical problems, this repository aims to provide you with the tools and knowledge to build and implement healthcare AI solutions using Microsoft AI ecosystem effectively: https://github.com/microsoft/healthcareai-examples/ (opens in new tab)

Here’s a quick look at what you’ll find:

Basic Usage Examples and Patterns:

MedImageParse call patterns (opens in new tab) – a collection of snippets showcasing how to send various image types to MedImageParse and retrieve segmentation masks. See how to read and package xrays, ophthalmology images, CT scans, pathology patches, and more.
Zero shot classification with MedImageInsight (opens in new tab) – learn how to use MedImageInsight to perform zero-shot classification of medical images using its text or image encoding abilities.
Training adapters using MedImageInsight (opens in new tab) – build on top of zero shot pattern and learn how to train simple task adapters for MedImageInsight to create classification models out of this powerful image encoder. For additional thoughts on when you would use this and the zero shot patterns as well as considerations on fine tuning, read our blog on Microsoft Techcommunity Hub.
Advanced calling patterns (opens in new tab) – no production implementation is complete without understanding how to deal with concurrent calls, batches, efficient image preprocessing, and deep understanding of parallelism. This notebook contains snippets that will help you write more efficient code to build your cloud-based healthcare AI systems.

Advanced Examples and Solution Templates

Detecting outliers in MedImageInsight (opens in new tab) – go beyond encoding single image instances and learn how to use MedImageInsight to encode CT/MR series and studies and detect outliers in image collections.
Exam Parameter Detection (opens in new tab) – dealing with entire MRI imaging series, this notebook explores an approach to a common problem in radiological imaging – normalizing and understanding image acquisition parameters. Surprisingly (or not), in many cases DICOM metadata cannot be relied upon to retrieve exam parameters. Look inside this notebook to understand how you can build a computationally efficient exam parameter detection system using an embedding model like MedImageInsight.
Multimodal image analysis using radiology and pathology imaging (opens in new tab) – can foundational models be connected to build systems that understand multiple modalities? This notebook shows a way this can be done using the problem of predicting cancer hazard score via a combination of MRI studies and digital pathology slides. Also read our blog (opens in new tab) that goes into more depth on this topic.
Image Search Series Pt 1: Searching for similar XRay images (opens in new tab) – an opener in the series on image-based search. How do you use foundation models to build an efficient system to look up similar Xrays? Read our blog (opens in new tab) for more details.
Image Search Series Pt 2: 3D Image Search with MedImageInsight (MI2) (opens in new tab) – expanding on the image-based search topics we look at 3D images. How do you use foundation models to build a system to search the archive of CT scans for those with similar lesions in the pancreas? Read our blog (opens in new tab) for more details.

Research Papers

Nature Publications:

Benchmarking:

ReXrank Chest X-ray Report Generation Leaderboard (hosted on Azure; MAIRA-2/CXRReportGen is the MSFT model) (opens in new tab)