Human Surgeome: Building a Foundation Model for Surgical Video

DKFZ and Aimbient are training a vision foundation model on 26,755 hours of real procedural video and testing it on more than 50 clinical tasks ahead of release.

In laparoscopic surgery, the camera is the surgeon's eye, so every operation is recorded by design. Those recordings hold moments worth learning from: a bleed brought under control, a difficult dissection handled with expert skill, an unusual technique that rescued a hard case. They sit in hospital archives by the thousands, almost all of them unannotated and out of reach for AI. Human Surgeome asks what becomes possible when a model learns from that footage directly, across procedure types and hospitals.

The project brings together the Division of Intelligent Medical Systems at the German Cancer Research Center (DKFZ) and Aimbient. Its goal is a general visual backbone for surgery: one model that many clinical applications can build on, tested broadly enough that the teams building on it know where it can be trusted.

Why surgery needs its own foundation model

Surgical AI has mostly been built one task at a time. A phase recognition model, an instrument detector and a bleeding classifier each required their own expert annotations, and annotation time from surgeons is scarce and expensive.

Foundation models change that economics. A model pretrained with self-supervised learning extracts general visual features from unlabeled data, and each downstream task then needs only a modest amount of labeled data on top.

Surgical video has its own visual language: smoke, blood, specular reflections from wet tissue, fast camera motion and instruments that partly hide the anatomy. A model that has seen millions of surgical frames learns these regularities directly, which gives every application built on it a stronger starting point.

A partnership between model builders and model users

The work divides along each partner's strengths.

DKFZ leads model development. The Division of Intelligent Medical Systems, headed by Prof. Lena Maier-Hein, works on surgical data science and on the trustworthy validation of medical AI. The team designs the pretraining pipeline, trains the models and runs the evaluation. Dr. Patrick Godau co-leads the project on the DKFZ side.

Aimbient contributes clinical use cases and validation. Its clinical and data science teams define the tasks that matter in practice and help test the models against them. On the Aimbient side, the project is supported by Teodor Grantcharov, MD, PhD, founder and CEO, and Peter Grantcharov, Chief Technology Officer.

"A foundation model is only as useful as the evidence behind it. We test Human Surgeome on more than 50 medical tasks that remain hidden throughout development, including data from independent institutions, so that everyone building on it knows where it works."
Lena Maier-Hein, DKFZ

How the model learns

Human Surgeome is a frame based vision model: it learns from individual video frames. It is pretrained with a DINO variant, a self-supervised method introduced by researchers at Meta AI and Inria, with data augmentations adapted to the surgical domain.

The idea is intuitive. Two copies of the network, a student and a teacher, look at different crops of the same frame. The student learns to produce the same representation as the teacher, while the teacher slowly follows the student. To succeed, the model has to learn what stays constant across views: which structure is the gallbladder, which object is a grasper, how tissue typically behaves. No human labels are needed at any point.

The same teacher and student idea builds the smaller models in the family. The trained flagship model acts as a fixed teacher, and each compact model learns to reproduce its features on the same surgical frames. The small models inherit much of what the large one has learned at a fraction of its size and computing cost, which makes them practical on the hardware hospitals already run.

The models learned from more than 50 million frames, drawn from a curated pool of more than 13,500 interventions with a total duration of 26,755 hours. Out of body and other uninformative frames were removed before sampling at one frame per second, and about 1,000 further videos were excluded during curation to protect data quality. Pretraining was enabled by a compute grant of 250,000 GPU hours on the JUWELS Booster supercomputer at Jülich Supercomputing Centre.

Validation at scale

Validation is where Human Surgeome invests most of its effort. A foundation model promises that one set of features serves many tasks, and that promise has to be checked task by task. The final evaluation covers more than 50 downstream tasks, each kept hidden from the team for the entire development process.

Several design choices make the evaluation demanding:

• Hidden until the end. A separate set of 30 tasks guided all development decisions. The final test tasks are evaluated only once the models are complete, and overlapping videos were carefully avoided so that no data leaks between development and testing.

• Independent external data. The final evaluation includes private datasets from external institutions, which tests how the models perform on new hospitals and new equipment.

• Strong baselines. Results are compared with both general and surgical frame based foundation models.

• Fit for purpose metrics. Metrics are selected per task following the Metrics Reloaded recommendations.

• Frozen and adapted use. Each model is tested as a frozen feature extractor and after adaptation, in particular with parameter-efficient fine-tuning (PEFT), which mirrors how developers will build on it.

• Real deployment tasks. The evaluation includes three tasks drawn from Aimbient's deployed products: bleeding detection, de-identification and instrument detection.

From benchmark to bedside

The partnership connects a team at the frontier of surgical AI research with a team that puts models in front of surgeons every day. Aimbient's platform runs in 650 procedural rooms across North America and Europe, and that deployment shapes the questions the project asks.

Model size is one of them. Human Surgeome will be released as a family of Vision Transformers, from tiny variants with a few million parameters up to Surgeome-L with about 300 million. The large model sets the ceiling for accuracy, and the smaller ones bring strong features to hardware hospitals already run, at a cost per case that scales to thousands of rooms.

The measure of success is value in the room: faster case review, earlier detection of safety events, less manual documentation, and a lower cost for every new application built on a shared backbone.

What comes next

The project is nearing completion, and the models will be released in early 2027 via Aimbient. Peer-reviewed publications describing the method and the full validation are in preparation. The work continues on the JUPITER Booster at Jülich Supercomputing Centre.

Explore Aimbient's Surgical AI Frontiers and Research Library, or contact the team to collaborate.

Acknowledgements

The authors gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for funding this project by providing computing time through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS Booster at Jülich Supercomputing Centre (JSC).