Imaging is a critical technique in biology—from identifying cancerous cells in biopsies to observing how immune cells like macrophages hunt down and destroy pathogens. Traditionally, distinguishing and labeling individual cells in images and videos has been an arduous task done by hand. Now, an interdisciplinary team of Caltech researchers has developed an artificial intelligence algorithm to identify cells in images for a wide variety of biological applications.
The new tool, called CellSAM (Cell Segment Anything Model), is the result of a collaboration between the laboratories of David Van Valen (PhD '11), an assistant professor of biology and biological engineering, Heritage Medical Research Institute Investigator, and Howard Hughes Medical Institute (HHMI) Freeman Hrabowski Scholar; and Yisong Yue, professor of computing and mathematical sciences. A paper describing the research appears in the journal Nature Methods.
"Before, students would spend countless hours identifying cells by hand or fixing an algorithm's mistakes," Van Valen says. "Now, our single model can do that work for you in many different applications. I'm really excited to see how this method pushes the frontier of biological discovery. There's a lot of fascinating, interesting data that we can now collect and the previous hurdles to getting insights from those data are slowly being knocked down one at a time."
Biological images can look vastly different, revealing phenomena such as tumor cells hidden among tissues and bacteria that secrete sticky antibiotic-resistant goo. Additionally, technological advances are ushering in an era of big data for biology. CellSAM is the first model that can be applied to myriad different use cases, allowing researchers to identify different cell types, and to see where they are located and how they are interacting with their neighbors. Characterizing these complex dynamics is critical for understanding situations such as why a certain cancer immunotherapy may work for one person but not for another.
The CellSAM algorithm was trained on vast amounts of biological images that had been labeled by hand. The team aims to continue improving CellSAM by continuing to train it on more types of biological data. The tool is currently available for researchers to use for free.
"Approaches like CellSAM don't just make existing image-analysis workflows more efficient—they make it possible to explore biological questions at scales that used to be impractical," Yue says. "When you can track millions of cells across many conditions, you can start probing things like how rare cell states appear or how subtle changes in cell shape relate to treatment response. These are the kinds of insights that become accessible only when the bottlenecks in analysis are removed.
The paper is titled "CellSAM: A Foundation Model for Cell Segmentation." Caltech postdoctoral scholar Uriah Israel and former postdoctoral scholar Markus Marks, now at Meta, are the study's first authors. Additional Caltech co-authors are graduate students Rohit Dilip, Qilin Li, and Changhua Yu; former graduate student Emily Laubscher (PhD '24); former undergraduates Ahamed Iqbal (BS '24), Elora Pradhan (BS '22), and Shenyi Li (BS '23); data technicians Ada Ates, Martin Abt, and Caitlin Brown; staff scientist Edward Pao; software engineer Alexander Pearson-Goulart; Pietro Perona, the Allen E. Puckett Professor of Electrical Engineering and director of Information Science and Technology; Georgia Gkioxari, an assistant professor of computing and mathematical Sciences and electrical engineering and a William H. Hurt Scholar; and software engineer Ross Barnowski.
Funding was provided by the Shurl and Kay Curci Foundation, the Rita Allen Foundation, the Susan E. Riley Foundation, the Pew-Stewart Scholars for Cancer Research program, the Gordon and Betty Moore Foundation, the Schmidt Academy for Software Engineering, the Michael J. Fox Foundation through the Aligning Science Across Parkinson's initiative, the Heritage Medical Research Institute, the National Institutes of Health, the HHMI Freeman Hrabowski Scholars program, the Tianqiao and Chrissy Chen Institute for Neuroscience at Caltech, The Emerald Foundation, Black in Cancer, and the Caltech Presidential Postdoctoral Fellows Program. Yisong Yue is an affiliated faculty member with the Tianqiao and Chrissy Chen Institute for Neuroscience at Caltech.
CellSAM model results on microscope images. Individual cells are marked in their own distinct color.
Credit: Caltech
