AI-based interactive segmentation approaches for annotating biomedical image data

This is a student research project at HTW Berlin by Fiona Büttner, supervised by Erik Rodner and Mario Koddenbrock.

Modern segmentation models such as Segment Anything Model 3 (SAM3) can identify objects not only using points or bounding boxes, but also through text descriptions and visual examples. This enables flexible and open interaction with unknown object classes.

First tests have shown that performance of modern segmentation models is largely determined by the quality and sequence of user interactions. Recent research shows that hybrid prompt strategies achieve significant improvements over purely point-based approaches. Not only does the type of prompt, such as positive or negative points or text descriptions, influence segmentation quality, but so does their sequential combination.

SAM3 allows for incremental refinement of segmentation, where user interactions build upon previous predictions.

Building on the multimodal capabilities of SAM3, this research project investigates efficient prompting strategies for robust segmentation in a biomedical context.