• Code
  • Dataset
  • Demo
  • Model

SCAM

Real-world typographic attacks on multimodal foundation models

The same wall clock three times: with a handwritten sticky note reading TAXI (SCAM), with a blank note (NoSCAM), and with the word taxi inserted synthetically (SynthSCAM)

The largest dataset of real-world typographic attack images, with handwritten notes that try to mislead vision-language models, plus clean and synthetic versions of every image.

A typographic attack puts misleading text into an image, for example a sticky note reading "car" on a ball, and makes a vision-language model see the written word instead of the object. SCAM collects 1,162 such photos taken in the real world, covering 660 object labels and 206 attack words.

Every image comes in three versions: SCAM, the real photo with the handwritten attack note; NoSCAM, the same photo with the note removed, as a clean baseline; and SynthSCAM, the clean photo with the attack word inserted synthetically.

In our benchmark, typographic attacks lower the accuracy of vision-language models by 26% on average and by up to 50% for smaller models. Large vision-language models inherit the weakness from their vision encoders, and larger language model backbones partly compensate for it. Synthetic attacks behave much like real ones, so they are a valid stand-in for research.

from datasets import load_dataset

ds = load_dataset("BLISS-e-V/SCAM", split="train")

The dataset is released under CC BY-NC 4.0.

Other Datasets which might be of interest

  • Three rows, each with a binary microtubule mask, the real IRM recording, and the image DiffuMT generated from the mask

    DiffuMT Dataset

    Triplets of a binary mask, the real IRM recording and the image DiffuMT generated from that mask, from our NLDL 2027 paper.
    Read more
  • Two real IRM microtubule images in the top row and the images DiffuMT generated from their masks in the bottom row

    DiffuMT

    A diffusion model that turns a segmentation mask into a realistic IRM microtubule image, for generating labeled training data.
    Read more

  • Redakto

    Everybody wants to paste text into LLMs - but some texts contains sensitive information. To protect your privacy and enable you to use LLMs, we developed and evaluated lightweight transformer models to redact German texts.
    Read more

  • From Heat to Force

    We conducted a comprehensive study to investigate the relationship between the contact forces exerted by the fingers and the thermal signature that people leave behind when interacting with objects. The result is a substantial dataset consisting of 703 thermal imaging videos and precise force measurements from 27 participants.
    Read more

  • SynTabFall

    In collaboration with Charité Berlin we released a synthetic data set for training AI fall risk assessement models. We trained a number of tabular genAI model classes on ~1Mio patient records of Charité Berlin and optimised the models with respect to fidelity, utility and privacy, in close collaboration with data protection officers.
    Read more
  • A synthetic microtubule image, its ground-truth instance masks, and predictions by FIESTA and SAM3Text

    SynthMT

    A synthetic benchmark for microtubule segmentation with pixel-perfect instance masks, plus real IRM images in the same format, from our paper in PLOS Computational Biology.
    Read more
  • Chart: model performance on RamanBench against the model's release date, from PLS to TabPFN

    raman-bench

    The Python package behind RamanBench, our NeurIPS 2026 benchmark. It loads 74 Raman datasets with fixed splits and scores your model against 28 baselines, without re-running them.
    Read more
  • Example Raman spectra from material science, biotechnology, medicine and chemistry

    raman-data

    A Python package that lists, downloads and loads public Raman spectroscopy datasets as arrays, from Kaggle, Hugging Face, Zenodo and other archives. It provides the datasets for RamanBench.
    Read more
  • Fluorescence microscopy images of cell nuclei from the Broad Bioimage Benchmark Collection, and a colour-coded segmentation of one of them

    bbbc-datasets

    A Python package that downloads the segmentation datasets of the Broad Bioimage Benchmark Collection (BBBC) and loads them as PyTorch datasets, in 2D and 3D.
    Read more