LLMStructBench: Why Prompting Beats Model Size for JSON Extraction
Extracting structured information from natural language is a core task in HR, finance, logistics, and e-commerce workflows. Large language models are increasingly used for this kind of ETL work, but their reliability in producing both syntactically valid and semantically correct JSON output is still poorly understood. With LLMStructBench we present an open benchmark designed to close that gap.
We benchmarked 22 open source models across five prompting strategies and added GPT 4o as a closed weight reference. The leaderboard shows no strict monotonic relationship between parameter count and overall score. Gemma3 in its 27B configuration takes the top spot with a composite score of 0.74, narrowly ahead of Llama3.1 70B and Gemma3 12B. GPT 4o offered no clear advantage over the best open source models, which reinforces how competitive deployable open systems have become.
The more interesting result lies in the prompting analysis. The standard P strategy gives the best overall balance of accuracy and validity for well aligned models, but fails catastrophically for some families like Phi. The schema enriched PJ+ strategy is the safest choice for guaranteeing parseable output, especially for small or structurally unreliable models, but it shifts the error profile toward more semantic mistakes. Wrong value errors remain the primary bottleneck across all configurations, which suggests that further gains depend on post generation validation or fine tuning rather than simply scaling up.
In short, the right prompting strategy matters more than raw model size. Matching the prompt design to the model family beats throwing more parameters at the problem.
The full paper is under IEEE submission. The dataset and benchmark code will follow.