Picking a doc AI mannequin is tough. Every vendor claims 95%+ accuracy. General-purpose benchmarks take a look at reasoning and code, not whether or not a mannequin can extract a fancy desk from a scanned bill.
So we constructed the Intelligent Document Processing (IDP) Leaderboard.
3 open benchmarks. 16+ fashions. 9,000+ actual paperwork. The duties that matter: OCR, desk extraction, key info extraction, visible QA, and lengthy doc understanding.
The level is not to offer you one quantity and declare a winner. It’s to allow you to dig into the specifics. See the place every mannequin is robust, the place it breaks, and resolve for your self which one matches your paperwork.
The outcomes shocked us. The #7 mannequin scores increased than #1 on one benchmark. Sonnet beats Opus. Nanonets OCR2+ matches frontier fashions at lower than half of the price.
Why 3 benchmarks?
Every benchmark measures one thing totally different. Use one and also you solely see one dimension. So we used three.
IDP Core: Can you extract what a enterprise truly wants? This one is ours. Invoices, handwritten textual content, ChartQA, DocVQA, 20+ web page paperwork, six sorts of tables. The stuff that breaks manufacturing pipelines. These are extra reasoning heavy duties than the opposite two benchmarks.
Each mannequin will get a functionality profile throughout six sub-tasks: textual content extraction, formulation dealing with, desk understanding, visible QA, structure ordering, and key info extraction.
Explore every mannequin’s functionality profile at: idp leaderboard
What the leaderboard truly permits you to do?
Most leaderboards provide you with a desk. You have a look at it. You choose the highest mannequin. You transfer on. It seems like being a by-stander and never hands-on.
We wished one thing extra clear and hands-on than that.
For that we created the Results Explorer that allows you to see precise predictions and evaluate fashions on actual paperwork. For any doc within the benchmark, you see the bottom fact subsequent to each mannequin’s uncooked output. This makes you see and evaluate the use-cases that is related to you.
This is highly effective because it additionally makes you query the bottom fact and offers you the total image of what is going on behind the scenes of every benchmark activity.
You can see precisely the place it hallucinated a desk cell or missed a handwritten phrase. Here’s an instance exhibiting how fashions deal with complicated formulation extraction.
1v1 Compare places two fashions facet by facet throughout all six functionality dimensions.
How did we run it?
We wished anybody to have the ability to run all three benchmarks. So we made setup as near zero as we might.
Everything pulls from HuggingFace. We pre-rendered all PDFs to PNGs and hosted them at shhdwi/olmocr-pre-rendered so you do not want a conversion pipeline. IDP Core embeds pictures straight within the dataset. Nothing to clone your self or unzip.
The runner works with any mannequin that has an API. Failed runs choose up the place they left off.
Here’s the Github repo hyperlink to strive it your self: IDP Benchmarking repo
Here’s what stood out.
Gemini 3.1 Pro dominates VQA duties
Gemini 3.1 scores 85 in VQA, properly above some other mannequin. Closest to it’s GPT-5.4 at 78.2. Rest all fashions are in 60’s.
This can also be seen within the newest benchmarks launched by Google. Gemini 3.1 professional is best at reasoning duties. Same holds true for Document VQA duties as properly.
Cheaper fashions are surprisingly good
This stored arising.
- Sonnet 4.6 (80.8) is pretty much as good as Claude 4.6 (80.3)
- Gemini-3 flash matches Gemini-3 professional and typically even higher (in Omnidoc bench)
Cost modifications the maths
Here’s the half that issues in the event you’re processing paperwork at any actual quantity.
The Nanonets OCR2+ mannequin is a good stability for each accuracy and price in relation to scale. Click right here for the mannequin’s full profile
Where issues nonetheless break!
Sparse, unstructured tables stay the toughest extraction activity.
Most fashions land under 55%. These are tables the place cells are scattered, many are empty, and there are not any gridlines to information the mannequin. Only Gemini 3.1 Pro and GPT-5.4 persistently deal with them at 94% and 87% respectively, nonetheless properly under their 96%+ on dense structured tables
Click Here to verify the Gemini 3.1-pro outputs on lengthy sparse docs
Here’s how different fashions break
Handwriting OCR hasn’t crossed 76%. The finest mannequin is Gemini 3.1 Pro at 75.5%. Digital printed OCR is 98%+ for frontier fashions. Handwriting is a essentially totally different downside and no mannequin has cracked it.
Handwritten type extraction hallucinates on clean fields. Every mannequin clusters between 80-84% on this activity. The failure mode is constant: fashions fill in values for fields which are clean on the shape. A reputation, a date, a standing that does not exist within the doc.
Gemini > Claude = OpenAI
The pecking order was settled. Gemini led, Claude adopted, OpenAI trailed. GPT-4.1 scored 70.0. Nobody was choosing OpenAI for doc work.
For GPT-5.4 Table extraction went from 73.1 to 94.8. DocVQA went from 42.1% to 91.1%. GPT-5.4 acquired higher at understanding paperwork and reasoning.
The total scores are actually 83.2, 81.0, 80.8. Close sufficient that the rating issues lower than the form. Claude leads on formulation. GPT-5.4 leads on tables and QA. Gemini leads on OCR and VQA.
One factor price noting: Claude fashions had stricter content material moderation that affected sure paperwork. Old newspaper scans, textbook pages, and historic paperwork typically triggered filters. This harm Claude’s scores (solely in OmniDoc and OlmOCR).
Now, Which Model Should you choose?
Every vendor will inform you their mannequin is 95%+ correct. On structured tables and printed textual content, they is likely to be proper. On sparse tables, handwritten varieties, and 20-page contracts, most fashions battle.
Running a high-volume OCR pipeline? Nanonets OCR2+ provides you top-tier accuracy at $10 per thousand pages.
Processing complicated tables or want excessive accuracy on reasoning over paperwork? Gemini 3.1 Pro is well worth the premium at $28/1K pages.
Building a easy extraction workflow on a finances? Sonnet and Flash match their costly siblings on extraction duties. Nanonets OCR2+ matches right here too, robust accuracy with out the frontier price ticket.
But do not take our phrase for it. The leaderboard has the scores. The Results Explorer has the precise predictions. Pick a activity that matches your workload. Look at what they output on actual paperwork. Then resolve.
What’s subsequent
We can be including extra open-source fashions and doc processing pipeline libraries to the leaderboard quickly. If you need a particular mannequin evaluated, request it on GitHub.
We’ll preserve refreshing datasets too. Benchmarks that by no means change turn out to be targets for overfitting.
The leaderboard is at idp-leaderboard.org. The Results are open. The code is open. Go have a look at what these fashions truly do together with your sorts of paperwork. The numbers inform one story. The Results Explorer tells a extra trustworthy one.
