Curious about what keeps experts, CEOs and other decision-makers in the Intelligent Document Processing (IDP) space on their toes? Get food for thought on IDP-related topics from the industry’s leading minds.
In this opinion piece, Andrew Bird, Head of AI at Intelligent Document Processing (IDP) vendor Affinda, explains the evolution of document processing, from rules to machine learning to agentic systems, and the stages within.
“Applying AI agents to simple tasks is like hiring a Michelin-star chef to microwave leftovers” – this was an argument by a major IDP vendor to suggest LLMs don’t belong in document automation software. It’s a clever analogy, but one that elicits a compelling counterargument. If the chef only costs you a few cents per meal, why wouldn’t you hire them?
Inference costs have dropped so dramatically that using powerful LLMs for routine document tasks isn’t wasteful – it’s practical. The economics have fundamentally shifted, but many IDP providers haven’t. They’re still raising concerns about hallucinations, defending architectures that are rapidly becoming legacy systems while their competitors move ahead. Understanding this progression – from rules to machine learning to agentic systems, and the stages within – reveals who’s adapting and who’s standing still.
The Evolution of Document Processing
Document processing has moved through three distinct phases. It started with Template IDP (Rules era) where you tell it where to look. You define anchors, set up regex patterns, and map out fixed zones on the page. It assumes that if Vendor A’s invoice looked a certain way last month, it’ll look the same this month. OCR reads the text, the system parses the layout, then executes your rules. Confidence scores come from simple heuristics about how well the match worked.
The trouble starts when layouts drift. For example, a vendor refreshes their branding, fields move around, or you need to handle documents in a new language. Each exception means writing another rule, and before long you’re drowning in rule debt that grows faster than you can manage it. The system struggles with messy scans, has difficulty with complex cross-page logic, and every new vendor or field type means another round of rules and testing.
Next came Machine Learning IDP (Transformer Era) which uses transformer models fine-tuned for specific document types. Instead of you telling the system where to look, the model learns to classify each word. Post-processing logic then assembles these labelled words into the final extracted values. The models learned visual and linguistic patterns that worked across different vendors and layouts. They handled noisy scans better, worked with multiple languages, and needed far fewer brittle rules than the template approach.
The challenge is in the extensive operational weight, with each use case needing its own fine-tuned model. And when users make corrections, the model doesn’t improve immediately, resulting in a slower feedback loop.
The latest shift to Agentic Document Processing (Generative Era) moves the extraction itself into a generative model using LLMs. Instead of labelling tokens that point to an invoice number, the system generates the invoice number. Then agents verify that output against the source document and business rules.
One model can now adapt to new document types and fields with just a few examples. Updates are as easy as adjusting context – no long training cycles or annotation sprints. The system coordinates tools for OCR, retrieval, validation, and integration with other systems. Human corrections instantly update the agent’s instructions and memory, improving the next document. The generative approach asks, “What is the invoice number?” and then grounds that answer by linking back to exact regions in the document and cross-checking against business rules.
One critical engineering challenge is preventing hallucination. Every extracted value must carry Provenance – citations to specific page regions. Schema-aware validators check accuracy. When evidence is insufficient, the agent re-reads, retrieves more context, or escalates for human review. These controls make unverified outputs essentially impossible to accept in production.
The payoff is speed and coverage. Standing up a new document type or adding fields takes minutes instead of weeks. The system handles messy PDFs, unfamiliar vendors, and can reason across multiple documents to reconcile invoices with purchase orders and receipts.
The Evolution Within Agentic Document Processing
Even within the generative era, not all agentic systems are created equal. The field is moving through distinct stages of sophistication, each unlocking new capabilities.
Stage One: Prompt-Led Systems – the system bundles field-level instructions into a large prompt, sends it along with the document to the LLM, and gets back structured data. It’s quick to start – no training, no labelling, just editable behaviour in the prompt. For stable document types with predictable schemas, it works well enough. But these systems don’t learn. When you correct a mistake, nothing changes for the next document unless you manually edit the prompt and fixing one thing can break others so you’re constantly A/B testing different prompt wordings rather than improving the system. You know you’re stuck here when your release notes are mostly “updated prompt for field X” and your team spends more time tweaking instruction wording than resolving the root causes of errors.
This brings us to Stage Two: Memory-Augmented System. When someone corrects an extraction or marks one as successful, the system captures that as a structured memory – essentially a mini case study of what was right or wrong, and why. Then, when processing a new document, retrieval pulls the most relevant memories and automatically adds them to the LLM’s context. This changes the dynamic completely. One correction now influences the next similar document immediately, without a human involvement. Different layouts trigger different memories, so the system adapts to each supplier automatically.
The economics improve too. Instead of hosting fleets of fine-tuned models, you’re managing context engineering – the memories, examples, and instructions that shape behaviour. That’s easier and cheaper to scale. You need good retrieval quality, observability, governance and to think about memory drift – old corrections becoming stale. These are manageable engineering problems, not fundamental limitations.
Stage Three: Tool-Orchestrated Agents don’t just read documents – they complete the entire job. An agent coordinates the LLM with a suite of tools to handle what a human would do from start to finish. Critically, these agents go beyond simply flagging exceptions for human review – they take proactive steps to resolve issues autonomously. When a new supplier invoice arrives with no vendor match, the agent can create the vendor record directly in the ERP system if policy allows. If human judgment is needed, it doesn’t just queue a task – it might send a Slack message to the procurement manager with invoice details and a quick approve/reject prompt or even initiate a voice call to walk through the discrepancy and capture the decision in real-time. The system orchestrates the right action at the right time, whether that’s autonomous execution or reaching out through the channels humans actually use.
Just as template systems once felt like magic compared to manual entry, agentic systems will soon feel like the only sensible way to process documents at scale. The winners in this space will be those who treat LLMs as the core engine, not a bolt-on feature, and who design for provenance, control and rapid adaptation from day one, giving customers faster onboarding, higher straight-through processing and workflows that evolve as their business changes. For everyone else, the question will not be “should we move to agentic IDP?” but “how did we wait so long?”
About the Author
Andrew Bird is Head of AI at global IDP provider Affinda where he is responsible for AI technologies for the automation of high-volume document workflows. He was recently named a finalist for AI Software Engineer of the Year at the Australian AI Awards 2025 for his work on Affinda’s “agentic” AI platform.
Click here to find more news from Affinda.
📨Get IDP industry news, distilled into 5 minutes or less, once a week. Delivered straight to your inbox ↓
