Traditional “template-based” solutions such as Optical Character Recognition (OCR) are no longer sufficient to meet the needs of growing enterprises. Mostly because they are not as dynamic as Intelligent Document Processing (IDP) solutions.
“Information and documents change all the time, so your intelligent document processing (IDP) software must learn and adapt with it.“
Niels Van Weereld, CCO Metamaze
As a result, the demand for Intelligent Document Processing (IDP) solutions is growing strongly. It is already reported that the market of IDP solutions already grew at 30% and will continue to grow in the coming years at an estimate of 60-70% CAGR, despite the current pandemic.
While the current covid19 crisis has negatively impacted growth, it has painfully exposed the need for such software in many companies. As Anil Vijayan, VP at Everest Group states:
“Post-COVID-19, enterprises will want to automate as much as possible to reduce dependence on the human workforce and to cut costs. Automated extraction of data using IDP solutions will enable organisations to process large volumes of semi-structured and unstructured documents with low-touch processing, greater accuracy, and faster turnaround times, thus helping deal with volume fluctuations as well.”
Anil Vijayan, VP at Everest Group
Apart from the motivations that arose during this period (e.g. the need for continuity and stability of a back office when working from home), there are four main reasons why companies start with or switch to IDP platforms. The following is a first introduction to these reasons, based on research and our daily interaction with different industries at Metamaze. If you would like to elaborate on this or recognize yourself in certain situations, do not hesitate to contact us via the contact details at the bottom.
There’s a rising need
The amount of documents grow
When companies grow, so does the number of customers, processes and ultimately incoming documents. Enterprises are confronted with larger volumes of semi- and unstructured documents, which should be processed quicker and more accurate than before to remain competitive.
Although many organizations aim to digitize certain processes, in practice we see that this turnaround often does not exist or happens slowly. For example: Bank customers are still forwarding a multitude of KYC documents in PDF form; claims are filed with a mountain of documents attached; and an enormous number of companies still receive their orders and invoices on PDF via email.
Time is money
Processing these large quantities of documents manually takes an enormous amount of time.
Simply imagine the workload office workers are facing daily. They must first know where to search for the document: is it in a folder somewhere? Or is it still in an e-mail box?
Then, based on the document they found, it must be correctly classified. Only then can all information from the document be retyped and validated within a second software package.
Studies show that for a ‘simple’ document type such as an invoice, the accumulated time spent on these repetitive tasks can take up to 2 minutes per document. And because a person will handle this document processing flow sequentially, you can multiply this time by the number of incoming invoices. Soon you will have at least 1 full-time ‘Document processor’ on your payroll, even at small / medium-sized organisations.
So if a “simple” document type wreaks that much havoc, just imagine how much time is wasted each year at companies that process hundreds of thousands or even millions of pages of more complex documents. To put it with the biggest cliché statement ever:
“Time is money”
Proverb
Many companies are still throwing both out by manually processing their documents.
Customer is king
As is often the case with digital transformation initiatives, the need is also largely driven by customer centricity. Customers (and suppliers) expect a certain speed in their interactions with companies, which is often hindered by the slow manual processing of files, requests or documents per se. Companies that do not automate nor innovate in this area, struggle to meet these expectations.
Not only are the expectations of customers high, the standard in terms of technology is also a lot higher than a few years ago. This means that in addition to speed, accuracy is also becoming essential when processing documents. Not only should the largest possible part of the incoming documents be fully automated (STP), the information extracted from the documents must be correct and as complete as possible. Because in the end, accuracy is what matters most as the foundation of a solid business-client relationship.
In summary, the amount of time spent on processing documents are growing whilst initiatives to digitise parts of these processes are lagging behind. In the meanwhile customers are expecting faster and accurate turnaround. The need for an Intelligent Document Processing (IDP) solution is therefore growing noticeably.
Complementary Technology
In combination with IDP platforms, companies are rapidly adopting complementary technologies such as Robotic Process Automation (RPA) that enables them to automate document processes from end to end. Companies that are already combining RPA software with IDP, are able to achieve unprecedented levels of automation, also known as hyperautomation.
In 2020, Gartner published an overview of the top 10 cost optimisation techniques for CIOs. The following was found at number 7:
“Implement robotic process automation: Leverage technologies like robotic process automation (RPA) and artificial intelligence (AI) to analyze and identify patterns that can then be used to make strategic business decisions. These technologies can identify new insights from data and automate or augment processes that are inefficient for humans to perform.”
Top 10 Cost Optimization Techniques for CIO’s, Gartner 2020
For example, a medium-sized company today can achieve a clear ROI by automating its accounts payable process from end-to-end using this combination of technologies. A robot will retrieve the documents in the correct (e-mail) folder, feed them to the IDP solution for classification / information extraction and then automatically enter them into the accounting application based on the recognised fields. At Metamaze we often use the following human metaphor for this hyperautomation combination:
“Robotic Process Automation can be seen as a set of handy, tireless arms. Intelligent Document Processing software is the brain that is able to really interpret things.”
Jo Cijnsmans, BDM at Metamaze
Technological Advances
Significant technological breakthroughs have happened in the field of AI, resulting in increased accuracy of IDP solutions and an even wider performance gap with traditional template-based OCR technologies.
Beyond OCR
Concretely, OCR ensures that scanned images are converted into text by transcribing each character on a document. The end result is a machine-readable text from which AI (eg. Natural Language Processing or NLP) can intelligently extract the correct information.
So, although OCR technology has been around for decades, it is becoming merely one part of a modern high-performance document processing solution.
Intelligent Document Processing (IDP) software augments the latest OCR technology with multiple domains in AI (such as computer vision). This gives users the unique capabilities to extract, understand ànd process entire documents (including text, dates, currencies, tables, signatures, stamps, page numbers and much more). In addition to OCR and AI, most leading IDP platforms also use image enhancement technology to increase the quality of scans (including contrast enhancement, noise reduction and anti-skewing techniques).
NLP as a game changer
The introduction of Natural Language Processing (NLP) technology was a huge game changer for straight-through-processing (STP) accuracy.
The deep-learning (or NLP) algorithms in IDP solutions ensure that IDP software can read and understand complex, long and unstructured documents. In layman’s terms: the algorithms learn from context and are flexible in adapting to format changes because they autonomously interpret text and layout, not just memorize it.
Essentially, NLP becomes the brain of your IDP solution. This amazing brain power makes it possible for AI to handle unstructured input like free form text (e.g. documents, scans, text, e-mails, documents, …) with unlimited variety.
In the concrete case study of invoices, NLP solutions will adapt and learn across different suppliers and document types, so that no templates need to be manually created for invoices from new suppliers. All this boils down to: less work for the enterprise user. And thus, faster and more accurate straight-through-processing, for which your bottom line will be the main beneficiary.
AI becomes increasingly smarter
In addition to their capacity to deal with more types of data, IDP solutions also offer the great advantage of becoming smarter over time. While classic ocr or template-based approaches offer a static performance that in many cases levels off around 60-70% automation, this self-learning AI approach often delivers accuracies of 90% + through continuous improvements. The multi-modal deep learning algorithms improve over time through exception validation.
Central to this self-learning approach is the use of annotated examples as training data for these algorithms. By ‘showing’ an algorithm enough examples, it is able to process new incoming documents in comparison to the previous examples. Starting from large, pre-trained basic models, it is possible to fine-tune models on customer-specific data. The main benefit is that it takes less time and effort to get up and running with the highest accuracy possible, yet remain flexible and autonomously adaptive.
Less time and effort needed
Technological breakthroughs have, among other things, ensured that the amount of training data (or examples) that one needs when building an accurate model, has been drastically reduced.
For example, the introduction of optimal document selection techniques has strongly influenced the process of data annotation and model training. In some cases it is possible to build a solid, production-ready model with only a few dozen examples. The combination of a lower entry effort with higher accuracy (read: efficiency gains) will only continue to make the ROI of Machine Learning (ML) based solutions more interesting for cost-efficient organisations.
Success stories
There’s a growing number of success stories by forward-looking companies. The ever increasing status of IDP as a proven, ROI-driving technology is likely to increase acceptance and adoption at other innovative organizations that are looking for a competitive edge.
Certain progressive sectors have been researching and implementing IDP solutions for years already. More specifically, the FSI and healthcare sector, where already more than 50% of IDP adoption is located today. In fact, the application of IDP in these sectors has gained momentum in recent years with the FSI having a 50% CAGR and healthcare growing at 65-75%.
The finance industry (e.g. banking and insurance) has seen the rise of new use cases for automation platforms: loan applications, automated customer onboarding, mortgage processing, claims processing and many more. In addition, the healthcare sector has used IDP technology to automate a completely different set processes, such as: Account Payable (AKA Invoice Processing), Know Your Customer (KYC) documents to detect and prevent identity fraud and patient onboarding automation.
Outside these two specific sectors, we at Metamaze see new, successful stories popping up on a daily basis in various other sectors and departments. For example, other innovative sectors such as chemistry, manufacturing, logistics and the legal sector find Intelligent Document Processing (IDP) to be a solution that finally enables them to extract valuable information from complex, unstructured and semi structured data.
Companies that still waste their time on manual work (or those who stick to outdated OCR or template-based technology), will sooner or later be forced to jump on the IDP rocket ship due to the abundance of success stories at other companies.
Jo Cijnsmans, BDM @ Metamaze
