What makes EDI so hard?
EDI – Electronic Data Interchange – is an umbrella term for many different “standardized” frameworks for exchanging business-to-business transactions. It dates back to the 1960s and remains a pain point in every commercial industry from supply chain and logistics to healthcare and finance. What makes it so hard? Why is it still an unsolved problem despite many decades of immense usage?
These are the questions we get most often from developers – both developers who want to become Stedi customers and developers who want to join us to help build Stedi. And these are our favorite questions to answer, too – the world of EDI is complex, opaque, and arcane, but it’s also enormously powerful, underpinning huge swaths of the world economy.
The problem is that there just aren’t any good, developer-focused resources out there that can help make sense of EDI from the ground up. The result is that this wonderfully rich ecosystem has been locked away from the sweeping changes that are happening in the broader world of technology.
We have a 60+ person-strong engineering team at this point, and so we’ve had the good fortune of ramping up a number of people from zero to a solid working knowledge of how the whole EDI picture fits together. It helps to start at the highest level and get more and more specific as you go; this piece you’re reading now is the first one that our engineers read when they join our team – a sort of orientation to the wide, wide world of EDI – and we’ve decided to post it publicly to help others ramp up, too.
Each business system uses a different internal transaction format, which makes it impossible to directly import a transaction from one business system into another; even if both businesses were using the exact same version of SAP – say, SAP ERP 6.0 – the litany of configuration and customization options (each of which affects the system’s transaction format) renders the likelihood of direct interoperability extraordinarily improbable. These circumstances necessitate the conversion of data from one format to another prior to ingestion into a new system.
Since each business has multiple trading partners, and each of its trading partner operates on different business systems, “point to point” integration of these transactions (that is, mapping Walmart’s internal database format directly to the QuickBooks JSON API) would require the recipient to have detailed knowledge of many different transaction formats – for example, a company selling to three customers running on NetSuite, SAP, and QuickBooks, respectively, would need to understand NetSuite XML, SAP IDoc, and QuickBooks JSON. Maintaining familiarity with so many systems isn’t practical; to avoid this explosion of complexity, businesses instead use commonly-accepted intermediary formats, which are broadly known as Electronic Data Interchange – EDI.
EDI is an umbrella term for many different “standardized” frameworks for exchanging business-to-business transactions, but it is often used synonymously with two of the most popular standards – X12, used primarily in North America, and EDIFACT, which is prevalent throughout Europe. It’s important to note that EDI isn’t designed to solve all of the problems of B2B transaction exchange; rather, it is designed to eliminate only the unrealistic requirement that a trading partner be able to understand each of its trading partner’s internal syntax and vocabulary.
Drop Shipment, or
Dropshipwill all be referenced as Dropship, which limits the vocabulary for which an adoptee might have to account. Similarly, X12 has designated the 850 Purchase Order’s BEG03 element – that is, the value provided in the third position of the BEG segment in the 850 Purchase Order transaction set – as the proper location for specifying the Purchase Order number. This reduces some of the burden of mapping a transaction into or out of an adoptee’s business system; only one value for drop shipping and one location for PO number must be mapped.
The problem with standards
Intermediary formats – that is, EDI – make life somewhat easier by limiting the number of different formats that a business must understand in order to work with a wide array of trading partners; a US-based brand selling to dozens of US-based retailers likely only needs to work with the X12 format. However, the brand still needs to account for the different ways that each retailer uses X12. Again, X12 is just a dictionary of possible values – since Walmart and Amazon run their businesses in different ways (and on different business systems), their implementation of an X12 intermediary format will differ, too.
There are other areas of inherent complexity in EDI, too. Because businesses change over time, the configurations of the businesses’ respective business systems must change, too; an example might be a retailer adding DHL as a shipping option, whereas it previously only offered FedEx. Those changes must be communicated to trading partners so that field mappings can be updated appropriately; because such communications and updates involve ‘best efforts’ from humans, some percentage of them will be missed or completed incorrectly, leading to integration failures on subsequent transactions. Even without inter-business changes, errors happen – for example, a business system’s API keys might expire, or the system might experience intermittent downtime. Such errors will need to be reviewed, retried, and resolved. Just as every solution’s setup process will always require per-trading-partner, human-driven field mappings, every solution must also provide functionality for managing configuration changes on the control plane and intermittent errors on the data plane.
The 'laws of physics' of the business universe, then, are as follows:
- There are many businesses, and those businesses must conduct business-to-business trade in order to deliver end products and services;
- Those businesses run on a large but bounded number of business systems;
- Due to the heterogeneity of business practices, those business systems offer configuration options that result in an unbounded number of different configurations;
- The heterogeneity of configurations makes it impossible for a single unified format, therefore necessitating a per-trading-partner setup process;
- The business impact of incorrect setup requires that a human be involved in setup;
- Businesses are not static, so configuration will change over time, again necessitating human input;
- Business systems are not perfectly reliable;
- Because neither human input not business systems are perfectly reliable, intermittent errors will occur on an ongoing basis;
- Errors must be resolved in order to maintain a reliable flow of business.
Such is the challenge of developing a generalized business integration system.
A familiar problem
The good news is that such circumstances (an unbounded set of heterogeneous, complex, mission-critical, web-scale use cases) are not unique to business integrations – they mirror the circumstances found in the broader world of software development.
Stedi serves three types of customers:
- Customers that need to build integrations to support their own business (e.g. an ecommerce retailer that wants to exchange EDI files with its suppliers);
- Customers that need to build EDI functionality into their own software offerings (e.g. a digital freight forwarder, transportation management system, or fulfillment provider that wants to embed self-service EDI integration into their platform);
- EDI providers or VANs that need to modernize their technology stack (e.g. a retail-specific EDI software provider that wants to replace legacy infrastructure or tools).
With Stedi, you can build massively scalable and reliable integrations on your own without being an EDI or development expert.
EDI Reference provides users with documentation on all transaction sets, segments, and elements of every X12 release, and all message types, segments and elements of every EDIFACT version.
EDI can be tough to look at, so Inspector turns EDI into a format that is readable by humans. Quickly visualize and debug any X12 or EDIFACT document. Share EDI files easily with trading partners.
Get blog posts delivered to your inbox.