Reimagine Your Customer Data Integration

Many businesses either work with a manual integration strategy where people, not digitized processes, keep track of customer data across different channels and maintain that in spreadsheets or rudimentary databases.

“For most businesses, customer data integration limps along — far from the latest standards of data integration.”

These manually maintained databases often become difficult to manage and serve as a bottleneck for business growth. And they certainly limit the extent to which a delightful CX can happen.

Slightly more evolved businesses rely on their IT teams to develop EDI or API-based mechanisms for exchanging data between different channels and tracking them in somewhat more sophisticated databases. In B2B interactions, IT teams on both sides agree on EDI or API standards and begin data exchange, but they are soon challenged by the sheer volume of data exchange and become “locked-in” to the data exchange, and its lack of visibility to business people (for both you and your customer). They ultimately face hardships while expanding the nature of your business relationship on the fragility of the data exchange’s architecture. Expanding in other areas of collaboration between you and your partners often reverts back to shared DropBox folders and spreadsheets until such time as highly skilled IT experts can productize the exchange into more instances of low-visibility traditional EDI or API-like exchanges. Agility suffers. Collaboration is low. Symptoms can also include degrading data quality and increasing skilled IT labor to correct data errors.

In any case, none of these approaches meet the demands of today’s business needs to provide for the best CX. A select few businesses using the current generation of enhanced self-service integration technologies are reaping the benefits of delivering sublime customer experience and satisfaction, but more on that later. Let’s first look at how integration strategies have evolved in sync with data communication technology evolution.

Before data could be exchanged electronically, businesses traded physical paper documents and the keen eye of business owners to capture, process, and deliver customer orders. The shortcomings of that process were too many, and may seem too trivial today. The first big breakthrough came in the form of electronic exchange of data, or popularly known as EDI (Electronic Data Interchange). EDI had the potential for a commercial application and soon had standards committees laying out rules for this electronic exchange of information. Standards, such as EDIFACT, X12, HL7, HIPAA, etc., were developed for specific industries and applications, and each had their own rules, formats, character sets, etc.

Fast forward to the 2000s, the EDI standards have proliferated into complex document exchanges typically between large trading hubs that are able to mandate compliance among their typically less-powerful trading partners (often suppliers). And over the past 10 years, APIs (over the internet) started making headway as an alternative to EDI. API, or Application Programming Interface, provides for greater agility and customization without as much lock-in to figuring out how to make the EDI standards accommodate more modern business data exchanges. Instead of relying on EDI document standards and rules, APIs used XML or JSON as response formats for communication. However, whatever technical debt you might avoid with less EDItech lock-in is doubled in terms of technical-debt as you try to keep up with API versioning that is often very specific to one trading relationship and more encompassing than just data formatting.

“Before data could be exchanged electronically, all businesses had were paper records and the keen eye of business owners to capture, process, and deliver customer orders. The shortcomings of that process were too many, and may seem too trivial today.”

Each of these methods for ‘better’ B2B data integration have their own set of drawbacks. For instance, the growing and myriad EDI standards and specifications made the technology hard to scale, turning into a complex motley of requirements that was too difficult to manage as businesses expanded their relationships. With each new trading partner, businesses had to employ expensive, high skilled IT to manage and grow the EDI support system. This was a barrier to scale and digital transformation.

With an XML based approach, the message sizes can grow unwieldy, become too complex and result with inefficiencies in processing a high volume of other small messages. When parsed in memory, they are resource-intensive and potentially disruptive to other B2B message traffic.

VANs, or Value-Added Networks, were another alternative to direct EDI connectivity. Essentially, VANs functioned as communication brokers in the middle, directing EDI messages between varied partner mailboxes. It was especially useful early on before many companies had constant internet connectivity and an ability to directly send and receive electronic documents. But this was long ago, when there were dialup connections and phone-line modems. It was also useful when there were additional requirements that direct, peer-to-peer EDI couldn’t tackle because of its nature, and that’s why a connectivity broker in the center was needed. However, VANs generally were too expensive, especially for small businesses who did not have too many transactions flowing through. In most cases VANs also left a lot to be desired when it came to tracking and transaction status. And in cases where there were multiple VANs between you and any of your trading partners, you could just forget about document transfer status or visibility.

Once Walmart and other major trading hubs supported and championed (read: “mandated”) the AS2 protocol for data transfer, it provided a means by which one might circumvent the traditional, expensive VANs. However, because of its reliance on internet protocols, even this method wasn’t the silver bullet small businesses needed. If anything, AS2 complicated things by imposing additional requirements, specialized software, and digital certificate management upon trading partners. IT to the rescue. But the cure wasn’t easy or inexpensive.

In short, all these technologies solve pieces of the puzzle but brought forth their own set of new problems and limitations. One set of limitations was replaced by another as mechanisms, standards, and implementation methods changed. And with each, a greater degree of IT involvement was needed to implement new trading partner connections. The resulting set of technologies involved in trading partner connections, data transfer, data reformatting, exception handling, data security, and so on evolved what was once a business person writing out a purchase order and mailing it to a supplier into a multi-layer technology masterpiece capable of processing thousands of orders per minute. But setting up, configuring, and servicing this beast for new customers is IT-intensive and extremely deficient from a CX point-of-view.

The key in understanding what would alleviate these headaches is considering all four dimensions of customer data integration. Instead of chasing some minor technical improvement or incremental gain in IT productivity by partially automating one of a hundred steps, perhaps it’s time to take a step back from the entirety of the process and technology and reassess the bigger picture.

Some old technology acronyms die off very slowly. (You can name your hourly rate as a COBOL programmer right now!) EDI is an example of this. Initially used in the early 1970s (long before disco), EDI became the de facto standard on how to exchange computer data along supply chains. Rather than have each business invent its own data layout for standard transactional business documents such as purchase orders and invoices, EDI standards bodies provided a format that accommodated most business relationships.

However, over the years, more and more things were added to the standards for each EDI document. There were numerous versions of the standards and not everyone could change their EDI maps that quickly, so large businesses quickly found that various segments of their supply chain were stuck on older standards of EDI. With the shear growth of all the optional fields and structures in just about any given EDI document — loaded with all the fields, repeating loop structures, hierarchical levels, and so forth to support just about every possible business relationship on a purchase order document, the overall full standard is not a very inviting experience for you, your customers, your trading partners, your business users, their business users, or really anyone — not even the well-seasoned EDI experts. One thing you can be sure of: there is no inventory planning, ERP, or other enterprise software, homegrown, bespoke, customized, or commercially available that natively accepts and stores EDI documents. Literally nobody uses the native EDI format for their own internal purposes. Everyone who accepts EDI must map received EDI documents into some other format for use by their applications. Everyone who sends EDI must map existing data from their applications and databases into an EDI format for subsequent sending. It’s a challenging format that only exists as an intermediary format between businesses. In every instance of an EDI document, there is upstream source data (from the sender) and downstream destination data (at the receiver) and neither of those formats is EDI. EDI persists only as the intermediary format in the middle and almost always between two business entities.

Need another example of how well-loved the EDI format is? Virtually every business that operates today beyond 150 employees has at least two separate internal applications that need to exchange data with each other. Larger businesses can have hundreds of such internal applications that exchange data with each other. Even with thousands of EDI document types available for use and available tools in use at all of these businesses (they’re used to exchange data externally: B2B) it is a very rare circumstance to find where two internal business applications use an EDI format to exchange data.

Now, before you get mad at your EDI team, you should know that trillions of dollars of commerce are conducted by use of EDI every year. This old data standard represents nearly half of all present-day B2B transactions. So, if you’re using EDI, you’re in good company. Alas, to do EDI with your next new customer doesn’t necessarily mean you’ll leverage existing EDI maps, you’ll write more maps. So, you’ll just be doing more EDI.

Some smart technology experts have been touting APIs as a superior approach to B2B transacting. But if you thought EDI was old (1971), APIs are nearly a generation older (1951). They represent a more independent and business-specific approach to formatting and accessing data ungoverned by central EDI committees and published standards.

Fueled by the proliferation of secure internet exchange, information-savvy businesses have formed the API economy whereby certain APIs form a business model whereby customers requesting certain information can subscribe to the use of a supplier’s APIs to pay for the information, on-demand. Complex and customized information delivery value-chains that don’t fit into defined and standard EDI documents and associated data transfer and access methods are better suited for implementation using APIs.

But if EDI requires specialized data integration skills to define, implement, and maintain, APIs are nearly an order of magnitude worse. For one, the mitigation of different data formats and layouts is often implemented directly in source code in the language of the initial implementer’s choice. Whatever programming language was chosen, Murphy’s law relegates you to having little to no such available skills on hand to upgrade that API transaction and data handling at a later time. The technical debt that you incur as you embody critical B2B information exchange into literal computer source code inclusive of access methods, security, workflows, data format mitigation, and so on, is far greater than a typical EDI technology stack which almost always has an architecture that carefully separates access, delivery, tracking, workflows, data formats, and so on.

There are pros and cons based on the task at hand regarding which to choose: EDI or APIs. Sadly, neither EDI nor APIs intrinsically enable a business user to implement a new B2B integration connection let alone empower a non-technical business or customer user. So, a debate of EDI versus API is the wrong debate for the current business environment where non-technical business and customer users need to be given tools to more quickly connect and conduct business.

To meet increasing customer expectations, consider your own increased expectations when conducting personal business. What used to be an involved process to get an auto or homeowners insurance quote, must now take place in a 15 minute session on a website and result in an immediate quote that lets you subsequently play what-if conditions with varying coverage and deductible parameters. With your bank, you may have grown accustomed to the ease of depositing checks from your smartphone and an app published by your bank. This was something just a few years ago that you expected would require an in-person visit to your nearest bank branch. And when you get the idea for a new item that you’d like to purchase, you now have the expectation that the next time you sit down with any one of your devices, you’ll be able to quickly find that item on Amazon and have it delivered to your door within a couple days. Speed, convenience, and ease define your expectations for getting things done. Does your company offer the same level of speed, convenience, and ease as your business customers interact with your front office? If not, someone else will.

Sure, the level of data integration between you and your corporate clients is much more intensive than your personal Amazon shopping cart. But if you have a team of onboarding professionals who still need to write data maps, manage customer-specific spreadsheets, and craft scripts and other artifacts (including code) using data integration tools, then it likely takes your company weeks, if not a few months to onboard new clients. This is not fast, convenient, or easy.

As the basis for reimagining, just consider what it could be like if your IT experts weren’t on the critical path to implementing each new customer onboarding. What if customers and front-office business people could implement new data integration onboarding?

Self-service integration ensures that the people in your value-chain (with albeit limited technical competency) can look directly at data streams and customer entities through intuitive screens and dashboards and make data onboarding connections by only pointing and clicking through easy screens. It can be so simple that anyone can be a data integrator, enabling everyone in the organization to delight customers.

At the same time, IT is freed up from day-to-day implementation but continues to serve in a high-level governance role, managing the integration environment while being freed from tedious and thankless API coding and EDI mapping. IT can focus on more strategic tasks instead of being bogged down with data feed integration tedium.

With technological advances in machine learning and security protocols, non-technical users can let artificial intelligence understand and define data semantics, and then make data connections following those data rules. Further, a single click is needed to retrieve, breakdown, and manage multi-dimensional, complex data and stream it in real time to execute modern day business transactions.

Business users can also process highly secure transactions between a protected ecosystem and their environment without worrying about breaches, only because the current generation of customer data integration is already taking care of that in the background.

Effectively, this more than takes care of all the 4 Vs of data and enables business users to make important business connections happen as quickly as they can be conceived.

Adeptia’s self-service integration product provides a business application experience for your business users. It has been cited by numerous Adeptia customers as having sped customer data onboarding by as much as 80%.

As we enter the third decade of the 3rd millennium, although you might think that advances in technology and consequent resulting business expectations couldn’t be much higher. However, as Moore’s law continues, the rise will be nearly exponential, and naturally, this will lead to even further evolution in customer data integration.

How would you reimagine your customer data integration if you were not limited by the constraints of current multi-enterprise integration?

As trends have shown, for integration to really make a difference, it needs to grow beyond the traditional limitations of IT. True power lies in making integration more collaborative and decentralized, and not as a monolithic function of one department in an organization.

“The onus of customer data integration should ideally be passed on to non-technical business users who can have more control over data flows through easy to use dashboards that only require pointing and clicking. Technology should enable them so much that they don’t need to know how the underlying data transformation works — that all should be driven by technology subliminally in the background. With just a few mouse clicks, and it should be done.”

From what we can see right now, the future of customer data integration is pretty exciting. For one, the increasing capacity of artificial intelligence and machine learning to handle the myriad details of customer data integration means computer hardware and software are doing more, so you can focus on the big picture.

It will be the job of AI to figure out data semantics, data mappings, required data transformations, and finally, protocols and execution of data exchange. AI will manage the 4Vs of data and define an execution strategy accordingly, equipped with anomaly detection and corrective workflow. AI will see how to best transfer data on the basis of volume and velocity, define what kind of data transformation is needed, and implement data quality and reconciliation controls to take care of veracity. All business users will need to do is to dictate what business relationships their companies will engage in and with whom.

As a fan of sci-fi, it has been fun watching how closely life has been imitating art. We now carry around tablets that we can directly write on and smartphones (I’m still waiting to get beamed up). Many of us verbally interact with virtual personal assistants (VPAs), “Alexa, please turn on the front porch lights.” And, while driving, “Hey Siri, call Allison.” As these VPAs get smarter and more capable, their users will have an expectation that data trapped inside corporate data warehouses could be interrogated and interacted with through voice commands. Perhaps you might say, “Hey Adeptia, show me all orders over $1M from our tier-one customer over the past six months.”

Or, think about the superhero, Tony Stark, in his lab, with floating projections of all customer data connections and movement of data between businesses presented in a multidimensional holographic view that fits in your palm. Imagine real time data tracking across this holographic network, with instantaneous flashing-red error reporting that only requires a few waves of the hand on the holographic network to remedy and return to uninterrupted business flow. Advances in Virtual Reality (VR) wearable gear are bringing the cutting-edge factors of Iron Man’s lab into your workplace in the coming years.

Instead of going to the bank, the bank will come to customers, powered by Virtual Reality interfaces, and all interactions and service delivery will take place digitally without the customer having to step outside their space. Or, when customers order something using the internet, logistics will be able to dive real-time into their warehouses, source and schedule delivery of the product to each customer, show them a view of their product in real time, its sourcing from any location in the world, and its travel right to the final delivery at their doorstep powered by virtual reality and AI interactions.

Reimagining Customer Data IntegrationA few decades ago, no one had imagined that people would be able to control their house lighting ambience, music, and connected devices with small mobile gadgets that fit in their pocket. Bring this leap in technology to B2B business transactions, deliver power in the hands of your business users, and let AI do more and more of the work as you worry less about the technical details of information integration and plan the next big endeavor to delight your customers and gain a competitive edge.