Fragmented data is one of those problems that enterprises rarely realize they have until it begins to come under control – and by that point, such disorganization has existed in myriad different forms for a considerable amount of time. The phenomenon occurs when, in an effort to optimize the performance of their individual units, departments begin designing processes that utilize data sources not shared anywhere beyond their unit, or systems previously implemented by external acquirers are not brought up to par with the standards of the larger enterprise. An integration project, then, is tasked with defining relationships and processes that allow disparate data sets to be reconciled into singular unified facts.
Diagnosing Fragmentation
Integration projects always begin with researching and compiling a picture of the current state of affairs: which sets of data exist in multiple systems and possess some level of functional equivalence, and which systems do not share data at all. This step usually takes more effort than one might expect, as the ways in which two systems store information about the same subject may not be immediately obviously disparate. Upon further inspection, it may turn out that while both contain fields that could be classified as “customer records,” on one, the field contains billing information that is absent in the other, or records are keyed differently so that automated processing fails, or updates to records in one system happen frequently enough that records in the two systems disagree on what the current state of affairs is. It is essential to establish exactly what information is held in which system so that the next step of the project can truly address the concerns at hand.
This step also establishes what specific inconsistencies between systems, if any, warrant immediate attention versus those that may be deferred. While most integration projects have the overarching goal of reducing fragmentation, the degree to which this is necessary varies considerably between use-cases. For some businesses, a complete unified view of all enterprise data is critical to operations, while for others, it is much more useful to ensure consistency in areas of active content and leave other data silos untouched for now. To prioritize these issues, it is crucial to estimate the potential business impact of the inconsistencies identified, which helps to transform a vague feeling that “things aren’t perfect yet” into a concrete list with particular items that require optimization.
As part of this step, it is also essential to distinguish between disagreements that stem from technical causes only and conceptual disagreements that are the result of two sides using the same words to mean different things. The former type, where two systems simply use different data structures to store the same data, can usually be resolved through technical means, while the latter requires human coordination to harmonize disparate concepts. It is useless to attempt to standardize terms if the two sides cannot agree on what these terms should mean.
Designing the Integration Architecture
With the issues to be addressed identified, the next step is, obviously, addressing them. Designing the unified data model is an iterative process that takes into account the findings of the previous steps: the records for which integration is going to be standardized, and the systems which possess these records. It is essential to determine, for each set of records applicable for integration, which system holds authoritative data that other systems should synchronize with.
In many cases, this is fairly obvious: an ERP system, for instance, is almost certainly the system of record for financial transactions, as opposed to, say, the CRM. However, there are also cases where this is not decided on clearly, with departments that interact closely with one another each believing they hold the correct information regarding a particular set of records (e.g. a product catalog). These issues should be ironed out at this stage, as rushing to synchronize data will only lead to one side overwriting the other without any agreement on the theoretical level.
After this, the next question is designing the technical implementation of the synchronization, which again requires considering the findings of the previous steps. Some data sets require immediate and frequent synchronization, as they are used extensively by downstream processes, while others can be updated periodically in accordance with established schedules. It is essential to keep these differences in mind when designing the technical implementation to ensure that the synchronization infrastructure matches the requirements of the data sets in question, as opposed to making things unnecessarily complicated by attempting to synchronize everything in real-time no matter what.
Governance to Keep It That Way
While synchronizing systems reduces fragmentation, it does not eliminate the possibility for data sets to become desynchronized again in the future. The next step, therefore, is designing policies that address data governance and provide grounds for resolving disputes that may arise after the synchronization process. This is probably the most underestimated part of any large-scale integration project, as it deals with the practical nuances of ensuring that nothing slips through the cracks, rather than exciting technical details.
However, it is also one of the most important parts, as the entire reason for embarking on an integration project in the first place is to eliminate data silos – and without adequate governance, new inconsistencies can arise long after the completion of an integration project. Governance policies typically address three areas: escalation procedures that data stewards should follow when disputes arise at the business level (which, even with perfect technical implementation, occasionally happen), evaluation of external parties introducing new systems that interact with enterprise data, and regular data quality checks to ensure that nothing slips through the cracks to cause fragmentation to re-emerge later down the line.
Building Organizational Buy-In for a Single Source of Truth
Finally, the last step in implementing a single source of truth is organizational adoption of the new system. In many ways, this is an extension of the data governance step, as it is the policies established there that determine what exactly changes from an implementation perspective. With regards to actual business practices, however, it is important to recognize that a unified source of truth typically means that individual departments no longer have complete autonomy over data they once managed themselves. Therefore, it is essential to ensure that the benefits provided by the integration project outweigh the costs of losing this autonomy before proceeding, and that departments which interact closely with unified data sets feel heard throughout the process rather than alienated.
The MonoSpear Advantage
The MonoSpear approach to data integration emphasizes the necessity of the combined approach to building a unified source of truth, which incorporates both elements of technical infrastructure and governance policies. By viewing both data standardization architecture and continuous data governance as equal pillars of the endeavor, MonoSpear recognizes the importance of long-term sustainability, as a technically perfect implementation that lacks governance will inevitably fail to account for changing business realities several years down the line.
FAQs
1. What causes data fragmentation in enterprises?
Fragmentation typically occurs when the same data is stored in multiple systems in ways that make reconciliation difficult, usually because of ad hoc measures to optimize the operations of individual departments or acquired businesses that did not bring their data management practices up to par with the acquiring company’s standards.
2. What is a “single source of truth?
It is a set of processes and systems that allow an enterprise to ensure complete consistency of data across all data sets, typically through standardized policies and technical infrastructure that facilitates ongoing synchronization between all data sources.
3. Can integration alone prevent fragmentation, or is additional governance necessary?
Additional governance is always necessary to prevent fragmentation, as integration alone cannot address all the potential causes of data inconsistency, particularly those that occur at the business level.
4. How does MonoSpear approach enterprise integration projects?
MonoSpear recommends developing a unified source of truth in accordance with best practices in enterprise data governance. This typically involves designing a consistent data architecture using Oracle Integration and similar tools to synchronize disparate systems while also developing policies that ensure the integrity of the unified set of facts going forward.