The Risks of Relying on an ETL / Excel Approach to Enterprise Data Migration

In building your approach to digital transformation, one key topic always arises – how are we going to move the data? There have been many approaches over the years, and for as long as migrating data has been around, Excel has sat at the heart of it. As technology developed, the concept of ETL (Extract, Transform and Load) arrived to offer a smoother (on the surface at least) method of getting data out of a legacy system and onto the new platform; however, there’s always been a problem – an Excel/ DIY approach using solely ETL technology works if the source landscape or data construction is complex.

In our experience at Syniti, we’ve seen many consequences from ETL/Excel projects over the years and witnessed the costly ramifications that it brought businesses. The reason the majority of these enterprises got into trouble is that they failed to understand the risks of the approach. There are plenty of ETL tools available so we won’t go into the specifics of any one of them, but what I’ll seek to do here in this blog is to analyze the general risks of an ETL/Excel approach.

Expansive Project Teams, working in harmful silos 

With an ETL-based approach to an enterprise-scale migration project, the data team, business people, and developers work independently while project managers work furiously in the background to hold things together. Excel is typically used to gather mapping requirements, collect missing data, and document transformation requirements that will be implemented in ETL tooling. With the potential for thousands of Excel spreadsheets being worked on and mailed around, version control can become a huge issue, as can testing and quality control.  For most companies, the answer to this controlled chaos is to throw expensive human resources at the problem to try to wrestle it to completion, but this does not come without radically increased costs. 

Bottom Line: Greater risks of a failed go-live and increased project costs 

Lack of Project Visibility for IT Leadership and Executives 

With so much individual work in a migration project, it’s a big challenge for program leaders to compile real-time status updates for themselves or for executive leadership. Without easy-to-access dashboards, fuelled by real-time, activity-based insights into project status and timeline, project leaders can struggle to pivot projects based on the reality of the situation, which leads to increased risk.  Executives too, are more likely to be more distrustful and concerned by the whole program's direction which can lead to a lack of resources or worse.

Bottom Line: Greater risk of not delivering on-time and on-expectation, and of the boss getting frustrated. 

Limited Code Automation Results in Inconsistent Results 

With thousands of data mapping and data harmonization challenges to address, developers and data stewards can quickly become overwhelmed. Relying on an ETL/Excel approach means limited or no automated code creation and a lack of no-code development capabilities. Additionally, ETL tools do not prevent the creation of poorly written ETL code. This significantly increases the need for bespoke modifications and long rounds of testing, increasing the chances of complications and delays. 

Bottom Line: Increased need for custom code generation and more points of potential failure 

You Get Where You’re Going and the Data is Still Lousy 

Getting data “lifted and shifted” from the source to the target system isn’t the definitive success factor for a data migration. While a technical migration is the fastest way to move from one system to another, if the data isn’t configured, harmonized, and mapped properly to the target system’s demands then the new system simply won’t be as functional as desired. Not “rightsizing” the data also means you are paying more for storage and the infrastructure required to maintain it. This lack of data quality and rightsizing focus in the ETL/Excel approach can cost a business valuable time and resources, limiting the effectiveness of the target system when it switches on.  

Bottom Line: Risk of elongated time-to-value, unhappy users/ customers, expensive IT upkeep

You Throw Away Your Hard Work 

While ETL and Excel (combined with a lot of manpower) may get a project over the line, it doesn’t naturally capture all the knowledge generated during the project and make it available for re-use in future migrations and data projects. Organizations looking to conduct another migration, data quality, master data management, or any other data initiative, will need to start from square one to rebuild any rules from the migration again from scratch. 

Bottom Line:  Your next project is going to cost more and take longer than it needs to 

Your Data is Vulnerable

The ETL/Excel approach has serious flaws in regard to its security of data. Mailing spreadsheets around with personal, private and business data exposes a business to a number of deliberate hacks and accidental human errors. Without a central repository, securely protected by enterprise-grade firewall technology, it’s only a matter of time before a vulnerability in the ETL/Excel process causes significant financial and reputational consequences for the enterprise.

Bottom Line:  Your data is vulnerable without a central data repository

ETL/ Excel may appear to be the simplest or the path of least resistance on the surface, but as we’ve shown here in this blog there are plenty of risks with taking this unautomated, unaligned approach to enterprise data migration. The good news is that ETL and Excel is not the only way to tackle data migration. If you’d like to learn more about how Syniti has successfully delivered thousands of successful enterprise migration projects, complete with Boring Go-Lives – for the world’s leading businesses download IDC’s Study on Syniti Migrate to reveal how much more valuable and risk-free data migration can be.