The true cost of migrating bad data: why you can’t afford not to profile your data before importing

There’s no doubt about it – system migration is costly. The top 500 companies spent over €4.5billion last year just migrating to new systems.

Though often the reasons to migrate are clearly defined, the true price estimation is sometimes neglected at the beginning of the project. Too often enterprises only face up to the real cost of the migration in the board meeting at the end when they are having to explain why the project has run over on time and budget – presumably a nightmare scenario for any harried exec!

A new system is usually chosen with the hope that it will perform better than the old one – but the truth is, if it’s running on the same bad data, it will not live up to its potential. However much you’ve spent on your new system, it really never had a fair chance if it is threatened by bad data.

> See also: The six steps to migrating to the cloud

Installation always puts a system at risk, and while modern best-of-breed systems often come with a form of data protection and the ability to force user input to follow a predefined set of processes and various checks as data runs in, these checks may be switched off to aid the migration effort. Sometimes they are even not applicable to the initial load of data, thus putting the system at risk before it has even got going.

Furthermore, problems also tend to arise over where exactly the responsibility for data quality lies. Typically, no-one is directly responsible for data quality as it moves into the new system – yet the vendor of the new system will be expecting the data in a specific format so it can be seamlessly imported into the newly acquired product.

The vendor of the old system (if applicable) will at most be able to export the current information and provide a snapshot view of what is currently being used to run the business. Any transformation that may happen in between these two phases is often only done to physically make the data fit in the new structure.

Only data that is physically aligned with its container can be called technically correct but it might not be functionally correct. Not a single system can claim to work at full efficiency if the data it runs on is out-of-date, inaccurate or partially missing.

Therefore, the various costs that should be considered when migrating core information systems are: time taken to develop/test/implement migration scripts, cost of training staff on how to use the new system,c osts associated with running both the new and the current system in parallel

Adjusting companywide processes and reports to the new system

Another cost that is rarely added, but should be factored in, is the potential cost of the migration failing. This also includes the very real cost of the migration being delivered far later than originally scheduled.

One way in which this cost can be expressed is as the risk (%) of the migration failing, factoring in the scenario that the new system does not deliver its promises on time. Gartner estimates that a staggering 31% of data related initiatives failed last year due to bad data. Another 24% endured the late delivery of their project due to data related problems.

It’s not all negative, however. Though data quality can be a migration’s greatest risk, it can also be its greatest asset. Migration is an opportune time to think about data quality and to ensure the new system is fed with the highest quality data it can be supplied it with.

> See also: Six 'people and process' tips for migrating to a hybrid cloud environment

Only then will the investment so convincingly promised in the marketing brochure be delivered to you and your organisation.

Data quality tools can automate the profiling of migration data and point out inaccuracies and problems that will significantly slow down the migration or reduce the effectiveness of the new system.

Alongside the statistical insights gained from profiling the data, there is the added bonus of being able to ensure that all data passes a data quality filter as it is being migrated. This guarantees that all migrated data complies with the standards set forth by the migration team and guarantees the correct import into the new system.

If you’re still left wondering whether or not all this additional effort is really worth it, remind yourself of what benefits reducing the risk of your implementation failing by 50% will bring. Migrations are costly, but don’t let these costs get the better of you.

Sourced from James Cotton, iWay Solution Director for EMEA, Information Builders

Avatar photo

Ben Rossi

Ben was Vitesse Media's editorial director, leading content creation and editorial strategy across all Vitesse products, including its market-leading B2B and consumer magazines, websites, research and...

Related Topics