Data Migration : Big Bang or Dripping Data

Data migration requires a migration plan and a migration strategy. How will the transition to the new system take place? After all, the data migration is aimed at successfully putting a new system into use. Data is a prerequisite. When the new system goes live, current data must be available, but until then, the old system is still in use. In the intelligent data-driven organization, there is no freeze period, It's business as usual during the transition. 

Migration strategy is more than data migration strategy, it is about ensuring that the data migration matches the go-live of the new system. The go-live plan is mostly an organizational issue. Data migration has both technical and functional aspects. In this article, we describe, from a data migration perspective, how to ensure continuity with different migration strategies. 

Big Bang 

Everything passes at once, overnight or over a weekend. The business considers this risky. From a technical perspective, this is often chosen as the most desirable strategy. No old and new system side by side, no temporary interfaces. Also from a data migration perspective, a big bang is often preferred. It is a clear choice that keeps scoping and integration issues simple. 

Even in a big bang scenario, the data migration is built iteratively through an agile process. Each sprint is followed by a test or trial migration. For financial data, we typically use two-week sprints and a monthly trial migration with new month-end data. With each trial migration, more and more data is successfully migrated. During the final trial migration, data is processed 100% correctly. So we know for sure that production migration will go as planned. 

Portfolio driven 

Instead of one big bang, you may opt for a number of little bangs. This is an option when the entire dataset can be divided into a number of independent sets that can be migrated separately. For example, with a breakdown by location, customer group or product group. We call such a dataset a (sub)portfolio. 

We first realize the data migration for a relatively simple portfolio of sufficient size. We can then quickly migrate this first portfolio. For the other portfolios, we build on it. With this migration strategy, we have an old and new system running side by side for some time. If the portfolios are independent, this poses few problems. It is clear what is managed in which system. The key issues are the financial links and the overall management reports. 

On demand 

If we are able to minimize the dataset that we want to migrate in one go to just one case, then we are able to migrate in many small tranches: on demand, also known as dropping data. This is an option if the target system supports case-oriented working, for example managing customer cases. Together with the business, it is determined how many and which cases are migrated during the night or in a weekend. 

Before the first case is migrated, the migration solution must be ready and tested. From that point on, you can start to drip data. It must be clear to the users which case is managed in which system. We have worked with an extra front-end where the end user selects a case and is automatically taken to the correct system. This front-end is linked to a migration administration repository in which the status of each case is tracked. This repository contains planning data such as Ready to Migrate, Scheduled to Migrate, and Migrated. The repository also triggers when a case needs to be closed in the old system. Furthermore, the repository contains information for supervisors such as audit trail and audit counts. 

Synchronization 

A special form of data migration is synchronization. This involves keeping data current in both the old and new systems for an extended period of time. The data remains available in both systems. This can be a solution when the legacy system contains specific functionality that is still needed for a period of time, for example, because it will be implemented in the new system at a later date, or because it may no longer be needed in the future. 

Synchronization can be thought of as a periodic or continuous data migration. Technically, there are several ways to synchronize systems. It is important to choose a stateless solution, a solution where synchronization is independent of previous synchronization. This creates a robust solution that works even if a previous run went wrong. 

The best choice 

A migration strategy is more than the choice between big bang or dripping data. It involves decisions that determine what data goes into the new system and what happens to historical data: leave it behind or migrate to the new system or to a solution for historical data management. What happens to data that simply does not fit into the new system: are products rationalized or is the system modified? There are also technical decisions to be made, such as how data will be loaded into the new system: using an import service or a direct load into the database. 

The migration strategy describes the fundamental choices. The best choice is the strategy that ensures business continuity and controls the transition. Sometimes it is migration in one go: clear to everyone. Sometimes it’s dripping data: a gradual transition to a new situation. 

A Dutch version of this blog is published at LinkedIn: : https://www.linkedin.com/pulse/datamigratie-big-bang-druppelen-henk-zwaan-dceke