The Data Migration Process

Here we present the workflow of a data migration. The number of resources needed for each step will vary from project to project, but these are the key stages we find are always necessary. Our DM REVOLVE Toolset guides the migration through each step, and WOM (Workflow-O-Matic) schedules, records, and verifies as we go along.

Stages of a Data Migration

By Paul Bimrose on July 27, 2021

Key Stages of a Data Migration

DM Planning and Scoping
DM Strategies
Mapping
Extract
Transform
Load
Check and cleanse
Validate

The following slides break down each process

Scope
DM Planning and Scoping

Time spent on planning early on in the project can be invaluable. Discovery and scoping in the planning stage helps to spot potential problems down the road. It includes, but is not limited to

Investigating legacy systems
Defining the data migration scope ie objects to be included
Hardware and security requirements
Data quality concerns
Data volumes

Scope
DM Strategies

DM strategies are used to define

DM scope within individual data objects eg open transactions only
The DM requirements of each individual data object
Cutover strategies for particular data objects
Specific data quality concerns and how to address them eg duplicates
Expected data volumes

Mapping
Mapping

Mapping the data from the legacy system to the new system can present many challenges.

Missing values
Incorrectly formatted data
Invalid values

SME’s (Subject Matter Experts) from the clients side will be essential in working through the legacy data sources for the relevant records and fields.

Extract
Extract

Once the data for each object is identified we can extract it from the legacy system. The relevant information can be obtained with the help of

Communications between ourselves and the SME’s
Select statements
DM REVOLVE
Hard work!

Once extracted the data objects are checked for completeness and correctness

Transformation
Transformation

Once the extraction has been completed any transformations that the data may need should be done. This will usually be changing legacy fields, for example

Months of the year, from January, February etc to Jan, Feb
Splitting a single name field out into first, middle, surname
A set value
Conversion from old to new reference mapping

Once the transformations have finished, each object is checked by the data owner for correctness

Load
Load

All of the data has now been

Extracted from the legacy system
Cleansed
Transformed

It’s time to load it into the new target system. This is done in a simulated testing environment, it will be done several times as the project progresses and new data objects are completed.

Check and Cleanse
Check and Cleanse

If a data object doesn’t pass validation it will need to be checked over again for further refinement. It will be progressed back to whichever stage is needed to fix the problem and stepped through the processes again. This will continue until it is validated and signed off. A data object that needs refinement can go through this process several times before passing.

With the help of our nightly refreshed ROM reports a focused data cleansing effort will be used.

Validate
Validate

As the various data objects are loaded into the target system they will need to be validated. This is done by the data owner and needs to be a thorough process, catching any data that is

Missing
Incorrect
Excess records eg duplicates
Financially verified
Functionally correct in the new target system

Once the data objects have been validated the migration procedures are locked down and ready for the deployment cutover.

previous arrow
next arrow
Stages of a Data Migration
Scope
Scope
Mapping
Extract
Transformation
Load
Check and Cleanse
Validate
previous arrow
next arrow
Shadow