The Data Migration Process

Here we present the workflow of a data migration. The amount of resources needed for each step will vary from project to project, but these are the key stages we find are always necessary. Our DM REVOLVE Toolkit helps guide the migration through each step, scheduling, recording and verifying as we work through it.

Stages of a Data Migration

By Paul Bimrose on July 27, 2021

Key Stages of a Data Migration
  • scoping
  • mapping
  • extract
  • transform
  • load
  • validate
  • check and cleanse

The following slides break down each process

Scope
Scope

Time spent on planning early on in the project can be invaluable. Scoping is the planning stage and helps to spot potential problems down the road. It includes, but is not limited to

  • investigating legacy systems
  • finding what objects are going to be needed
  • hardware requirements
  • data quality
  • amount of data
Mapping
Mapping

Mapping the data from the legacy system to the new system can present many challenges.

  • missing values
  • incorrectly formatted data
  • invalid data

SME’s (Subject Matter Experts) from the clients side will be essential in working through the legacy data sources for the relevant records and fields.

Extract
Extract

Once we know the data needed for each object we can start working on extracting it from the legacy system. The relevant information can be extracted from the source data with the help of

  • communications between ourselves and the SME’s
  • select statements
  • DM REVOLVE
  • hard work!

Once extracted the data objects are checked for completeness and quality

Transformation
Transformation

Once the extraction has been completed any transformations that the data may need should be done. This will usually be changing legacy fields, for example

  • months of the year, from January, February etc to Jan, Feb
  • splitting a single name field out into first, middle, surname
  • a set value

Once the transformations have finished each object is checked by the data owner for correctness

Load
Load

All of the data has now been

  • extracted from the legacy system
  • cleaned
  • transformed

It’s time to load it up on to the target system. This is done in a simulated testing environment, it will be done over and over again as the project progresses and new data objects are completed.

Validate
Validate

As the various data objects are loaded onto the target system they will need to be validated. This is done by the data owner and needs to be a thorough process, catching any data that is

  • missing
  • incorrect
  • excess records

Whatever data is signed off on at this stage will be used in the final data set on the Big Day.

Check and Cleanse
Check and Cleanse

If a data object doesn’t pass validation it will need to be checked over again. It will be sent back to whichever stage is needed to fix the problem and stepped through the whole process again. This will continue until it is validated and signed off. A data object that has failed validation can go through this process several times before passing.

previous arrow
next arrow
Stages of a Data Migration
Scope
Mapping
Extract
Transformation
Load
Validate
Check and Cleanse
previous arrow
next arrow
Shadow

Leave a Comment

Your email address will not be published. Required fields are marked *