The Data Migration Process
Here we present the workflow of a data migration. The amount of resources needed for each step will vary from project to project, but these are the key stages we find are always necessary. Our DM REVOLVE Toolkit helps guide the migration through each step, scheduling, recording and verifying as we work through it.
By Paul Bimrose on July 27, 2021
- scoping
- mapping
- extract
- transform
- load
- validate
- check and cleanse
The following slides break down each process
Time spent on planning early on in the project can be invaluable. Scoping is the planning stage and helps to spot potential problems down the road. It includes, but is not limited to
- investigating legacy systems
- finding what objects are going to be needed
- hardware requirements
- data quality
- amount of data
Mapping the data from the legacy system to the new system can present many challenges.
- missing values
- incorrectly formatted data
- invalid data
SME’s (Subject Matter Experts) from the clients side will be essential in working through the legacy data sources for the relevant records and fields.
Once we know the data needed for each object we can start working on extracting it from the legacy system. The relevant information can be extracted from the source data with the help of
- communications between ourselves and the SME’s
- select statements
- DM REVOLVE
- hard work!
Once extracted the data objects are checked for completeness and quality
Once the extraction has been completed any transformations that the data may need should be done. This will usually be changing legacy fields, for example
- months of the year, from January, February etc to Jan, Feb
- splitting a single name field out into first, middle, surname
- a set value
Once the transformations have finished each object is checked by the data owner for correctness
All of the data has now been
- extracted from the legacy system
- cleaned
- transformed
It’s time to load it up on to the target system. This is done in a simulated testing environment, it will be done over and over again as the project progresses and new data objects are completed.
As the various data objects are loaded onto the target system they will need to be validated. This is done by the data owner and needs to be a thorough process, catching any data that is
- missing
- incorrect
- excess records
Whatever data is signed off on at this stage will be used in the final data set on the Big Day.
If a data object doesn’t pass validation it will need to be checked over again. It will be sent back to whichever stage is needed to fix the problem and stepped through the whole process again. This will continue until it is validated and signed off. A data object that has failed validation can go through this process several times before passing.
