The Data Migration Process
Here we present the workflow of a data migration. The number of resources needed for each step will vary from project to project, but these are the key stages we find are always necessary. Our DM REVOLVE Toolset guides the migration through each step, and WOM (Workflow-O-Matic) schedules, records, and verifies as we go along.
By Paul Bimrose on July 27, 2021
DM Planning and Scoping
DM Strategies
Mapping
Extract
Transform
Load
Check and cleanse
Validate
The following slides break down each process
Time spent on planning early on in the project can be invaluable. Discovery and scoping in the planning stage helps to spot potential problems down the road. It includes, but is not limited to
Investigating legacy systems
Defining the data migration scope ie objects to be included
Hardware and security requirements
Data quality concerns
Data volumes
DM strategies are used to define
DM scope within individual data objects eg open transactions only
The DM requirements of each individual data object
Cutover strategies for particular data objects
Specific data quality concerns and how to address them eg duplicates
Expected data volumes
Mapping the data from the legacy system to the new system can present many challenges.
Missing values
Incorrectly formatted data
Invalid values
SME’s (Subject Matter Experts) from the clients side will be essential in working through the legacy data sources for the relevant records and fields.
Once the data for each object is identified we can extract it from the legacy system. The relevant information can be obtained with the help of
Communications between ourselves and the SME’s
Select statements
DM REVOLVE
Hard work!
Once extracted the data objects are checked for completeness and correctness
Once the extraction has been completed any transformations that the data may need should be done. This will usually be changing legacy fields, for example
Months of the year, from January, February etc to Jan, Feb
Splitting a single name field out into first, middle, surname
A set value
Conversion from old to new reference mapping
Once the transformations have finished, each object is checked by the data owner for correctness
All of the data has now been
Extracted from the legacy system
Cleansed
Transformed
It’s time to load it into the new target system. This is done in a simulated testing environment, it will be done several times as the project progresses and new data objects are completed.
If a data object doesn’t pass validation it will need to be checked over again for further refinement. It will be progressed back to whichever stage is needed to fix the problem and stepped through the processes again. This will continue until it is validated and signed off. A data object that needs refinement can go through this process several times before passing.
With the help of our nightly refreshed ROM reports a focused data cleansing effort will be used.
As the various data objects are loaded into the target system they will need to be validated. This is done by the data owner and needs to be a thorough process, catching any data that is
Missing
Incorrect
Excess records eg duplicates
Financially verified
Functionally correct in the new target system
Once the data objects have been validated the migration procedures are locked down and ready for the deployment cutover.
