Lifecycle of a Data Migration
DM Planning and Scoping
Time spent on planning early on in the project can be invaluable. Discovery and scoping in the planning stage helps to spot potential problems down the road. It includes but is not limited to
- Investigating legacy systems
- Defining the data migration scope ie objects to be included
- Hardware and security requirements
- Data quality concerns
- Data volumes
DM Strategies
DM strategies are used to define
- DM scope within individual data objects eg open transactions only
- The DM requirements of each individual data object
- Cutover strategies for particular data objects
- Specific data quality concerns and how to address them eg duplicates
- Expected data volumes
Mapping
Mapping the data from the legacy system to the new system can present many challenges.
- Missing values
- Incorrectly formatted data
- Invalid values
SME’s (Subject Matter Experts) from the client’s side will be essential in working through the legacy data sources for the relevant records and fields.
Extract
Once the data for each object is identified we can extract it from the legacy system. The relevant information can be obtained with the help of
- Communications between ourselves and the SME’s
- Select statements
- DM REVOLVE
- Hard work!
Once extracted the data objects are checked for completeness and correctness
Transformation
Once the extraction has been completed any transformations that the data may need should be done. This will usually be changing legacy fields, for example
- Months of the year, from January, February etc to Jan, Feb
- Splitting a single name field out into first, middle, surname
- A set value
- Conversion from old to new reference mapping
Once the transformations have finished, each object is checked by the data owner for correctness
Load
All of the data has now been
- Extracted from the legacy system
- Cleaned
- Transformed
It’s time to load it into the target system. This is done in a simulated testing environment, it will be done several times as the project progresses and new data objects are completed.
Validate
As the various data objects are loaded into the target system they will need to be validated. This is done by the data owner and needs to be a thorough process, catching any data that is
- Missing
- Incorrect
- Excess records eg duplicates
- Financially verified
- Functionally correct in the new target system
Once the data objects have been validated the migration procedures are locked down and ready for the deployment cutover.
Check and Cleanse
If a data object doesn’t pass validation it will need to be checked over again for further refinement. It will be progressed back to whichever stage is needed to fix the problem and stepped through the processes again. This will continue until it is validated and signed off. A data object that needs refinement can go through this process several times before passing.
With the help of our nightly refreshed ROM reports a focused data cleansing effort will be used.