The Data Migration Process

DM Planning and Scoping

Time spent on planning early on in the project can be invaluable. Discovery and scoping in the planning stage helps to spot potential problems down the road. It includes but is not limited to

  • Investigating legacy systems
  • Defining the data migration scope ie objects to be included
  • Hardware and security requirements
  • Data quality concerns
  • Data volumes

DM Strategies

DM strategies are used to define

  • DM scope within individual data objects eg open transactions only
  • The DM requirements of each individual data object
  • Cutover strategies for particular data objects
  • Specific data quality concerns and how to address them eg duplicates
  • Expected data volumes

Mapping

Mapping the data from the legacy system to the new system can present many challenges.

  • Missing values
  • Incorrectly formatted data
  • Invalid values

SME’s (Subject Matter Experts) from the client’s side will be essential in working through the legacy data sources for the relevant records and fields.

Extract

Once the data for each object is identified we can extract it from the legacy system. The relevant information can be obtained with the help of

  • Communications between ourselves and the SME’s
  • Select statements
  • DM REVOLVE
  • Hard work!

Once extracted the data objects are checked for completeness and correctness

Transformation

Once the extraction has been completed any transformations that the data may need should be done. This will usually be changing legacy fields, for example

  • Months of the year, from January, February etc to Jan, Feb
  • Splitting a single name field out into first, middle, surname
  • A set value
  • Conversion from old to new reference mapping

Once the transformations have finished, each object is checked by the data owner for correctness

Load

All of the data has now been

  • Extracted from the legacy system
  • Cleaned
  • Transformed

It’s time to load it into the target system. This is done in a simulated testing environment, it will be done several times as the project progresses and new data objects are completed.

Validate

As the various data objects are loaded into the target system they will need to be validated. This is done by the data owner and needs to be a thorough process, catching any data that is

  • Missing
  • Incorrect
  • Excess records eg duplicates
  • Financially verified
  • Functionally correct in the new target system

Once the data objects have been validated the migration procedures are locked down and ready for the deployment cutover.

Check and Cleanse

If a data object doesn’t pass validation it will need to be checked over again for further refinement. It will be progressed back to whichever stage is needed to fix the problem and stepped through the processes again. This will continue until it is validated and signed off. A data object that needs refinement can go through this process several times before passing.

With the help of our nightly refreshed ROM reports a focused data cleansing effort will be used.