Migration Guidebook
a practical guidebook for data platform migrations
Phase 1 of 5
Phase 01

Vision

Define the scope, strategy, and success criteria for the migration.

People Involved

MA
Migration Architect
Owns the migration vision by defining the scope, strategy, and success criteria.
PA
Platform Architect
Assesses the target platform and validates that it can support the AS-IS capabilities including but not limited to scheduling and connectors of the current/new platform.

Migration Scope

On-Prem
→
On-Prem
Rehost
On-Prem
→
Cloud
Replatform
Cloud
→
Cloud
Rehost
Cloud
→
On-Prem
Replatform

Replatform means moving to a different type of platform, not just moving code. It requires some platform engineering work before migration can start. Aim for a minimal working setup first: enough target environment, access, and connectivity to begin migrating. Expand further as the migration progresses.

It is advisable to run a technology parity check early. This will help in assessing whether the AS-IS capabilities be supported on the TO-BE platform. Check SQL dialects, scheduling, connectors, data types, and any platform-specific features. Gaps found here are scope blockers: discovering them mid-execution is costly.

Define Success Metrics

What is the success criteria for migration? Define this upfront and get agreement from all stakeholders before work begins. Examples:

  • Output fidelity: 1-to-1 match for critical pipelines; 95–99% similarity acceptable for lower-priority ones.
  • Timeline: Hard cutover date (e.g. license expiry) and number of pipelines live by that date.
  • Scope completeness: % of pipelines migrated for the project to be considered done.
  • Stability window: How long new pipelines must run without incidents before legacy is decommissioned.

Migration Strategy

Most migration programs do not sit at one extreme. Strategy is usually a spectrum between Lift & Shift and Refactor, chosen per pipeline based on value, risk, and deadline pressure.

Lift & Shift Refactor
Lift & Shift

Move data pipelines and workflows as-is with minimal logic change.

Lower risk Predictable timeline Carries technical debt
Refactor

Reorganise workflows and adjust logic where needed; scope can evolve.

Higher risk Unclear timing Better long-term design
Lower Change Intensity Higher Change Intensity →
Phase 02

Governance

Align stakeholders, define roles, and set the rules of engagement.

People Involved

MA
Migration Architect
Essential to sell the idea to the business and get the onboarding of different teams.
PA
Platform Architect
Designs the target platform foundation (environment setup, access model, connectivity, and platform guardrails) so migration work can run safely and at scale.
DO
Data Owners
Important in unblocking accesses, laying down business definitions, and taking ownership in the new migrated platform.

System access is a critical early blocker. Getting read access to legacy systems (schemas, catalogs, pipelines) and write access to the target environment must be arranged early. Delays here stall everything downstream. Data Owners are key to unblocking this.

Bound by a Deadline?

  • Create a hard deadline if there are licenses that need to be decommissioned.
  • The deadline shapes your cutover approach: do you migrate in waves (Phased) or all at once (Big Bang)? This is independent of whether each pipeline is Lift & Shift or Refactor.

Phased Cutover

Migrate in waves. For example, 50% of pipelines by date X, the remainder by date Y. Lower risk, more time to course-correct between rounds.

Big Bang Cutover

Migrate everything in one go. All pipelines cut over at once. Faster, but leaves no room to course-correct.

Agree on Success Metrics

The success metrics defined in Vision are only meaningful once all stakeholders have explicitly signed off on them. Surface disagreements now, not during Validation when results are already in.

  • Walk through each metric with Data Owners and consumer teams. Do they agree that 95% similarity is acceptable for their dashboards?
  • Document the agreed thresholds and get written sign-off, even a shared doc or email thread counts.
  • Any metric not agreed on here becomes a negotiation point in Validation. Avoid that.

A shared doc with signed-off thresholds per pipeline (or pipeline group) removes ambiguity later. The Migration Architect owns this artefact.

Stakeholder Map & Communication Plan

Dashboard and API consumer teams are stakeholders too, not just the Data Owners. Identifying them early sets expectations, secures their bandwidth for the Validation phase (rewiring), and avoids last-minute surprises.

Responsible
Migration team: Doing the migration work
Accountable
Migration architect & Business sponsor
Consulted
Data owners, & Data teams: provide context during discovery
Informed
Consumer teams: Kept updated during execution, become active at rewiring
  • Identify all teams that own dashboards or APIs dependent on the legacy system. They need to be mapped here, not discovered during Validation.
  • Get manager-level buy-in from consumer teams so their engineers are available when rewiring begins.
  • Define a simple communication plan: who gets updates and who is the single point of contact per team. This can be owned by the Migration Architect.
  • Agree on the margin of error tolerance per deliverable upfront, so there is no negotiation when validation results come in.

Even a one-page stakeholder map and a recurring 30-minute weekly sync prevents Validation from turning into a coordination bottleneck. The cost of setting this up early is minimal compared to the cost of chasing alignment mid-execution.

Phase 03

Discovery & Inventory

Migration is 75% analysis, 25% execution.

People Involved

MA
Migration Architect
Leads the discovery process, coordinates access requests, and owns the dependency diagram and prioritisation output.
DO
Data Owners
Validate the inventory, confirm business value of pipelines, and provide context on undocumented logic or dependencies.
DE
Data Engineer
Writes and runs the scripts (or AI-assisted tooling) to extract dependencies from the codebase. Produces the dependency diagram and feeds findings into the prioritisation table.

Freeze Legacy Development

As soon as this project is in the picture, there should be complete agreement between all teams that no extra development in the legacy systems is allowed, i.e. only small bug fixes or improvements. If you start introducing data sources while the migration team is analysing, you might miss out on dependencies.

If a full freeze isn't realistic, put a change-control process in place instead: all legacy changes must be logged and reflected in the inventory before being merged. This adds overhead but prevents silent dependency drift.

Start from the End Goal

  • Deliverables are usually dashboards or APIs.
  • Which workflows and pipelines directly feed those deliverables?
  • Which data sources do those pipelines depend on?

Legacy-to-Target Source Mapping

A simple Excel sheet is more than enough for this exercise.
  • Between old and new systems, how do the data sources map to each other?
  • Any data sources that have been refactored? If so, the table structure has changed, how do they still map?

Pipeline Prioritisation Table

Start from your deliverables (dashboards, APIs), then identify which pipelines feed them, not the other way around. A pipeline that serves multiple deliverables inherits the priority of the highest-value one.
Workflows Deliverable Business Value Frequency Owner? Complexity Data Sources Success Criteria Strategy
Pipeline 1 Dashboard 1 High Daily Yes High Available 1-1 with legacy Lift & Shift
Pipeline 2 Dashboard 2 Low Weekly No Low Not Available 1-1 with legacy Lift & Shift
Pipeline 3 Dashboard 2 Medium Daily Yes Low Refactored 95% similarity Refactor
Business Value
  • Not easily measurable. Consider usage frequency or number of consumers. Is this an operational or analytical use case?
Owner Defined?
  • Strict: If no owner/sponsor found, exclude it from scope.
  • Lenient: Keep it in scope but chase ownership along the way.
Complexity
  • Define a simple metric such as lines of SQL code.
Success Criteria
  • Does it require 1-1 similarity with the legacy output? Or is 95-99% similarity acceptable?

Create a Dependency Diagram

Code access is required. You need read access to the legacy codebase, scripts, and pipeline definitions before you can map dependencies reliably. Arrange this in Phase 1 alongside system access.

Scripting is needed and AI can help. Do not rely on tribal knowledge alone. Use scripts or AI to parse the codebase and surface hidden dependencies you would otherwise miss.

  • Identify the nature of the workflows that need to be migrated (e.g. SQL-based code, SAS, Python, stored procedures, etc.).
  • List all dependencies between projects and sources.

Build the Priority List

The goal is to create a realistic timeline. If migration time exceeds it, the project scope may not be achievable.
  • Prioritise medium-to-high business value projects with low complexity.
  • Deprioritise complex, low-business-value projects.
QUICK WINS High Value · Low Complexity ★ START HERE STRATEGIC BETS High Value · High Complexity Plan carefully, phase if needed FILL-INS Low Value · Low Complexity Migrate when capacity allows DEPRIORITISE Low Value · High Complexity Consider decommissioning P2 P3 P1 COMPLEXITY → BUSINESS VALUE →
From the example table: P2 & P3 are quick wins P1 is a strategic bet (high value but high complexity)
Phase 04

Execution & Validation

Where the real work happens.

People Involved

DE
Data Engineers / Migration Team
Execute the migration by running AI agents for Lift & Shift, and writing or redesigning logic for Refactor projects. Also run validation scripts and investigate any discrepancies.
MA
Migration Architect
Provides oversight, unblocks technical decisions, and ensures the migration stays aligned with the strategy defined in Vision.
CT
Consumer Teams
Actively involved in the validation portion. They reconnect their dashboards and APIs to the new pipelines and confirm the output meets their expectations.

Development vs. Production Environments

Always migrate from the production legacy source. Production is the ground truth. Dev data is often incomplete, stale, or unrepresentative. Migrating from anything else risks missing logic or data that only exists in prod.

Always land in development environment first. Migrate into the target dev environment, validate, and only promote to production once the pipeline passes validation. This keeps production clean and gives you a safe rollback point.

For Lift & Shift Projects

Rely on AI Agents (Validator Agent & Migrator Agent) ~95% success rate
Migrator Agent

Writes to the new target. It takes the legacy code (SQL, SAS, etc.) as input, applies the source-mapping context whenever a FROM statement references a legacy table, and produces translated code that runs on the new platform ready for the Validator to verify.

Validator Agent

Reads the legacy source. It introspects the old database schema and existing SQL/ETL logic to understand what the pipeline currently produces. After the Migrator finishes, the Validator re-runs equivalent queries on both the legacy and new environments and flags any discrepancies acting as the automated QA step.

  • Provide the mapping of legacy data sources to new data sources as context.
  • Give the Validator agent access to the old database within the scope of every project.
  • Give the Migrator agent access to the new database within the scope of every project.
  • Define a clear goal: "Migrate X to Y. Give the source so the agent knows what table is targeted."
  • When working with sensitive data, you can sample a mock dataset to represent the underlying sources.

170,000 lines of SQL migrated over 8 months is proven. Done manually, the same scope would take a team of 3 engineers roughly 3 years. With agents handling the mechanical translation, human effort shifts to reviewing and fixing edge cases rather than line-by-line rewriting.

For Refactor Projects

  • Define the models needed and how they interact. Think about reusability.
  • Use LLMs to speed up the process as an assistant, not the decision-maker.

The data owner/architect should draw the final conclusions. Models must be human-understandable to be reusable.

Tips & Tricks

How to approach a pipeline: code-first vs. business-first?
  • Lift & Shift → code-first. Translate the SQL or ETL directly.
  • Refactor → business-first. Understand what the pipeline produces and why before touching code.
Automated validation: how do the Migrator and Validator agents compare old vs. new?
  • The Migrator agent translates legacy code to the new platform using the source mapping as context.
  • The Validator agent reruns equivalent queries on both environments and diffs the outputs. It applies the agreed thresholds from Governance (e.g. exact match on sensitive columns, 95% on others).
Data quality testing and documentation
Health checks

Verify no unexpected nulls, duplicate PKs, or out-of-range values.

Ongoing tests

Add dbt tests, Great Expectations checks, or SQL assertions so quality runs on every refresh.

Audit trail

Log what passed, what fell within the agreed margin, and what was waived.

Known differences

Document intentional deviations from legacy output. Each one needs explicit data owner sign-off.

Validation Steps

1
Schema Validation
Is the schema the same? Column names, types, nullability. Check everything at the structural level first.
2
Immediate Red Flags
A column that is completely NULL probably means a join failed. Catch obvious failures before investing in deeper checks.
3
Row Count Check
Once schema matches, check row counts, a simple but powerful metric to verify overall logic sanity.
4
Data Content Check
Spot-checking (10–20 rows) or invest in a reusable validation script.
Column-level comparison guidance
Abc
String columns

Is complete equality needed or allow a ±5–10% margin of error? Interpreting special characters might differ between legacy and new systems.

123
Numeral columns

Is complete equality needed or allow a ±5–10% margin of error?

Dates

Do the format differences matter?

Best practice: Flag sensitive columns as requiring complete equality; configure a margin of error on other tolerable columns.

Rewire Dashboards / APIs

  • Ensure expectations are clear and alignment exists between teams. They must be aware of any agreed margin of error in the dashboard output (e.g. 2%).
  • If dashboards require extra transformations, shift them left to the ETL project if time allows.
Phase 05

Decommission & Cutover

Start decommissioning legacy dashboards, stop legacy workflows, and archive everything cleanly.

People Involved

MA
Migration Architect
Owns the cutover plan, coordinates the final switch-over sequence, and makes the call to hard decommission legacy systems after the soft shutdown stability window has passed.
DO
Data Owners
Give final sign-off before legacy pipelines are stopped. They are accountable for confirming the new platform is the production source of truth.
CT
Consumer Teams
Monitor their dashboards and APIs post-cutover and report any unexpected outcomes. Their silence (no red flags) is the green light to decommission the corresponding legacy pipelines.
DE
Data Engineer
Executes the decommission to stop legacy workflows, archive the code, and ensures nothing is silently still running on the old platform.

Cutover Steps

  1. Switch over one deliverable at a time. As soon as a deliverable is migrated and validated, point it to the new platform, meaning the dashboard or API now reads from the new pipelines instead of the legacy ones. Do this one deliverable at a time and watch for unexpected outcomes before moving to the next.
  2. Soft shutdown first. Stop all legacy schedulers and pipeline triggers and do not schedule any new runs. Monitor for a stability window (typically 1 to 2 weeks) to confirm nothing is still depending on the old platform. Only once that window passes with no incidents should you hard decommission the servers. Skipping this risks destroying data that a downstream process was still quietly reading.
  3. Archive legacy code & documentation. Preserve the legacy pipeline code and orchestration configs, business definitions, and any documented logic for reference and auditing. Data remains governed independently in the source systems.

Knowledge Transfer

Platform architecture

Pipeline structure, orchestration configs, and how to run or debug from scratch.

Source mapping

The legacy-to-target table map is essential reference for anyone maintaining the new pipelines.

Known quirks & decisions

Design choices, logic deviations, and workarounds that are not obvious from the code.

Walkthrough session

At least one session with the teams taking over, preferably recorded.

The best test: can a new engineer reproduce a failing pipeline without asking the migration team? If yes, handover is done.