--- title: "Databricks Migration for a Major National Retailer | Smart Data" url: https://www.smartdata.net/case-studies/databricks-migration-national-retailer description: "Smart Data built an AI-assisted tool that converts legacy ETL into tested Databricks pipelines for a national retailer: 78% of transformation types handled." source: "Smart Data (smartdata.net). Markdown copy of the page; the HTML page is canonical." --- # AI-assisted migration of legacy ETL to Databricks Smart Data built an AI-assisted migration tool that converts legacy ETL workflows, orchestration scripts and deployment configurations into Databricks pipelines, with unit tests and infrastructure as code generated alongside every conversion. Built on[Databricks](https://www.smartdata.net/services/databricks-consulting)[Claude](https://www.smartdata.net/services/ai-assisted-modernization) 78% of legacy transformation types handled by the AI-assisted migration tool Industry Retail Engagement Data Platform Timeline Three-month first phase A major national retailer had committed to Databricks as its data platform. Standing between that decision and the outcome were hundreds of legacy ETL workflows and orchestration jobs, built up over years across enterprise domains, each one encoding business logic the company depends on every day. Rather than staffing a large manual translation effort, Smart Data built an AI-assisted migration tool that converts legacy logic into Databricks pipelines on PySpark and generates the tests and deployment automation that make each conversion verifiable. A three-month first phase tested the approach against a real cross-section of the estate rather than a curated sample. ## The challenge The conventional path through a migration this size is manual translation: engineers read each legacy workflow, understand what it does and rebuild it on the new platform. At this scale, manual translation wouldn’t have scaled to the size of the estate, and four things stood in the way: - Hundreds of legacy ETL workflows and orchestration jobs spread across enterprise domains, each encoding business logic the company runs on daily - Manual translation could not scale across the estate - Drift, fatigue and inconsistent quality across a program measured in years rather than months - A platform decision already made, with migration cost threatening to stall it before the business case could be realized Adding engineers would have scaled the cost of manual translation with the size of the estate, so we built a tool that converts the repeatable logic and leaves the special cases to senior engineers. ## The solution Smart Data built an AI-assisted migration tool using Claude and structured prompts. The tool converts legacy ETL logic, orchestration scripts and deployment configurations into Databricks pipelines built on PySpark, generating unit tests and infrastructure as code alongside every converted pipeline. ### Key implementation highlights - AI-assisted conversion of legacy ETL logic and orchestration scripts into PySpark pipelines on Databricks - Unit tests generated with every conversion, so each one arrives with the tests that check it - Infrastructure as code through Terraform and GitHub Actions, generated alongside the pipelines - A first phase scoped across 31 workflows, 36 mappings, 60 orchestration jobs and 50 shell scripts - Scope drawn from real production domains rather than a curated sample, to test the approach against the variety of a true enterprise codebase - Senior engineering effort redirected from writing translations by hand to validating output against known behavior Without generated tests and deployment automation, AI conversion produces code a team still has to read line by line, and with them the platform team can review, run and verify the output. ## The results The first phase tested whether the approach would hold across the rest of the estate: First phase delivered by a small team Converted pipelines shipped with generated unit tests and infrastructure as code, so the client’s platform team could verify the output rather than take it on faith This first phase is also where Smart Data’s Databricks partnership was earned. It showed that automated conversion, with tests generated alongside it, can move a legacy ETL estate onto Databricks pipelines on a timeline that keeps the business case intact. Client Retail and distribution Platform and technology - [Databricks](https://www.smartdata.net/services/databricks-consulting) - ETL & Data Pipelines - [Claude](https://www.smartdata.net/services/ai-assisted-modernization) - AI-Assisted Development - Terraform - CI/CD - Lakehouse Architecture ## Working on something similar? Tell us what you are building and what it has to connect to. Here is what happens after you reach out. ### A short call about the work What runs today, what needs to change and the date you are working toward. ### An engineer looks at what’s running Scope comes from your systems as they are, before anything is priced. ### You get a written proposal Covering scope, team, approach and price. [info@smartdata.net](mailto:info@smartdata.net)[(937) 886-9166](tel:+19378869166)