Skip to content
Smart Data
Services
All services
Modern data platforms
Data strategy consultingData engineeringData governance and qualityPower BI and analytics on Fabric

Platforms we build on

Microsoft FabricDatabricksSnowflake
AI
AI-assisted modernizationAI on your dataAI developmentAI readiness workshop

Start here

AI Enablement GuideReadiness self-check
Applications
Application developmentLegacy modernizationERP and enterprise integrationAPIs and API security

ERP integration

SAP S/4HANA and ECC, NetSuite, Dynamics, Oracle, Epicor and others
How we work
Managed servicesScoped projectsStaff augmentationDirect placement

Where the work happens

Dayton, OhioSmart Data Global Center
Work
All case studiesNational distributor620+ known vulnerabilities eliminated across the customer portalsManaged servicesNuvei: Managed services for payments-to-ERP integrationsGoogleGoogle: A custom LMS for the internal sales team
Industries
HealthcareFinancial servicesInsuranceManufacturing and industrialRetail and distributionAll industries
Insights
BlogAI Enablement GuideEvents
Company
AboutCareersSmart Data Global CenterPartners
Talk to us about scope
info@smartdata.net(937) 886-9166

AI-assisted migration of legacy ETL to Databricks

Smart Data built an AI-assisted migration tool that converts legacy ETL workflows, orchestration scripts and deployment configurations into Databricks pipelines, with unit tests and infrastructure as code generated alongside every conversion.

Built onDatabricksClaude

  • 78%

    of legacy transformation types handled by the AI-assisted migration tool

Industry
Retail
Engagement
Data Platform
Timeline
Three-month first phase
Rows of red and orange light glowing in the dark

A major national retailer had committed to Databricks as its data platform. Standing between that decision and the outcome were hundreds of legacy ETL workflows and orchestration jobs, built up over years across enterprise domains, each one encoding business logic the company depends on every day.

Rather than staffing a large manual translation effort, Smart Data built an AI-assisted migration tool that converts legacy logic into Databricks pipelines on PySpark and generates the tests and deployment automation that make each conversion verifiable. A three-month first phase tested the approach against a real cross-section of the estate rather than a curated sample.

The challenge

The conventional path through a migration this size is manual translation: engineers read each legacy workflow, understand what it does and rebuild it on the new platform. At this scale, manual translation wouldn’t have scaled to the size of the estate, and four things stood in the way:

  • Hundreds of legacy ETL workflows and orchestration jobs spread across enterprise domains, each encoding business logic the company runs on daily
  • Manual translation could not scale across the estate
  • Drift, fatigue and inconsistent quality across a program measured in years rather than months
  • A platform decision already made, with migration cost threatening to stall it before the business case could be realized

Adding engineers would have scaled the cost of manual translation with the size of the estate, so we built a tool that converts the repeatable logic and leaves the special cases to senior engineers.

Grocery shelves stocked with oranges, apples and other fruit

The solution

Smart Data built an AI-assisted migration tool using Claude and structured prompts. The tool converts legacy ETL logic, orchestration scripts and deployment configurations into Databricks pipelines built on PySpark, generating unit tests and infrastructure as code alongside every converted pipeline.

Key implementation highlights

  • AI-assisted conversion of legacy ETL logic and orchestration scripts into PySpark pipelines on Databricks
  • Unit tests generated with every conversion, so each one arrives with the tests that check it
  • Infrastructure as code through Terraform and GitHub Actions, generated alongside the pipelines
  • A first phase scoped across 31 workflows, 36 mappings, 60 orchestration jobs and 50 shell scripts
  • Scope drawn from real production domains rather than a curated sample, to test the approach against the variety of a true enterprise codebase
  • Senior engineering effort redirected from writing translations by hand to validating output against known behavior

Without generated tests and deployment automation, AI conversion produces code a team still has to read line by line, and with them the platform team can review, run and verify the output.

The results

The first phase tested whether the approach would hold across the rest of the estate:

  • First phase delivered by a small team

  • Converted pipelines shipped with generated unit tests and infrastructure as code, so the client’s platform team could verify the output rather than take it on faith

This first phase is also where Smart Data’s Databricks partnership was earned. It showed that automated conversion, with tests generated alongside it, can move a legacy ETL estate onto Databricks pipelines on a timeline that keeps the business case intact.

Client

Industry

Retail and distribution

Platform and technology

  • Databricks
  • ETL & Data Pipelines
  • Claude
  • AI-Assisted Development
  • Terraform
  • CI/CD
  • Lakehouse Architecture

Related services

Databricks consultingAI-assisted modernizationData engineeringRetail and distribution

Working on something like this?

Talk to us about scope

Related case studies

All case studies

Working on something similar?

Tell us what you are building and what it has to connect to. Here is what happens after you reach out.

  1. A short call about the work

    What runs today, what needs to change and the date you are working toward.

  2. An engineer looks at what’s running

    Scope comes from your systems as they are, before anything is priced.

  3. You get a written proposal

    Covering scope, team, approach and price.

info@smartdata.net(937) 886-9166

Tell us what you are working on

We use your details only to reply.

Your privacy choices

Choose what this browser shares with us. The site works the same either way. Privacy policy