--- title: "Data Engineering Consulting | Smart Data" url: https://www.smartdata.net/services/data-engineering description: "Data engineering on Microsoft Fabric, Databricks and Snowflake: pipelines off SSIS and legacy ETL, warehouses that reconcile, quality and lineage built in." source: "Smart Data (smartdata.net). Markdown copy of the page; the HTML page is canonical." --- Modern data platforms # Data engineering consulting for analytics and AI We build the pipelines, warehouse layers and reporting that finance, operations and AI all read from, on Microsoft Fabric, Databricks or Snowflake. Old and new run side by side until the numbers match. Platforms we build on - [Microsoft Fabric Including Power BI](https://www.smartdata.net/services/microsoft-fabric-consulting) - [Databricks Lakehouse builds and migration](https://www.smartdata.net/services/databricks-consulting) - [Snowflake Architecture and optimization](https://www.smartdata.net/services/snowflake-consulting) Picked to fit the estate you already run. Where it starts ## Problems you’ve probably lived with Sales, finance and operations report three different revenue numbers. Each team builds its number from its own extract. We define it once, upstream, so every report reads the same one. Start with: the numbers people argue about, and where each one comes from. Our best analysts fix the same spreadsheet every month. The cleanup happens by hand because nothing upstream does it. We move it into the platform, where it runs on every load. Start with: the manual fixes that repeat every month. Every new report turns into another integration project. When each source lands once in the platform, a new report reads what is already there instead of needing its own pipeline. Start with: the sources behind your most requested reports. An ERP upgrade breaks every dashboard downstream. Dashboards wired straight to the ERP break when it changes. With layers in between, the change is absorbed once instead of in every report. Start with: the dashboards that read your ERP directly. Our data services ## Where most data engineering work begins Data engineering work usually starts from something already running: SSIS packages, Azure Data Factory pipelines, a SQL Server or Oracle warehouse and the reports built on them. Data architecture ### Data engineering and architecture We build in Bronze, Silver and Gold layers: a raw copy of every source, a layer where trust gets built and business-ready data that dashboards and AI read. Each transformation is built once and serves every consumer. - Modern data lakehouse and warehouse design - Real-time data architecture for streaming and event-based use cases - Batch and stream processing Tools we work in - Apache Kafka - Apache Spark - Airflow - dbt - Microsoft Fabric - Databricks Cloud data migrations ### Cloud and platform data migrations We move warehouses and pipelines to Azure, AWS or Google Cloud and run the old and new side by side until the numbers match, so reporting keeps running through the move. - Migration sequencing and cutover plans - Platform-specific migrations (AWS, Azure, GCP) - Hybrid and multi-cloud strategies - Azure Data Factory - Snowflake - SSIS Serverless data pipelines ### Serverless data pipelines For event-driven loads we build pipelines on Lambda, Step Functions, Azure Data Factory and Dataflow, so there are no servers to patch and compute is billed only while data is moving. - Right-sized workflows, billed per run - Cloud-native orchestration (AWS Lambda, Azure Data Factory, GCP Dataflow) - Infrastructure as Code (IaC) and CI/CD for data - AWS Lambda - Python Data quality and governance ### Data quality and governance Quality rules, lineage and access controls go into the pipelines themselves, so an audit question has an answer in the platform and a bad load is caught before it reaches a dashboard. - Validation, de-duplication and reconciliation checks on every load - Controls mapped to HIPAA and GDPR requirements - Metadata and lineage tracked from source system to report - Microsoft Purview - Unity Catalog - SQL and T-SQL - Data auditing Analytics and BI ### Analytics and business intelligence Power BI, Tableau or Looker on governed semantic models, so finance and operations read the same number from the same definition. - Embedded BI and reporting (Power BI, Tableau, Looker) - Predictive analytics and forecasting models - Executive dashboards and self-service analytics - Power BI If you’re not sure where to start, the first call sorts that out. [Talk to us about scope](https://www.smartdata.net/contact-us) How we build it ## Build it once and serve every consumer Land the raw data, clean it once and publish the version people and AI models can rely on. The pattern works the same on Microsoft Fabric, Databricks or Snowflake. For most companies already on Microsoft, Fabric is the affordable place to start. ### Bronze A raw copy of every source. Keep the evidence, so a mistake in your logic doesn’t become a permanent mistake in your history. ### Silver Where trust gets built: cleaned, checked and conformed into one definition of each number. ### Gold Business-ready data that dashboards, analysts and AI read. ### Where to start Pick your most valuable sources, stand up the layers and put the effort into Gold. One Gold use case in 90 days, not a full warehouse rebuild. How this connects to AI ## Not sure your data is ready for AI? The AI Enablement Guide shows how platform modernization connects to AI readiness. How we deliver ## From first review to a platform your team runs A defined path from understanding the estate to a platform your organization can use and support. ### 01Understand and design Review the estate and agree the architecture, first workload and responsibilities. Deliverable An architecture and first-workload plan, with owners named on both sides. ### 02Implement and validate Build the pipelines and models, and check the results against the business needs. Working pipelines and models, validated against the checks the business agreed. ### 03Operate and improve Hand the platform to your team, or keep our engineers on to support it. Runbooks and a documented handover, or an ongoing support plan. Industries ## Industries we work in ### Healthcare Patient, pharmacy and clinical data systems held to HIPAA, including where AI is allowed to touch them. - Medicaid programs - Pharmacy services - Clinical documentation [Healthcare](https://www.smartdata.net/industries/healthcare) ### Manufacturing and industrial ERP, shop-floor and supplier data on one platform, so plant reporting and finance reporting reconcile. - SAP and ERP reporting - Shop-floor integration - Supplier portals [Manufacturing and industrial](https://www.smartdata.net/industries/manufacturing) ### Retail and distribution Order, inventory, credit and customer data connected across portals, branches and the platforms behind them. - Vendor and customer portals - Order and credit systems - Data platforms [Retail and distribution](https://www.smartdata.net/industries/retail) ### Financial services Bank data platforms, reporting that holds up to an audit and managed support for the financial systems the business runs on. - Bank data platforms - Managed ERP support - Legacy migration [Financial services](https://www.smartdata.net/industries/banking-financial-services) ### Insurance Claims, member and policy systems for health plans and insurers. - Health plans - Claims data - Member apps [Insurance](https://www.smartdata.net/industries/insurance) Case studies ## Recent data engineering work 78% ### of legacy transformation types handled by the AI-assisted migration tool An AI-assisted tool that converts legacy ETL into tested Databricks pipelines. In a three-month first phase across 177 legacy jobs and scripts, it handled 78% of the legacy transformation types, with tests generated alongside every conversion. - ETL & Data Pipelines - Claude [of legacy transformation types handled by the AI-assisted migration tool](https://www.smartdata.net/case-studies/databricks-migration-national-retailer) ### SAP ECC 6.0 to Microsoft Fabric for a global industrial group Six operating entities on one SAP ECC 6.0 system, brought into Fabric with no RFC bulk extraction and entity-level security. The client's Basis and SAP teams found the process much easier than they expected. - SAP Integration - Lakehouse Architecture [SAP ECC 6.0 to Microsoft Fabric for a global industrial group](https://www.smartdata.net/case-studies/sap-ecc-microsoft-fabric-industrial) ### Unified data pipeline from three charging platforms to a Snowflake dashboard Charger and fleet data from Synop, Geotab and Ampcontrol unified through AWS serverless pipelines into Snowflake, behind a customer dashboard on live data. - React - TypeScript - AWS [Unified data pipeline from three charging platforms to a Snowflake dashboard](https://www.smartdata.net/case-studies/project-ev-data-pipeline) Taste of IT · November 18 ## See the medallion architecture live Chris St. Amand and Chad Puterbaugh on “Wrangling Your Data with the Medallion Architecture.” Bronze keeps every source raw, Silver builds trust and Gold feeds dashboards, analysts and AI, on Fabric, Databricks or Snowflake. Prizes in bronze, silver and gold. 21 days away A few practical questions ## Work through the decisions Questions we hear on first calls. We start with the business process, then the technology. Map data lineage and dependencies, define data quality rules before moving anything, build incremental pipelines so old and new systems can run in parallel, and validate at every stage with automated testing. We phase the work so each step is validated and can be rolled back. Data that is clean, governed, accessible and structured for AI to use: consistent formats across systems, documented lineage, quality monitoring with automated alerts and governed access. You can start before the data is perfect, as long as you know which data is trustworthy and build from there. It comes from three places: lower cost from consolidating redundant systems and automating manual work, recovered revenue ($16M+ in lost Medicaid revenue came back to a health plan after our team noticed a nursing facility file was mishandling records), and a shorter path from data to decision. Name the measurable outcomes before starting. It depends on your existing stack, your team and how you want to pay for compute. For most companies already on Microsoft, Fabric is the affordable place to start: engineering and Power BI on one capacity, with governance and security included. Databricks is the better fit for heavy engineering, streaming and machine learning, and works alongside Fabric when the work outgrows it. Snowflake works well for SQL teams that want little to administer and need to share governed data. We build on all three. Related services ## Where data engineering connects ### Data strategy Where your data stands, which platform fits and what to build first. [Data strategy](https://www.smartdata.net/services/data-consultancy) ### Data governance Quality monitoring, access controls and lineage tracking built into the platform you already run. [Data governance](https://www.smartdata.net/services/data-governance) ### AI development AI-assisted modernization, AI on your data and model features built into the applications you already run. [AI development](https://www.smartdata.net/services/ai-development) ### Managed services Monitoring, fixes and enhancements for data pipelines and the platforms under them, with P1 to P4 response tiers. [Managed services](https://www.smartdata.net/how-we-deliver/managed-services) Next step ## Which pipelines need to hold up next year? Tell us what feeds your reporting today and what has to change. Here is what happens after you reach out. ### A short call about your sources, pipelines and platform What runs today, where it breaks and what the business needs from it. ### An engineer looks at what’s running Scope comes from your systems as they are, before anything is priced. ### You get a written proposal Covering scope, team, approach and price. [info@smartdata.net](mailto:info@smartdata.net) [(937) 886-9166](tel:+19378869166)