Data Engineer · ex-Full-Stack SWE

I own data end‑to‑end — from the app that emits the event, to the lakehouse that models it, to the dashboard that explains it.

Full-stack engineer turned Data Engineer. I speak both languages, so I build data platforms engineers actually want to use — tested, versioned, and built to not break at 2 a.m.

Let’s talk Download résumé

Microsoft Certified Fabric Data Engineer (DP-700) 5+ years · healthcare, manufacturing & pharma

Production software at scale

200K+ users

A lab-analytics data warehouse serving users globally — real production load, not a demo.

Modern data-platform depth

8s → 1.5s

Real-time streaming ingestion of PLC/sensor telemetry over MQTT/OPC UA into a medallion lakehouse.

Engineering discipline

95%+ coverage

CI/CD, TDD, and zero-downtime migrations — cutting production incidents by 35%.

Signature work

Case studies you can click through

Three platforms, three angles — data in motion, scale & rigor, and vendor-neutral proof. Walk each pipeline a stage at a time, from source to serve.

Data in motion Microsoft Fabric Streaming Bytebridge Technologies · Client: Johnson & Johnson · 2025–Present

Real-time manufacturing telemetry lakehouse

High-throughput PLC and sensor data from the factory floor had ~8 seconds of lag before it was usable — too slow to catch equipment failures as they happened. I built the streaming path end-to-end, from the edge protocol to the Power BI dashboard.

Click a stage — data flows left to right

01 · Source

High-throughput edge telemetry

PLC and sensor data streams off the factory floor over industrial protocols. I captured it at the source over MQTT and OPC UA, the standard machine-to-machine transports — the raw signal every downstream metric depends on.

MQTTOPC UAPLC / sensors

1000s / s

events ingested per second

02 · Ingest

Fabric Eventstreams, made reliable

I architected the real-time ingestion path in Microsoft Fabric Eventstreams with structured retry and dead-letter-queue patterns, so a bad message never stalls the stream. This is where the headline win happened: end-to-end latency dropped from ~8s to under 1.5s.

Fabric EventstreamsRetry + DLQReal-Time Intelligence

8s → 1.5s

data latency, end to end

03 · Lakehouse

Medallion architecture on OneLake

Streaming edge data lands in a Fabric Lakehouse on a Bronze → Silver → Gold medallion architecture. I wrote the PySpark notebooks that transform raw signal into clean, query-ready Delta Parquet — each layer testable and reprocessable on its own.

PySparkDelta ParquetMedallion B/S/GOneLake

3-layer

Bronze · Silver · Gold

04 · Orchestrate

Scheduled + event-driven automation

I engineered orchestration in Fabric Data Factory — both scheduled and event-driven — to automate validation reports and log parsing. The pipelines self-monitor and recover, hitting a 92% automated task success rate without manual babysitting.

Fabric Data FactoryEvent-driven triggersData quality alerts

92%

automated task success rate

05 · Serve

Real-time OEE & anomaly dashboards

The gold layer feeds Power BI in Direct Lake mode — live OEE and anomaly-detection dashboards reading straight from OneLake with no import step. They surface recurring equipment-failure patterns to the people on the floor, in near real time.

Power BI Direct LakeOEE metricsAnomaly detection

<1.5s

from sensor to dashboard

Scale & rigor SQL Server .NET Eurofins IT Solutions · Healthcare LIMS · 2021–2023

Healthcare LIMS data platform for 200K+ users

A lab-information platform served 200K+ users globally, but reports needed manual Excel extracts and the whole thing sat on one giant coupled database. I re-platformed it — a modeled warehouse, domain decomposition with zero downtime, and the engineering discipline to keep it stable under load.

Click a stage — data flows left to right

01 · Ingest

High-performance .NET ETL

Lab feeds arrived as complex XML and JSON. I built .NET ETL with schema validation and ActiveMQ messaging that transformed them reliably — cutting pipeline processing time by 38% and preventing the data loss the old flow was prone to.

C# / .NETXML / JSONActiveMQSchema validation

−38%

pipeline processing time

02 · Model

Snowflake-schema warehouse, tuned

I designed a snowflake-schema data warehouse on SQL Server serving 200K+ users, centralizing sample trends that used to need manual extracts. Then I tuned it — date-based fact partitioning, composite indexes, and refactoring legacy nested queries into CTEs — for 60% faster reports.

SQL ServerSnowflake schemaPartitioningCTEs · indexes

60% faster

consumer-facing reports · 200K+ users

03 · Decompose

Monolith → domain-driven stores

I migrated a monolithic LIMS database into 6 decoupled, domain-driven data stores (Results, Reports, Master Data). The migrations were staged and zero-downtime, running in parallel states so the platform never went dark for its users.

Domain-driven designZero-downtime migrationParallel run

6 stores

from 1 monolith · 0 downtime

04 · Serve

Self-serve analytics dashboards

I delivered Angular + Apache ECharts dashboards for sample-trend, contract-performance, and turnaround-time metrics — with PNG/SVG/PDF/CSV/Excel export and lazy-loaded chart modules — replacing 6 manually maintained Excel reports.

AngularApache EChartsMulti-format exportLazy loading

6 → 0

manual Excel reports retired

05 · Ship

CI/CD and test discipline

I established CI/CD (Jenkins, Octopus, Git) with TDD (XUnit, 95%+ coverage), peer-review gates, and runbooks across a 5-engineer team — reducing production incidents 35% and new-hire ramp-up 30%.

JenkinsOctopus DeployXUnit · TDDRunbooks

95%+

test coverage · −35% incidents

Vendor-neutral dbt Core Prefect Modern data stack · open-source ELT

Open-source ELT for public healthcare data

Proof that the rigor travels beyond any one vendor. A fully open-source stack — Prefect and dbt Core — turning raw public Medicare data into tested, documented, analytics-ready models, exactly the way I'd build it on any cloud.

Click a stage — data flows left to right

01 · Source

Raw public healthcare datasets

The pipeline ingests public Medicare / healthcare datasets in their raw form — the messy, real-world input any analytics layer has to be resilient to before it can be trusted.

Medicare dataRaw ingestCSV / flat files

raw

public Medicare / CMS data

02 · Orchestrate

Prefect for end-to-end ELT

I used Prefect to orchestrate the full ELT — with explicit dependency mapping and failure alerts, so a broken upstream task is caught and surfaced instead of silently corrupting everything downstream.

PrefectDependency mappingFailure alerts

Prefect

alerted, dependency-aware ELT

03 · Transform

dbt Core — modular SQL, raw → gold

Transforms are modular SQL models in dbt Core, promoting data from raw through staging to gold. Each model is small, version-controlled, and independently testable — the same software discipline I bring to application code.

dbt CoreModular SQLraw → staging → gold

raw→gold

layered, version-controlled models

04 · Test

Data-quality tests & schema validation

Every model carries dbt data-quality tests and schema validation, so bad data fails the build instead of reaching a dashboard. Quality is a gate in the pipeline, not a manual afterthought.

dbt testsSchema validationBuild-time gates

tested

quality enforced at build time

05 · Document

Auto-documented DAGs & lineage

dbt generates auto-documented DAGs — a browsable lineage graph of how every gold model is built from source. New teammates can see the whole pipeline without reading a line of code.

Auto-doc DAGsLineage graphSelf-serve docs

auto-doc

lineage from source to gold

Also in the lab

AWS · Healthcare MLJan–Apr 2025

Cloud Diabetes Prediction

A serverless ML pipeline on AWS — S3, API Gateway, Lambda, and SageMaker (XGBoost) — with sub-second inference and CI/CD via AWS CodeBuild, trained on the PIMA dataset.

85% accuracy80% recallSageMaker XGBoost

Data warehousingAug–Dec 2023

Formula 1 Relational Data Warehouse

A normalized schema across 8 entities spanning 70+ years of racing data, tuned with composite indexing and execution-plan analysis, surfaced through a Tableau dashboard.

8 entities · 70+ yrs20% query tuningTableau

Like the way I think about pipelines? Let's talk about yours.

Start a conversation

Skills

The pipeline, and what I reach for at each stage

Concept first, tools second. Here's the shape of a data platform — source to serve — and the stack I use to build each part.

01
Source

MQTTOPC UAEvent HubsXML / JSONAPIs

02
Ingest

EventstreamsData FactoryAirflowPrefect.NET ETL

03
Lakehouse

OneLakeADLS Gen2Delta ParquetMedallionS3

04
Transform

PySparkSpark SQLdbtpandasSCD · CDC

05
Serve

Power BIDirect LakeKQL / RTIWarehouseTableau

Languages & querying

SQLT-SQLPL/SQLPythonPySparkpandas · NumPyC# / .NET

Modeling & architecture

MedallionStar / snowflakeDimensionalSCD · CDCPartitioningMicroservices

Streaming & real-time

EventstreamsEvent HubsMQTT / OPC UAKQL / RTIDelta Parquet

Orchestration & DevOps

AirflowPrefectData FactoryJenkinsAzure DevOpsDocker · Git

Cloud platforms

Microsoft FabricAzure (ADF · Synapse · Databricks)AWS (S3 · Lambda · Glue · SageMaker)

DataOps & governance

Dev/QA/Prod promotionData-quality alertsSchema-drift detectionRLS / CLSSOC 2 / HIPAA aware

Experience

Five years shipping data & software

From full-stack engineering to data platforms — the same instinct throughout: build it to last, measure everything.

Sept 2025 – Present Bytebridge Technologies Client: Johnson & Johnson · Remote, US

Data Engineer — Digital Manufacturing & Agentic AI

Architected an end-to-end real-time ingestion pipeline in Microsoft Fabric (Eventstreams over MQTT/OPC UA), cutting data latency from ~8s to under 1.5s.
Designed a Fabric Lakehouse on a medallion architecture, processing streaming edge data into Delta Parquet via PySpark with retry and dead-letter-queue patterns.
Engineered scheduled + event-driven orchestration in Fabric Data Factory, achieving a 92% automated task success rate, and built real-time OEE / anomaly dashboards in Power BI Direct Lake.

Apr 2021 – Aug 2023 Eurofins IT Solutions Bangalore, India

Senior Software Engineer — Healthcare LIMS Data Platform

Designed a snowflake-schema data warehouse on SQL Server serving 200K+ users globally, and accelerated consumer-facing reports by 60% via partitioning, composite indexes, and CTE refactoring.
Migrated a monolithic LIMS database into 6 decoupled domain-driven data stores with staged, zero-downtime migrations; built high-performance .NET ETL that cut processing time 38%.
Established CI/CD (Jenkins, Octopus, Git) with TDD (XUnit, 95%+ coverage) and runbooks — reducing production incidents 35% across a 5-engineer team.

Apr 2019 – Mar 2021 Pratian Technologies Bangalore, India

Software Engineer — Data Pipelines & Automation

Built production ETL transforming raw XML/JSON ERP data into normalized relational models, reducing manual data-entry workloads by 75% and cutting processing time 38%.
Developed metadata-driven ingestion validators and async processing (RxJS, NgRx, Polly retry/circuit-breaker), achieving 99.7% uptime on customer-facing modules.
Integrated automated testing gates in Jenkins CI/CD, reducing regression defects 35%.

Education

M.S., Data ScienceUniversity of Michigan – Dearborn · 2023–2025
B.Tech, Electronics & CommunicationGITAM University, India · 2014–2018

Certifications

Microsoft Certified: Fabric Data Engineer Associate (DP-700)
Microsoft Certified: Power Platform Developer (PL-400)
Microsoft Certified: Azure Data Fundamentals (DP-900)
Salesforce Agentforce Specialistagentic AI workflows

Contact

Let’s build a data platform that doesn’t break at 2 a.m.

I’m open to Data Engineering roles at product-based companies — and open to relocate. The fastest way to reach me is email or a quick call.

Email me Book a call Résumé (PDF)

alluri@umich.edu LinkedIn GitHub rahulalluri.tech