Back to Talent
Available Now
SS
Shivam Shukla
10+ years architecting large-scale data & ML pipelines on Databricks, Delta Lake, and Apache Spark. Subject Matter Expert in enterprise data lakehouse and MLOps delivery.
Professional Summary
- Overall 10+ years architecting and optimizing large-scale data processing pipelines within Azure Data Lakehouse environments using Databricks, Delta Lake, and Apache Spark.
- Proven Subject Matter Expert (SME) in configuring and managing Databricks pipelines for performance, scalability, and reliability across diverse enterprise use cases.
- Extensive hands-on expertise in cloud-native infrastructure (Azure, GCP, AWS), microservices architecture, container orchestration, and infrastructure-as-code (IaC).
- Proficient in DevOps tooling on Azure and scripting in Python for automation and deployment.
- Demonstrated leadership in building and migrating scalable data and ML platforms to cloud environments for FinOps, AI/ML, and analytics use cases.
- Adept in Unity Catalog migrations, CDC pipeline architecture, MLOps, and end-to-end LLM and data science project delivery.
- Experienced in designing organization-wide observability systems and enterprise-scale Customer Data Platforms (CDP), including AIOps-based health checks using large language models.
Work History
Lead Software Engineer — Data & ML (Senior Manager)
FCS Software Solution
- Architecting large-scale data processing pipelines on Azure Databricks, integrating with Azure Data Lake Gen2, Delta Lake, and Unity Catalog for secure, governed, scalable data management.
- Designing and optimizing Delta Live Tables (DLT) pipelines using Python, PySpark, and SQL for real-time and batch data transformation across FinOps datasets.
- Building reusable CDC-based DLT Python modules integrated with Terraform and GitOps workflows, reducing pipeline creation time by over 50%.
- Migrating legacy pipelines from Apache Hive and traditional Spark clusters to Databricks Unity Catalog for finer governance and Delta Sharing across business units.
- Implementing Databricks MLflow to track experiments and streamline deployment of ML models via Azure ML and AKS.
- Porting ML workloads (cost forecasting, anomaly detection) from AKS to Databricks Spark MLlib, reducing training time by 60% via distributed compute.
- Integrating data from Snowflake, AWS S3, and Salesforce into Delta Lake using DLT, Auto Loader, and Lakeflow Connect for unified analytics.
- Leading a multi-cloud data lakehouse architecture on Databricks (Azure + GCP) for FinOps workloads, saving over $500K annually vs. third-party platforms.
- Building real-time monitoring pipelines for billing anomalies integrated with AIOps via REST APIs and Azure Event Hubs.
- Deploying drift detection pipelines using feature store monitoring to trigger automatic retraining workflows.
- Creating and maintaining “Genie Agent,” a GenAI assistant on Databricks that indexes internal knowledge base documents for contextual Q&A.
- Leading Skylight, an in-house FinOps platform on Databricks that has enabled over $1M in annualized cost savings via Cost Optimization Patterns.
Senior Databricks Engineer
Publicis Media
- Led development of a Customer Data Platform (CDP) for a leading CPG client, integrating LiveRamp, Gigya, and Kafka to enhance RFM segmentation.
- Architected CDC pipelines using Debezium and Databricks within Azure Data Lakehouse, achieving 30% latency reduction.
- Built a Marketing Mix Modeling (MMM) solution on Databricks, improving marketing reach by 10% through cross-channel attribution.
- Developed containerized microservices for data transformation on AKS, and automated pipeline monitoring using Python.
- Enforced best practices for security, cost, and scalability across ADLS, Azure Key Vault, and Azure Monitor.
Senior Data Engineer
Coforge Tech Ltd.
- Designed a scalable data & ML pipeline framework on Azure using ADF, ADLS, Azure Functions, Synapse, and Databricks AutoML.
- Migrated complex on-premise ETL and ML workflows to Azure and GCP cloud-native architectures.
- Provisioned Azure Databricks Workspaces and configured cluster policies for collaborative data teams.
- Delivered POCs integrating Logic Apps, Azure ML, and Databricks for low-code orchestration and automated model deployment.
- Implemented AKS-based microservices for containerized model inference within the pipeline.
Senior Associate
Nagarro
- Integrated legacy AS400 systems with Azure Data Lake for downstream analytics ingestion.
- Built and deployed a CNN model on Databricks to estimate car damage repair costs for insurance automation.
- Migrated legacy email systems to a cloud-native AWS solution using Route 53, Lambda, SES, and SNS.
- Orchestrated batch and streaming workflows using Databricks Autoloader and Delta Lake with ACID compliance.
Programmer
Fidelity International
- Supported the GFAS record-keeping platform for European mutual fund clients with focus on data integrity within an Azure-hosted architecture.
- Contributed to GDPR compliance by developing “Right to be Forgotten” and AML use cases using Databricks for data tracing.
- Developed reusable Python scripts to automate data validation, transformation, and anonymization.
System Engineer
Tata Consultancy Services
- Built Marketing Mix Models (MMM) using Python, SQL, and Spark to measure and optimize marketing channel impact on sales.
- Developed PySpark data pipelines for ingesting and cleansing large marketing and sales datasets.
- Re-architected AS400 batch jobs to Spark-based distributed workflows, significantly reducing runtime.
Notable Achievements
$1M+
Annualized cost savings delivered through the Skylight FinOps platform
$500K
Saved annually by replacing third-party FinOps tools with multi-cloud Databricks lakehouse
60%
Reduction in ML training time via distributed compute migration to Databricks
50%
Faster pipeline creation using reusable CDC-based DLT Python modules