ML Engineer – Experimentation Platform

Job Title: ML Engineer – Experimentation Platform
Experience: 3 – 4 Years

Location: Remote
Notice Period: Immediate Joiners Only

About the Role
We are looking for a highly skilled ML Engineer to join our Test & Learn Platform team. In this role, you will build and scale experimentation and causal inference services that enable business teams to make data-driven decisions globally.

You will work across statistical modeling, API development, cloud-native infrastructure, and large-scale data processing to deliver reliable and production-ready ML solutions.

Key Responsibilities

Develop and maintain statistical and machine learning modules for:

Difference-in-Differences (DID)

Synthetic Control

A/B Testing

Multi-Treatment Effects

Build and extend RESTful APIs using FastAPI and integrate them with web applications through SDK wrappers

Design and optimize large-scale data pipelines using PySpark, Delta Lake, and Azure Data Lake

Diagnose and resolve Out-of-Memory (OOM) issues in PySpark workloads by optimizing:

Memory allocation

Partitioning

Broadcast joins

Caching strategies

Spark configurations

Deploy and manage Databricks workloads including notebooks, job clusters, and Delta Lake tables

Containerize and deploy services using Docker, Kubernetes, and CI/CD pipelines

Ensure code quality, testing, and security using PyTest, SonarCloud, and Snyk

Collaborate closely with Data Scientists and Product teams to convert research concepts into scalable production systems

 

Mandatory Skills

Strong experience in Python (3.9+)

Hands-on expertise in:

PySpark & Spark Internals

Databricks

FastAPI / API Development

Azure Cloud Platform

Kubernetes & Docker

PyTest

Strong understanding of:

DID

Synthetic Control

A/B Testing

Hypothesis Testing

Panel Data Methods

Expertise in statistical and ML libraries:

statsmodels

scikit-learn

SciPy

Pandas

NumPy

Technical Requirements

PySpark & Spark Internals

Strong understanding of Spark memory model

Executor tuning and shuffle optimization

Diagnosing and resolving OOM errors

Experience with:

Broadcast thresholds

Partition skew handling

Spill-to-disk optimization

 

GC tuning

Databricks

Hands-on experience with:

Job orchestration

Cluster configuration

Notebook workflows

Delta Lake optimization

Z-ordering, compaction, and caching

Cloud & DevOps

Azure Storage, Azure ML, and Azure Data Lake

Docker-based containerization

Kubernetes orchestration for ML workloads

CI/CD pipeline integration

Testing & Quality

Unit and integration testing using PyTest

Familiarity with SonarCloud, Snyk, and GitHub Actions

Good-to-Have Skills

Experience with Celery and Redis for async task orchestration

Familiarity with Polars, PyArrow, or SQLAlchemy

Background in econometrics or experimental design

Experience with Spark UI profiling and performance benchmarking

Knowledge of advanced CI/CD tooling and automation practices

Preferred Candidate Profile

Strong analytical and problem-solving abilities

Ability to work independently in a remote setup

Excellent collaboration and communication skills

Passion for building scalable ML and experimentation platforms

Tech Stack

Languages & Libraries: Python, Pandas, NumPy, SciPy, statsmodels, scikit-learn
Big Data: PySpark, Spark Internals, Delta Lake
Cloud & Platforms: Azure, Databricks, Azure Data Lake
APIs & Backend: FastAPI
DevOps: Docker, Kubernetes, GitHub Actions
Testing & Security: PyTest, SonarCloud, Snyk