[Remote] Data Engineering Lead

Note: The job is a remote job and is open to candidates in USA. MediaRadar, Inc. is a company that provides marketing, sales, and analytics leaders with critical intelligence through its AI-enabled platform. The Data Engineering Lead will be responsible for designing scalable systems, integrating Machine Learning models into ETL pipelines, and leading a team towards agile practices while maintaining high performance and cost-efficiency.ResponsibilitiesArchitect and implement complex, end-to-end data pipelines using Azure Databricks and PySparkOptimize Apache Spark jobs, tune Databricks units, and define cluster policies to minimize compute costsDevelop a proactive monitoring and alerts framework to ensure 99.9% reliability and mitigate system issues before they impact end-usersArchitect and design high-performance schemas in PostgreSQL, managing indexing, partitioning, and optimizing complex analytical queriesLead a lean team toward cross-trained agility, moving away from 'siloed specialists'Anticipate future data needs and design High-Velocity Architecture that is highly scalable and manageable to handle sudden volume increasesCollaborate with ML teams to integrate automated model orchestration into robust ETL pipelinesCollaborate with the offshore team lead to facilitate seamless knowledge transfer and operational continuity across time zonesSkillsCore: Python, PostgreSQL + pgvectorBig Data: Azure Databricks, PySpark, Delta LakeDevOps: Docker, Git, Azure DevOps, CI/CD10+ years of experience in Data or Software Engineering with deep codebase involvement3+ years as a Technical Lead managing agile teamsProven ability to lead lean, high-impact teams while maintaining high individual outputExperience with cross-training advocacy and scaling data processing through automationWorkflow Orchestration: Experience with Apache AirflowContainerization: Familiarity with Azure Kubernetes Service (AKS)Company OverviewMediaRadar, now including the data and capabilities of Vivvix, powers the mission-critical marketing and sales decisions that drive competitive advantage. It was founded in 2006, and is headquartered in New York, New York, USA, with a workforce of 501-1000 employees. Its website is http://www.mediaradar.com.Company H1B SponsorshipMediaRadar, Inc. has a track record of offering H1B sponsorships, with 1 in 2026, 1 in 2025, 2 in 2024, 2 in 2021. Please note that this does not guarantee sponsorship for this specific role.