Azure Databricks Engineer
About the position The Azure Databricks Engineer will design, build, and optimize scalable data pipelines and lakehouse solutions using Azure Databricks. This role is critical for enabling advanced analytics and supporting machine learning initiatives. This role requires US Citizenship. The preferred location for this role is Lafayette, LA, however this position can also be filled out of Belton, TX, Lebanon, VA or Knoxville TN. (Hybrid role) Responsibilities • Develop and maintain ETL/ELT pipelines using Databricks workflows, Delta Lake, and Apache Spark. • Architect and optimize data lakehouse solutions for efficient storage and retrieval. • Configure and manage Databricks workspaces, clusters, and implement security and cost optimization strategies. • Build real-time streaming and batch processing solutions for diverse data sources. • Collaborate with data scientists to deploy ML models and automate ML pipelines using MLflow. • Monitor pipeline performance, troubleshoot issues, and optimize Spark jobs for efficiency. Requirements • BA/BS/MS in Computer Science, Engineering, or Data Science. • 5+ years as a Data Engineer, with 3+ years in Azure Databricks. • Expertise in Apache Spark, Delta Lake, Python, SQL, Scala. • Proficiency in Azure Data Factory, Data Lake Storage, Synapse Analytics. • Strong understanding of ETL/ELT, data modeling, and big data technologies. Nice-to-haves • Certifications: Microsoft Certified: Azure Data Engineer Associate (DP-203) or Databricks Certified Data Engineer. • Experience with Agile methodology, data governance, and security. Benefits • Competitive compensation • Comprehensive insurance options • Matching contributions through the 401(k) plan and the share purchase plan • Paid time off for vacation, holidays, and sick time • Paid parental leave • Learning opportunities and tuition assistance • Wellness and Well-being programs