Data Pipeline Engineer-5
Job Type: Contract
Job Category: IT
Job Description
Job Title: Data Pipeline Engineer
Location: Remote
Contract
Job Summary:
We are seeking a highly skilled Data Pipeline Engineer to design, build, and optimize robust data pipelines and platforms supporting large-scale data analytics and processing workloads. The ideal candidate will have hands-on experience with Databricks, Azure SQL Server, PySpark, and modern ETL/ELT frameworks, coupled with a solid understanding of data modeling, containerization, and scalability best practices.
Key Responsibilities:
• Design, develop, and maintain data pipelines using Databricks (Python and PySpark) to support ingestion, transformation, and storage of large datasets.
• Implement ETL/ELT processes to extract data from various sources, ensuring data integrity, consistency, and accuracy across systems.
• Develop and optimize data models to support analytical and operational data needs.
• Build and manage Azure SQL Server instances including Managed Instances, Azure SQL Database, and SQL Server VMs for data storage and access.
• Apply data engineering best practices for performance tuning, scalability, and cost optimization across the data pipeline infrastructure.
• Utilize Docker and Azure Kubernetes Service (AKS) for containerized deployment and orchestration of data pipeline components.
• Collaborate with Data Scientists, Analysts, and Business Stakeholders to translate business requirements into technical solutions.
• Monitor and troubleshoot data pipeline performance, ensuring high availability, reliability, and data quality.
• Maintain thorough documentation of data architecture, data flows, and pipeline configurations.
• Drive automation initiatives to streamline deployment, testing, and data operations.
Required Skills & Qualifications:
• Proven experience with Databricks using Python, SQL, and PySpark.
• Strong knowledge of Azure SQL Server ecosystem — including Managed Instance, Azure SQL DB, and SQL Server VMs.
• Expertise in data modeling, ETL/ELT design patterns, and data integration frameworks.
• Solid understanding of containerization (Docker) and Kubernetes (AKS) for data engineering workloads.
• Experience in performance optimization and scalability in cloud data environments.
• Strong SQL development skills, including query optimization and stored procedure development.
• Hands-on experience with Azure Data Services (ADF, Synapse, Storage Accounts) is a plus.
• Familiarity with CI/CD pipelines for automated deployment of data solutions.
Required Skills
DEVOPS ENGINEER
SENIOR EMAIL SECURITY ENGINEER
Apply Now
Apply Now
Job Category: IT
Job Description
Job Title: Data Pipeline Engineer
Location: Remote
Contract
Job Summary:
We are seeking a highly skilled Data Pipeline Engineer to design, build, and optimize robust data pipelines and platforms supporting large-scale data analytics and processing workloads. The ideal candidate will have hands-on experience with Databricks, Azure SQL Server, PySpark, and modern ETL/ELT frameworks, coupled with a solid understanding of data modeling, containerization, and scalability best practices.
Key Responsibilities:
• Design, develop, and maintain data pipelines using Databricks (Python and PySpark) to support ingestion, transformation, and storage of large datasets.
• Implement ETL/ELT processes to extract data from various sources, ensuring data integrity, consistency, and accuracy across systems.
• Develop and optimize data models to support analytical and operational data needs.
• Build and manage Azure SQL Server instances including Managed Instances, Azure SQL Database, and SQL Server VMs for data storage and access.
• Apply data engineering best practices for performance tuning, scalability, and cost optimization across the data pipeline infrastructure.
• Utilize Docker and Azure Kubernetes Service (AKS) for containerized deployment and orchestration of data pipeline components.
• Collaborate with Data Scientists, Analysts, and Business Stakeholders to translate business requirements into technical solutions.
• Monitor and troubleshoot data pipeline performance, ensuring high availability, reliability, and data quality.
• Maintain thorough documentation of data architecture, data flows, and pipeline configurations.
• Drive automation initiatives to streamline deployment, testing, and data operations.
Required Skills & Qualifications:
• Proven experience with Databricks using Python, SQL, and PySpark.
• Strong knowledge of Azure SQL Server ecosystem — including Managed Instance, Azure SQL DB, and SQL Server VMs.
• Expertise in data modeling, ETL/ELT design patterns, and data integration frameworks.
• Solid understanding of containerization (Docker) and Kubernetes (AKS) for data engineering workloads.
• Experience in performance optimization and scalability in cloud data environments.
• Strong SQL development skills, including query optimization and stored procedure development.
• Hands-on experience with Azure Data Services (ADF, Synapse, Storage Accounts) is a plus.
• Familiarity with CI/CD pipelines for automated deployment of data solutions.
Required Skills
DEVOPS ENGINEER
SENIOR EMAIL SECURITY ENGINEER
Apply Now
Apply Now