Senior Data Engineer (Azure + Databricks Lakehouse)
We are seeking a highly skilled Senior Data Engineer to join a high-performing data engineering team focused on building scalable data pipelines, integrations, and enterprise-grade data products that support analytics and operational use cases.
This role requires deep expertise in Azure cloud data engineering within a Databricks Lakehouse environment. The ideal candidate will collaborate with engineers, architects, analysts, and product managers to design and implement robust, scalable, and high-performance data solutions.
You will work with minimal supervision, exercise strong technical judgment, and proactively recommend and implement solutions aligned with business and technology goals.
Key Responsibilities
Data Engineering & Technical Delivery
• Design, develop, and maintain scalable data pipelines and ETL/ELT processes using PySpark and SparkSQL
• Build and manage orchestration workflows using Azure Data Factory
• Work with Azure Data Lake and cloud-based storage systems for large-scale data processing
• Implement streaming solutions using Kafka and/or Azure Event Hub
• Optimize data pipelines for performance, scalability, reliability, and cost efficiency
• Apply best practices for data partitioning, indexing, and storage formats such as Parquet
• Analyze DAGs and system performance to identify bottlenecks and improve efficiency
• Implement and maintain robust CI/CD pipelines using Azure DevOps
System Design & Architecture
• Contribute to data architecture design including HLDs, LLDs, and data models
• Understand system interactions, dependencies, and cross-platform data flows
• Build end-to-end data solutions across ingestion, transformation, and consumption layers
• Apply distributed computing concepts such as fault tolerance, idempotency, and scalability
• Identify opportunities to automate and optimize existing data processes
Cross-Functional Collaboration
• Partner with engineering, analytics, and product teams to deliver scalable technical solutions
• Translate business requirements into technical designs and implementation strategies
• Lead technical discussions and contribute to sprint planning and solution design
• Mentor junior engineers and support overall team development
Code Quality, Testing & Documentation
• Write clean, maintainable, and efficient code aligned with engineering standards
• Conduct code reviews and ensure adherence to best practices
• Develop and review unit tests and test plans
• Maintain technical documentation including architecture diagrams and process documentation
• Perform root cause analysis (RCA) and implement quality improvements
Project & Delivery Management
• Deliver assigned modules and user stories within timelines
• Support effort estimation, sprint planning, and release management activities
• Monitor delivery progress and ensure compliance with engineering standards
• Participate in deployment and production support processes
Innovation & Continuous Improvement
• Design and implement modern data engineering solutions and frameworks
• Evaluate emerging technologies and explore AI/ML and Agentic AI use cases
• Continuously improve systems for performance, scalability, and maintainability
• Operate effectively in fast-paced and evolving environments
Communication & Leadership
• Create clear technical documentation and presentations for stakeholders
• Communicate architecture decisions, implementation strategies, and technical processes
• Mentor engineers and contribute to knowledge-sharing initiatives
• Collaborate with stakeholders to clarify requirements and present solutions
Required Skills & Qualifications
Technical Skills
• Strong hands-on experience with the Azure Data Engineering ecosystem, including:
• Azure Data Factory
• Azure Data Lake
• Azure DevOps (CI/CD)
• Proficiency in:
• SQL (T-SQL, PostgreSQL)
• PySpark
• SparkSQL
• Experience with:
• Databricks Lakehouse architecture
• Kafka and/or Azure Event Hub
• Parquet and modern data storage formats
• Strong understanding of:
• Data partitioning and indexing
• Distributed computing principles
• Performance tuning and optimization of data pipelines
Professional Skills
• Strong analytical and problem-solving abilities
• Ability to work independently with minimal supervision
• Excellent communication and documentation skills
• Experience working in Agile environments (Scrum/Kanban)
• Ability to manage multiple priorities in fast-paced environments
Preferred Qualifications
• Experience designing end-to-end data platforms or lakehouse architectures
• Exposure to AI/ML or Agentic AI applications
• Prior experience mentoring or leading engineering teams
• Relevant Azure or Data Engineering certifications
Performance Expectations
• Deliver high-quality, scalable, and maintainable solutions
• Adhere to coding standards and engineering best practices
• Reduce defects and improve system performance
• Contribute to team knowledge sharing and continuous improvement initiatives
About Brickred Systems:
Brickred Systems is a global leader in next-generation technology, consulting, and business process service companies. We enable clients to navigate their digital transformation. Brickred Systems delivers a range of consulting services to our clients across multiple industries around the world. Our practices employ highly skilled and experienced individuals with a client-centric passion for innovation and delivery excellence.
With ISO 27001 and ISO 9001 certification and over a decade of experience in managing the systems and workings of global enterprises, we harness the power of cognitive computing hyper-automation, robotics, cloud, analytics, and emerging technologies to help our clients adapt to the digital world and make them successful. Our always-on learning agenda drives their continuous improvement through building and transferring digital skills, expertise, and ideas from our innovation ecosystem.
Apply tot his job
Apply To this Job
This role requires deep expertise in Azure cloud data engineering within a Databricks Lakehouse environment. The ideal candidate will collaborate with engineers, architects, analysts, and product managers to design and implement robust, scalable, and high-performance data solutions.
You will work with minimal supervision, exercise strong technical judgment, and proactively recommend and implement solutions aligned with business and technology goals.
Key Responsibilities
Data Engineering & Technical Delivery
• Design, develop, and maintain scalable data pipelines and ETL/ELT processes using PySpark and SparkSQL
• Build and manage orchestration workflows using Azure Data Factory
• Work with Azure Data Lake and cloud-based storage systems for large-scale data processing
• Implement streaming solutions using Kafka and/or Azure Event Hub
• Optimize data pipelines for performance, scalability, reliability, and cost efficiency
• Apply best practices for data partitioning, indexing, and storage formats such as Parquet
• Analyze DAGs and system performance to identify bottlenecks and improve efficiency
• Implement and maintain robust CI/CD pipelines using Azure DevOps
System Design & Architecture
• Contribute to data architecture design including HLDs, LLDs, and data models
• Understand system interactions, dependencies, and cross-platform data flows
• Build end-to-end data solutions across ingestion, transformation, and consumption layers
• Apply distributed computing concepts such as fault tolerance, idempotency, and scalability
• Identify opportunities to automate and optimize existing data processes
Cross-Functional Collaboration
• Partner with engineering, analytics, and product teams to deliver scalable technical solutions
• Translate business requirements into technical designs and implementation strategies
• Lead technical discussions and contribute to sprint planning and solution design
• Mentor junior engineers and support overall team development
Code Quality, Testing & Documentation
• Write clean, maintainable, and efficient code aligned with engineering standards
• Conduct code reviews and ensure adherence to best practices
• Develop and review unit tests and test plans
• Maintain technical documentation including architecture diagrams and process documentation
• Perform root cause analysis (RCA) and implement quality improvements
Project & Delivery Management
• Deliver assigned modules and user stories within timelines
• Support effort estimation, sprint planning, and release management activities
• Monitor delivery progress and ensure compliance with engineering standards
• Participate in deployment and production support processes
Innovation & Continuous Improvement
• Design and implement modern data engineering solutions and frameworks
• Evaluate emerging technologies and explore AI/ML and Agentic AI use cases
• Continuously improve systems for performance, scalability, and maintainability
• Operate effectively in fast-paced and evolving environments
Communication & Leadership
• Create clear technical documentation and presentations for stakeholders
• Communicate architecture decisions, implementation strategies, and technical processes
• Mentor engineers and contribute to knowledge-sharing initiatives
• Collaborate with stakeholders to clarify requirements and present solutions
Required Skills & Qualifications
Technical Skills
• Strong hands-on experience with the Azure Data Engineering ecosystem, including:
• Azure Data Factory
• Azure Data Lake
• Azure DevOps (CI/CD)
• Proficiency in:
• SQL (T-SQL, PostgreSQL)
• PySpark
• SparkSQL
• Experience with:
• Databricks Lakehouse architecture
• Kafka and/or Azure Event Hub
• Parquet and modern data storage formats
• Strong understanding of:
• Data partitioning and indexing
• Distributed computing principles
• Performance tuning and optimization of data pipelines
Professional Skills
• Strong analytical and problem-solving abilities
• Ability to work independently with minimal supervision
• Excellent communication and documentation skills
• Experience working in Agile environments (Scrum/Kanban)
• Ability to manage multiple priorities in fast-paced environments
Preferred Qualifications
• Experience designing end-to-end data platforms or lakehouse architectures
• Exposure to AI/ML or Agentic AI applications
• Prior experience mentoring or leading engineering teams
• Relevant Azure or Data Engineering certifications
Performance Expectations
• Deliver high-quality, scalable, and maintainable solutions
• Adhere to coding standards and engineering best practices
• Reduce defects and improve system performance
• Contribute to team knowledge sharing and continuous improvement initiatives
About Brickred Systems:
Brickred Systems is a global leader in next-generation technology, consulting, and business process service companies. We enable clients to navigate their digital transformation. Brickred Systems delivers a range of consulting services to our clients across multiple industries around the world. Our practices employ highly skilled and experienced individuals with a client-centric passion for innovation and delivery excellence.
With ISO 27001 and ISO 9001 certification and over a decade of experience in managing the systems and workings of global enterprises, we harness the power of cognitive computing hyper-automation, robotics, cloud, analytics, and emerging technologies to help our clients adapt to the digital world and make them successful. Our always-on learning agenda drives their continuous improvement through building and transferring digital skills, expertise, and ideas from our innovation ecosystem.
Apply tot his job
Apply To this Job