Senior Data Engineer Consultant – Data Bricks, Technology Consulting

Job Description: • Design and implement enterprise-scale data architectures, including data lakes, warehouses, and real-time streaming platforms. • Develop and maintain ETL/ELT pipelines that efficiently process large volumes of structured and unstructured data from diverse sources. • Ensure data quality, governance, and security standards are embedded throughout all data engineering processes. • Hands-on development of Databricks notebooks using PySpark, Python, and SQL for ETL automation. • Create and optimise PySpark scripts for efficient data extraction, transformation, and loading from large datasets. • Implement custom data manipulation, validation, and error handling solutions to enhance ETL robustness. • Provide technical mentorship to junior data engineers and analysts. • Lead code reviews, establish best practices, and drive adoption of modern data engineering tools and methodologies. • Collaborate with cross-functional teams including data scientists, analysts, and software engineers to deliver integrated solutions. • Monitor and optimise data pipeline performance, implementing solutions for scalability and cost-effectiveness. Requirements: • 5-7 years of data engineering experience with AWS-native implementations • Proven experience working as part of a data engineering team on ETL migration projects • Hands-on experience implementing Databricks notebooks using PySpark, Python, and SQL for ETL automation • Demonstrated expertise developing PySpark scripts for efficient data extraction, transformation, and loading from large datasets • Experience utilising Python for custom data manipulation, validation, and error handling in ETL processes • Proficiency employing SQL for complex joins, aggregations, and database operations within Databricks environments • Track record of testing, debugging, and optimising data transformation processes for accuracy and performance • Experience verifying data integrity throughout pipeline stages and resolving troubleshooting issues • Proven ability collaborating with cross-functional teams to align ETL migration tasks with project goals and deadlines • Experience in government sector data projects and compliance requirements (preferred) • Experience in HR domain including workforce analytics, payroll systems, and employee data management (preferred) • Experience providing technical mentorship to junior team members (preferred) • Certified in: AWS Certified Data Engineer - Associate (preferred) • AWS Certified Solutions Architect or AWS Certified Big Data - Specialty (preferred) • Databricks Certified Data Engineer Associate or Professional • Amazon Redshift or Snowflake certifications • Agile/Scrum Master certification (preferred) Benefits: • Health insurance • 401(k) matching • Flexible work hours • Paid time off • Remote work options Apply tot his job