QA Automation Engineer – Enterprise Data & AI
Role Summary:
We are seeking a QA Automation Engineer / SDET to support quality engineering for our Enterprise Data Platform. This role will focus on validating data pipelines and extending existing data quality frameworks within Databricks, ensuring data accuracy, quality, completeness, and reliability across the data lifecycle. This role will play a key part in implementing, enhancing, and executing data validation and data quality checks across the medallion architecture.
Key Responsibilities:
Execute and extend existing automated tests within Databricks using PySpark, Python, SQL, and notebooks to validate data pipelines
This is complete Remote role - within the United States
Perform data reconciliation between source systems and target datasets
Validate ingestion processes including batch and incremental loads
Test transformations, joins, aggregations, and business rules for accuracy
Extend and enhance existing data quality frameworks and rule sets
Implement validation checks for data completeness, accuracy, consistency, and quality.
Validate thresholds, alerts, and exception handling mechanisms
Support tracking of data quality metrics and trends
Develop and maintain reusable and scalable test scripts aligned with existing frameworks
Integrate and execute tests within CI/CD pipelines (e.g., Azure DevOps)
Support testing activities across environments (QA, Staging)
Ensure consistent and reliable execution of automated tests
Partner closely with Data Engineers and Data Quality Engineers to identify, troubleshoot, and resolve data issues
Participate in Agile ceremonies and contribute to sprint deliverables
Support defect triage, root cause analysis, and retesting
Ensure data accuracy and consistency for downstream consumption and business reporting in data visualization tools such as Tableau.
Required Skills & Expertise:
5+ years of experience in data validation, QA engineering, or SDET roles
Hands-on experience with creating Databricks notebooks and data pipeline validation
Strong proficiency in PySpark, Python, SQL, and Databricks notebooks
Experience working with and extending existing data validation or data quality frameworks
Strong experience in data reconciliation and large-scale data validation
Experience executing tests within CI/CD pipelines (e.g., Azure DevOps)
Strong analytical and problem-solving skills
Experience with tools such as Azure Purview and Profisee MDM is preferred.
We are seeking a QA Automation Engineer / SDET to support quality engineering for our Enterprise Data Platform. This role will focus on validating data pipelines and extending existing data quality frameworks within Databricks, ensuring data accuracy, quality, completeness, and reliability across the data lifecycle. This role will play a key part in implementing, enhancing, and executing data validation and data quality checks across the medallion architecture.
Key Responsibilities:
Execute and extend existing automated tests within Databricks using PySpark, Python, SQL, and notebooks to validate data pipelines
This is complete Remote role - within the United States
Perform data reconciliation between source systems and target datasets
Validate ingestion processes including batch and incremental loads
Test transformations, joins, aggregations, and business rules for accuracy
Extend and enhance existing data quality frameworks and rule sets
Implement validation checks for data completeness, accuracy, consistency, and quality.
Validate thresholds, alerts, and exception handling mechanisms
Support tracking of data quality metrics and trends
Develop and maintain reusable and scalable test scripts aligned with existing frameworks
Integrate and execute tests within CI/CD pipelines (e.g., Azure DevOps)
Support testing activities across environments (QA, Staging)
Ensure consistent and reliable execution of automated tests
Partner closely with Data Engineers and Data Quality Engineers to identify, troubleshoot, and resolve data issues
Participate in Agile ceremonies and contribute to sprint deliverables
Support defect triage, root cause analysis, and retesting
Ensure data accuracy and consistency for downstream consumption and business reporting in data visualization tools such as Tableau.
Required Skills & Expertise:
5+ years of experience in data validation, QA engineering, or SDET roles
Hands-on experience with creating Databricks notebooks and data pipeline validation
Strong proficiency in PySpark, Python, SQL, and Databricks notebooks
Experience working with and extending existing data validation or data quality frameworks
Strong experience in data reconciliation and large-scale data validation
Experience executing tests within CI/CD pipelines (e.g., Azure DevOps)
Strong analytical and problem-solving skills
Experience with tools such as Azure Purview and Profisee MDM is preferred.