QA Engineer – AI Automation Validation
Job Description:
• Develop comprehensive test strategies: Create and maintain test plans covering functional, performance, and regression testing for AI agents and automation workflows.
• Build and maintain automated test frameworks: Develop and maintain automated test scripts/frameworks to ensure bot responses and workflow outputs remain accurate, consistent, and reliable over time.
• Build and maintain test assets: Create/curate golden conversation sets, synthetic test data, evaluation rubrics, and regression suites to measure quality over time.
• Validate AI agent safety and reliability: Test tool-calling end-to-end (tool selection, parameter mapping, sequencing, retries/timeouts, error handling, and recovery), including negative/edge cases; confirm outputs align to business rules and document risk mitigations.
• Test integration/workflow integrity: Validate API contracts, data transformations, and downstream impacts (idempotency, duplicate prevention, partial failures, and latency) across environments.
• Manage defects through resolution: Identify, log, prioritize, and track defects; partner with developers to reproduce issues and validate fixes.
• Provide release certification: Deliver final quality sign-off prior to production release, ensuring defined acceptance criteria and quality standards are met.
• Align on acceptance criteria: Partner with Business Analysts to ensure user stories include clear, testable acceptance criteria and that test coverage maps to requirements.
• Document testing and metrics: Maintain test cases, data, results, and quality metrics; communicate outcomes and risks.
• Participate in Agile ceremonies: Engage in planning, grooming, standups, and retrospectives to embed quality throughout delivery.
• Drive continuous improvement: Identify opportunities to improve automation coverage, testing effectiveness, and quality practices.
Requirements:
• 3+ years in a QA, test automation, or software testing role with demonstrated experience developing and executing test strategies.
• Test automation expertise: Strong hands-on experience building automated test frameworks using tools such as Selenium, Playwright, Cypress, Jest, Pytest, or similar frameworks.
• Scripting and programming skills: Proficiency in at least one programming or scripting language (Python, JavaScript, Java, C#) for test automation development.
• Agile methodology: Direct experience working in an Agile/Scrum environment with hands-on involvement in sprint cycles and continuous integration.
• Defect management tools: Experience using bug tracking and test management tools such as Jira, Azure DevOps, TestRail, or similar platforms.
• Bachelors level degree is preferred, but not required.
Benefits:
• Health insurance
• 401(k) matching
• Flexible work hours
• Paid time off
• Remote work options
Apply tot his job
Apply To this Job
• Develop comprehensive test strategies: Create and maintain test plans covering functional, performance, and regression testing for AI agents and automation workflows.
• Build and maintain automated test frameworks: Develop and maintain automated test scripts/frameworks to ensure bot responses and workflow outputs remain accurate, consistent, and reliable over time.
• Build and maintain test assets: Create/curate golden conversation sets, synthetic test data, evaluation rubrics, and regression suites to measure quality over time.
• Validate AI agent safety and reliability: Test tool-calling end-to-end (tool selection, parameter mapping, sequencing, retries/timeouts, error handling, and recovery), including negative/edge cases; confirm outputs align to business rules and document risk mitigations.
• Test integration/workflow integrity: Validate API contracts, data transformations, and downstream impacts (idempotency, duplicate prevention, partial failures, and latency) across environments.
• Manage defects through resolution: Identify, log, prioritize, and track defects; partner with developers to reproduce issues and validate fixes.
• Provide release certification: Deliver final quality sign-off prior to production release, ensuring defined acceptance criteria and quality standards are met.
• Align on acceptance criteria: Partner with Business Analysts to ensure user stories include clear, testable acceptance criteria and that test coverage maps to requirements.
• Document testing and metrics: Maintain test cases, data, results, and quality metrics; communicate outcomes and risks.
• Participate in Agile ceremonies: Engage in planning, grooming, standups, and retrospectives to embed quality throughout delivery.
• Drive continuous improvement: Identify opportunities to improve automation coverage, testing effectiveness, and quality practices.
Requirements:
• 3+ years in a QA, test automation, or software testing role with demonstrated experience developing and executing test strategies.
• Test automation expertise: Strong hands-on experience building automated test frameworks using tools such as Selenium, Playwright, Cypress, Jest, Pytest, or similar frameworks.
• Scripting and programming skills: Proficiency in at least one programming or scripting language (Python, JavaScript, Java, C#) for test automation development.
• Agile methodology: Direct experience working in an Agile/Scrum environment with hands-on involvement in sprint cycles and continuous integration.
• Defect management tools: Experience using bug tracking and test management tools such as Jira, Azure DevOps, TestRail, or similar platforms.
• Bachelors level degree is preferred, but not required.
Benefits:
• Health insurance
• 401(k) matching
• Flexible work hours
• Paid time off
• Remote work options
Apply tot his job
Apply To this Job