Automation QA Engineer
Ciklum is looking for a Automation QA Engineer to join our team full-time in Poland.
We are a custom product engineering company that supports both multinational organizations and scaling startups to solve their most complex business challenges. With a global team of over 4,000 highly skilled developers, consultants, analysts and product owners, we engineer technology that redefines industries and shapes the way people live.
About the role:
As a Automation QA Engineer, become a part of a cross-functional development team engineering experiences of tomorrow.
Responsibilities:
Programmatic RAG Evaluation: Build and run automated validation test suites using Python frameworks like RAGAS to continuously score pipeline faithfulness, answer relevance, and context precision
CI/CD Quality Gates: Integrate automated LLM evaluation scorecards directly into the client's Azure DevOps pipelines to prevent regression in code-generation quality
Data Integrity Validation: Develop automated checks to monitor the performance of documentation ingestion pipelines, chunking consistency, and vector store retrieval accuracy
Phase-Gate Reporting: Generate programmatic quality scorecards and regression reports to satisfy technical criteria at each project gate approval milestone
Squad Collaboration: Work closely with senior and mid-level ML implementation engineers to isolate, reproduce, and patch chunking or retrieval flaws exposed by the validation metrics
Requirements:
Core Experience Baseline: 3+ years of experience in backend test automation, data quality engineering, or Python software development within structured delivery environments
Automation Engineering Stack: Mastery of Python for backend testing, scripting, and data manipulation. Traditional UI automation tools like Selenium or Cypress are not applicable for this scope
AI Evaluation Frameworks: Direct, hands-on experience or deep technical familiarity with LLM validation frameworks (specifically RAGAS or directly equivalent automated validation engines)
CI/CD Infrastructure: Practical experience configuring automated script execution, regression test triggers, and quality gates inside Git-based pipelines, preferably Azure DevOps
Data & Search Familiarity: Basic functional understanding of relational databases (PostgreSQL), data profiling techniques, and search indexing concepts.
Location Constraint: Must be physically located in Poland or Eastern Europe to remain aligned with the embedded co-delivery squad
What`s in it for you?
Strong community: Work alongside top professionals in a friendly, open-door environment
Growth focus: Take on large-scale projects with a global impact and expand your expertise
Tailored learning: Boost your skills with internal events (meetups, conferences, workshops), Udemy access, language courses, and company-paid certifications
Endless opportunities: Explore diverse domains through internal mobility, finding the best fit to gain hands-on experience with cutting-edge technologies
Flexibility: Enjoy flexibility â full remote working possibilities
Care: Weâve got you covered with company-paid medical insurance
About us:
At Ciklum, we are always exploring innovations, empowering each other to achieve more, and engineering solutions that matter. With us, youâll work with cutting-edge technologies, contribute to impactful projects, and be part of a One Team culture that values collaboration and progress.
With delivery centers in WrocĹaw and GdaĹsk, our 300+ professionals in Poland drive forward-thinking solutions for global clients. Join a community where collaboration sparks innovationâand your impact reaches millions.
Explore, empower, engineer with Ciklum!
Interested already? We would love to get to know you! Submit your application. We canât wait to see you at Ciklum.
#LI-JW1
We are a custom product engineering company that supports both multinational organizations and scaling startups to solve their most complex business challenges. With a global team of over 4,000 highly skilled developers, consultants, analysts and product owners, we engineer technology that redefines industries and shapes the way people live.
About the role:
As a Automation QA Engineer, become a part of a cross-functional development team engineering experiences of tomorrow.
Responsibilities:
Programmatic RAG Evaluation: Build and run automated validation test suites using Python frameworks like RAGAS to continuously score pipeline faithfulness, answer relevance, and context precision
CI/CD Quality Gates: Integrate automated LLM evaluation scorecards directly into the client's Azure DevOps pipelines to prevent regression in code-generation quality
Data Integrity Validation: Develop automated checks to monitor the performance of documentation ingestion pipelines, chunking consistency, and vector store retrieval accuracy
Phase-Gate Reporting: Generate programmatic quality scorecards and regression reports to satisfy technical criteria at each project gate approval milestone
Squad Collaboration: Work closely with senior and mid-level ML implementation engineers to isolate, reproduce, and patch chunking or retrieval flaws exposed by the validation metrics
Requirements:
Core Experience Baseline: 3+ years of experience in backend test automation, data quality engineering, or Python software development within structured delivery environments
Automation Engineering Stack: Mastery of Python for backend testing, scripting, and data manipulation. Traditional UI automation tools like Selenium or Cypress are not applicable for this scope
AI Evaluation Frameworks: Direct, hands-on experience or deep technical familiarity with LLM validation frameworks (specifically RAGAS or directly equivalent automated validation engines)
CI/CD Infrastructure: Practical experience configuring automated script execution, regression test triggers, and quality gates inside Git-based pipelines, preferably Azure DevOps
Data & Search Familiarity: Basic functional understanding of relational databases (PostgreSQL), data profiling techniques, and search indexing concepts.
Location Constraint: Must be physically located in Poland or Eastern Europe to remain aligned with the embedded co-delivery squad
What`s in it for you?
Strong community: Work alongside top professionals in a friendly, open-door environment
Growth focus: Take on large-scale projects with a global impact and expand your expertise
Tailored learning: Boost your skills with internal events (meetups, conferences, workshops), Udemy access, language courses, and company-paid certifications
Endless opportunities: Explore diverse domains through internal mobility, finding the best fit to gain hands-on experience with cutting-edge technologies
Flexibility: Enjoy flexibility â full remote working possibilities
Care: Weâve got you covered with company-paid medical insurance
About us:
At Ciklum, we are always exploring innovations, empowering each other to achieve more, and engineering solutions that matter. With us, youâll work with cutting-edge technologies, contribute to impactful projects, and be part of a One Team culture that values collaboration and progress.
With delivery centers in WrocĹaw and GdaĹsk, our 300+ professionals in Poland drive forward-thinking solutions for global clients. Join a community where collaboration sparks innovationâand your impact reaches millions.
Explore, empower, engineer with Ciklum!
Interested already? We would love to get to know you! Submit your application. We canât wait to see you at Ciklum.
#LI-JW1