Software Developer in Test (Python) – Senior
Overview:
SOFTSWISS is hiring a Software Developer in Test to join our team. We are looking for an automation focused QA professional with strong Python skills who will help build reliable test automation, improve product quality and contribute to quality evaluation for AI and LLM based systems.
Purpose of the role:
You will be responsible for defining testing approaches, developing and maintaining Python based automated tests, and building quality evaluation solutions for AI systems. The role combines backend/API test automation, test framework development, CI/CD improvements and AI quality evaluation, including RAG, agentic workflows, MCP servers, tool call behavior, regression checks and LLM observability.
Key responsibilities:
Analyze requirements and define the testing approach for new features and product changes
Automate test cases using the existing framework based on Python and PyTest
Build automated quality evaluation pipelines for AI systems using metrics and LLM-as-judge approaches
Test MCP servers, tool schemas and tool-call behavior, including edge cases and invalid arguments
Evaluate agentic workflows, including tool selection, multi-step reasoning, error handling, loop recovery and state correctness
Maintain and improve the test automation framework and contribute to internal testing tools, including mocks
Prepare and maintain test documentation, including checklists, test cases and quality reports
Participate in test design, estimations, release testing and product quality assessment
Contribute to CI/CD and QA process improvements
Design and maintain evaluation suites and golden datasets for RAG and agentic flows
Perform adversarial testing for AI systems, including prompt injection, jailbreaks, tool misuse and data leakage risks
Set up regression checks for changes in prompts, models, retrieval settings and chunking strategies
Track AI system quality together with cost, latency and token usage
Use tracing and observability tools to debug, measure and improve LLM application behavior
Required Experience:
5+ years of experience in Quality Assurance, including both manual and automation testing
Solid understanding of QA principles, test design, test coverage, test pyramid and SDLC
Experience with Python-based test automation frameworks, such as PyTest, Behave or similar
Experience with CI/CD and monitoring or alerting tools, such as Datadog, ELK, Sentry or similar
Interest in testing AI/LLM-based systems. Hands-on experience is preferred, but we are also open to candidates who can learn quickly and want to grow in this area
Familiarity with RAG, LLM evaluation and quality metrics, such as groundedness, faithfulness, answer relevance and retrieval quality
Experience or interest in AI evaluation tools, such as RAGAS, DeepEval, promptfoo, LangSmith Eval, TruLens, Arize Phoenix or similar
Understanding of how to test non-deterministic systems, where there may be no single correct output
Familiarity with LangChain, LangGraph, MCP, vector databases, semantic search or LLM observability tools would be a strong plus
Good spoken and written English (B2 level or higher)
Nice to have:
Load, stress or performance testing experience
Kubernetes experience
Prompt engineering experience
AI safety, responsible AI or guardrails testing experience
Experience curating evaluation datasets, data labeling or annotation
Experience with Keycloak or OIDC auth-flow testing
Our Benefits:
Full-time work opportunities
Private insurance
An additional Day Off (1) per calendar year
Sports program compensation
Comprehensive Mental Health Programme
Free online English lessons with native speakers
Generous referral program
Training, internal workshops, and participation in international professional conferences and corporate events
SOFTSWISS is hiring a Software Developer in Test to join our team. We are looking for an automation focused QA professional with strong Python skills who will help build reliable test automation, improve product quality and contribute to quality evaluation for AI and LLM based systems.
Purpose of the role:
You will be responsible for defining testing approaches, developing and maintaining Python based automated tests, and building quality evaluation solutions for AI systems. The role combines backend/API test automation, test framework development, CI/CD improvements and AI quality evaluation, including RAG, agentic workflows, MCP servers, tool call behavior, regression checks and LLM observability.
Key responsibilities:
Analyze requirements and define the testing approach for new features and product changes
Automate test cases using the existing framework based on Python and PyTest
Build automated quality evaluation pipelines for AI systems using metrics and LLM-as-judge approaches
Test MCP servers, tool schemas and tool-call behavior, including edge cases and invalid arguments
Evaluate agentic workflows, including tool selection, multi-step reasoning, error handling, loop recovery and state correctness
Maintain and improve the test automation framework and contribute to internal testing tools, including mocks
Prepare and maintain test documentation, including checklists, test cases and quality reports
Participate in test design, estimations, release testing and product quality assessment
Contribute to CI/CD and QA process improvements
Design and maintain evaluation suites and golden datasets for RAG and agentic flows
Perform adversarial testing for AI systems, including prompt injection, jailbreaks, tool misuse and data leakage risks
Set up regression checks for changes in prompts, models, retrieval settings and chunking strategies
Track AI system quality together with cost, latency and token usage
Use tracing and observability tools to debug, measure and improve LLM application behavior
Required Experience:
5+ years of experience in Quality Assurance, including both manual and automation testing
Solid understanding of QA principles, test design, test coverage, test pyramid and SDLC
Experience with Python-based test automation frameworks, such as PyTest, Behave or similar
Experience with CI/CD and monitoring or alerting tools, such as Datadog, ELK, Sentry or similar
Interest in testing AI/LLM-based systems. Hands-on experience is preferred, but we are also open to candidates who can learn quickly and want to grow in this area
Familiarity with RAG, LLM evaluation and quality metrics, such as groundedness, faithfulness, answer relevance and retrieval quality
Experience or interest in AI evaluation tools, such as RAGAS, DeepEval, promptfoo, LangSmith Eval, TruLens, Arize Phoenix or similar
Understanding of how to test non-deterministic systems, where there may be no single correct output
Familiarity with LangChain, LangGraph, MCP, vector databases, semantic search or LLM observability tools would be a strong plus
Good spoken and written English (B2 level or higher)
Nice to have:
Load, stress or performance testing experience
Kubernetes experience
Prompt engineering experience
AI safety, responsible AI or guardrails testing experience
Experience curating evaluation datasets, data labeling or annotation
Experience with Keycloak or OIDC auth-flow testing
Our Benefits:
Full-time work opportunities
Private insurance
An additional Day Off (1) per calendar year
Sports program compensation
Comprehensive Mental Health Programme
Free online English lessons with native speakers
Generous referral program
Training, internal workshops, and participation in international professional conferences and corporate events