Generative AI Automation Tester - Remote / Telecommute
About the position Responsibilities • Evaluate and test Generative AI POC models built using Open AI and Vertex LLM Models. • Design, develop, and execute detailed test plans to validate the model performance with real customer data. • Perform data validation, ensuring accuracy and completeness of the input/output data flow through AI models. • Collaborate with data scientists and engineers to ensure the model's output is aligned with the expected results. • Create test cases to assess model accuracy, bias, performance, and edge cases. • Identify model weaknesses, inaccuracies, and areas for optimization. • Report bugs, issues, and improvement areas, providing detailed feedback to development teams. • Use automated testing tools and frameworks for model testing. • Maintain comprehensive documentation of the QA process and results. Requirements • Proven experience in testing Machine Learning/AI models. • Familiarity with Generative AI models like Open AI GPT, Vertex AI models. • Strong knowledge of data validation, model performance, and quality assurance practices. • Experience with model evaluation metrics such as accuracy, precision, recall, F1 score, and bias analysis. • Proficiency in Python or other relevant programming languages. • Familiarity with testing frameworks and automated testing tools for AI models. • Strong analytical and problem-solving skills. Nice-to-haves • Experience in testing AI models with real-world datasets. • Knowledge of model versioning, deployment, and monitoring. Apply tot his job