[Remote] AI Agent Evaluation Analyst (Freelance)

Note: The job is a remote job and is open to candidates in USA. Mindrift is a company focused on shaping the future of AI through collective human intelligence. They are seeking an AI Agent Evaluation Analyst to review and improve the evaluation of autonomous AI agents, requiring strong analytical skills and attention to detail. Responsibilities • Reviewing evaluation tasks and scenarios for logic, completeness, and realism • Identifying inconsistencies, missing assumptions, or unclear decision points • Helping define clear expected behaviors (gold standards) for AI agents • Annotating cause-effect relationships, reasoning paths, and plausible alternatives • Thinking through complex systems and policies as a human would to ensure agents are tested properly • Working closely with QA, writers, or developers to suggest refinements or edge case coverage Skills • Excellent analytical thinking: Can reason about complex systems, scenarios, and logical implications • Strong attention to detail: Can spot contradictions, ambiguities, and vague requirements • Familiarity with structured data formats: Can read, not necessarily write JSON/YAML • Ability to assess scenarios holistically: What's missing, what's unrealistic, what might break? • Good communication and clear writing (in English) to document your findings • Experience with policy evaluation, logic puzzles, case studies, or structured scenario design • Background in consulting, academia, olympiads (e.g. logic/math/informatics), or research • Exposure to LLMs, prompt engineering, or AI-generated content • Familiarity with QA or test-case thinking (edge cases, failure modes, 'what could go wrong') • Some understanding of how scoring or evaluation works in agent testing (precision, coverage, etc.) Benefits • Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments • Participate in an advanced AI project and gain valuable experience to enhance your portfolio • Influence how future AI models understand and communicate in your field of expertise Company Overview • Welcome to Mindrift — a space where innovation meets opportunity. It was founded in undefined, and is headquartered in , with a workforce of 501-1000 employees. Its website is Apply tot his job