AI Statistical Analyst (Volunteering Position - Unpaid Role)

Role Overview: We are seeking a highly analytical and methodologically rigorous AI Statistical Analyst. This individual will play a critical role in applying statistical inference tools to evaluate, validate, and improve the outputs of machine learning and generative AI models. The ideal candidate will bridge the gap between classical statistical analysis and cutting-edge AI by helping us assess the reliability of probabilistic claims generated by large language models (LLMs). You will be tasked with answering questions such as: “When an LLM reports that the probability of an individual attending Yale in the 21st century is X%, how do we know that this estimate is valid, testable, and actionable?” You will help us formalize methodologies for extracting, validating, and justifying such probabilistic outputs, and you will design experiments that ensure model-generated insights are statistically grounded and interpretable. Key Responsibilities • Develop methodologies for validating probabilistic outputs from LLMs using classical statistical techniques. • Design experiments and statistical tests to evaluate the calibration and reliability of probabilistic claims made by AI models. • Work cross-functionally with ML engineers, data scientists, and prompt engineers to integrate statistical validation into generative AI workflows. • Apply statistical inference techniques (e.g., regression analysis, Bayesian inference, significance testing) to model outputs and ground-truth data. • Construct and test hypotheses about LLM behaviors under different prompt and context configurations. • Communicate findings and methodologies clearly to both technical and non-technical audiences. Required Skills and Qualifications • Strong foundation in probability theory and its applications to uncertainty quantification in AI systems. • Proven experience with regression analysis, including linear, logistic, and multivariate techniques in real-world machine learning or AI projects. • Deep understanding of hypothesis testing and statistical significance, with practical experience in applying these concepts to AI/ML evaluation. • Familiarity with generative AI models such as LLMs (e.g., GPT-4, Claude, Gemini), including experience in crafting, refining, and interpreting prompts. • Proficiency with Python and statistical tools such as statsmodels, scipy, R, or similar. • Familiarity with Bayesian inference frameworks and uncertainty quantification methods (e.g., Monte Carlo simulation, posterior predictive checks). • Ability to design and interpret statistical experiments involving LLM-generated outputs. • Excellent communication skills, including the ability to convey statistical findings in clear, actionable terms. Preferred Qualifications • Graduate degree (MS or PhD) in Statistics, Data Science, Applied Mathematics, Econometrics, or a closely related field. • Prior experience in validating or auditing generative AI systems or developing statistical evaluation frameworks for ML outputs. • Experience working with LLM APIs and evaluating probabilistic reasoning in LLM-generated content. Apply tot his job