[Remote] Data Scientist

Note: The job is a remote job and is open to candidates in USA. Jobber is a company dedicated to empowering small businesses with technology solutions. They are seeking a Data Scientist to ensure the quality and reliability of their machine learning models, focusing on validation, monitoring, and collaboration with various teams to enhance model performance.ResponsibilitiesDesign, implement, and maintain ML model validation frameworks, including custom evaluation metrics, loss functions, and statistical tests, to ensure model quality before and after deploymentBuild and own regression test suites for ML and LLM models, catching performance regressions and unexpected behaviour across model updates and data drift scenariosDevelop and execute MCP evaluations, systematically assessing model capabilities, edge cases, and failure modes across relevant business contextsMonitor models in production using statistical process control, drift detection, and alerting pipelines; proactively surface issues before they impact customersCollaborate with senior data scientists to contribute to the design and refinement of ML model architectures, offering feedback grounded in validation resultsDocument evaluation methodologies, test results, and monitoring runbooks clearly enough that stakeholders across technical and business teams can understand model healthStay current with advancements in LLM evaluation techniques, AI safety, and model observability, and apply emerging best practices to our workflowsCommunicate findings clearly and concisely to stakeholders, translating model performance signals into actionable recommendationsSkillsIndustry experience in data science, machine learning, or a closely related quantitative fieldProficiency in Python and the core DS stack: Pandas, Scikit-Learn, XGBoost, and at least one deep learning framework (PyTorch or TensorFlow)Solid grasp of statistical concepts underpinning model evaluation: bias–variance tradeoff, calibration, confidence intervals, A/B testing, and data driftExperience with LLM evaluation frameworks (e.g. RAGAS, Eleuther AI Eval Harness, or custom LLM eval pipelines)Hands-on experience designing custom evaluation metrics; you've gone beyond off-the-shelf metrics when the problem demanded itStrong understanding of ML and LLM model architectures — you can reason about how a model is built and why it behaves the way it doesHigh proficiency in SQL for data exploration, feature validation, and debugging model inputsExceptional attention to detail — you treat model validation with the same rigour as software QAStrong written and verbal communication skills; comfortable presenting findings to both technical peers and non-technical stakeholdersExperience building model evaluation or monitoring solutions within a Snowflake environment, e.g. logging predictions and ground truth to Snowflake tables, computing eval metrics via Snowpark, or building monitoring dashboards on top of Snowflake dataFamiliarity with Snowpark (Python) for running data transformations and ML workflows directly within SnowflakeExposure to model regression testing patterns in a CI/CD context (e.g. running evals on every PR)Familiarity with prompt engineering and evaluation strategies for LLM-powered featuresExperience working in a SaaS environment and an appreciation for how model quality translates to customer impactBenefitsEquity rewardsAnnual stipends for health and wellnessRetirement savings matchingAn extended health package with fully paid premiums for body and mindAccess to a dedicated talent development program that includes career coaching and opportunities for career developmentExtended health benefits package with fully paid premiums for both body and mindMatching in RRSP, TFSA or FHSAStock optionsA dedicated Talent Development team and access to coaching, learning, and leadership programs to help you grow your career, reach your goals, and unlock your full potentialCompany OverviewJobber is cloud software that helps mobile service businesses organize their scheduling, invoicing, CRM, and team management. It was founded in 2011, and is headquartered in Edmonton, Alberta, CAN, with a workforce of 1001-5000 employees. Its website is https://getjobber.com.Company H1B SponsorshipJobber has a track record of offering H1B sponsorships, with 1 in 2024. Please note that this does not guarantee sponsorship for this specific role.