[Remote] Senior Software Engineer II - Applied AI and Evaluations (Remote Eligible)

Note: The job is a remote job and is open to candidates in USA. Smartsheet has been empowering teams for over 20 years with innovative work management solutions. The Senior Software Engineer II will be responsible for owning agent quality, diagnosing failures, and driving improvements in AI-powered work management tools.ResponsibilitiesOwn agent quality end-to-end: diagnosis, improvement, and validation across SmartAssist's orchestrator and subagentsIdentify failure modes across quality dimensions factual accuracy, completeness, tone, actionability, and latency and prioritize what to fixDrive quality improvements through prompt engineering, context engineering, and RAG retrieval tuningExtend and mature our evaluation framework: scorers, golden datasets, regression gates, and online evaluation for production trafficClose the feedback loop ensure that every change has a measurable, attributable quality signalCollaborate with our Agent Architecture lead to distinguish quality problems that require prompt/context solutions from those that require structural fixesEstablish repeatable methodology that scales beyond any single agent or subagentSkills8+ years of software engineering experience, with at least 2 years working directly with LLMs in productionDeep, hands-on experience with prompt engineering and context engineering, you understand how model behavior changes with framing, structure, and input designStrong working knowledge of RAG architectures: chunking strategies, embedding models, retrieval evaluation, and failure diagnosisExperience building or extending LLM evaluation frameworks, you have designed scorers, worked with golden datasets, and thought carefully about what good looks likeFluency in agent system design, you don't need to own the architecture, but you can engage as a peer on architectural tradeoffs that affect qualityStrong Python skills; comfortable working in data-heavy environments (Databricks, Delta tables, or equivalent)Ability to communicate complex quality findings (written and verbal) to both technical and non-technical stakeholders, you can explain what's broke, why it matters, and what needs to happen next without losing the roomStrong cross-functional judgment, you know when to escalate, when to resolve independently, and how to build credibility across engineering, product, and AI platform teamsA bias for clarity in ambiguous situations, when failure modes are murky and trade-offs are real, you bring structure and a clear point of view rather than waiting for consensusLegally eligible to work in the U.S. on an ongoing basisBS or MS in Computer Science, a related field, or equivalent industry experienceExperience with MLflow or similar experiment tracking platformsFamiliarity with CI-integrated evaluation pipelinesExperience with multi-agent orchestration frameworksPrior work in an Applied AI or LLMOps function within a product companyBenefitsEmployer subsidized medical/vision and dental coverage for full-time employees401k Match to help you save for your future (50% of your contribution up to the first 6% of your eligible pay)Monthly stipend to support your work and productivityFlexible Time Away Program, plus Sick Time OffUS employees are automatically covered under Smartsheet-sponsored life insurance, short-term, and long-term disability plansUS employees receive 12 paid holidays per yearUp to 24 weeks of Parental LeavePersonal paid Volunteer Day to support our communityOpportunities for professional growth and development including access to Udemy online coursesCompany Funded Perks, including a counseling membership, local retail discounts, and your own personal Smartsheet accountTeleworking options from any registered location in the U.S. (role specific)This role will be eligible for a market competitive incentive opportunityCompany OverviewSmartsheet is a cloud-based work management platform that empowers collaboration, drives better decision-making, and accelerates innovation. It was founded in 2005, and is headquartered in Bellevue, Washington, USA, with a workforce of 1001-5000 employees. Its website is https://www.smartsheet.com.