Senior Analyst Conversational AI Analytics

Overview: National Debt Relief is looking for a Senior Analyst, Conversational AI Analytics to join the Digital & CX analytics team. This role sits at the intersection of data, user experience (UX), and Agentic AI - you will be the analytical engine behind our conversational AI experiences, translating how real users interact with our voice and chat AI into insights that shape better conversations, smarter flows, and stronger outcomes. You are not the engineer who builds AI Agents, but you are essential to making them work well. You will own the measurement framework for our conversational AI products - defining what good looks like, tracking where experiences break down, and partnering with designers, product managers, and our Applied AI team to close the gap between current performance and the ideal user journey. This is a high-impact, cross-functional role with direct exposure to senior leadership. The right person brings both analytical rigor and a genuine curiosity for how people communicate with AI. Responsibilities: Essential Duties/Responsibilities Conversational AI Analytics & Measurement Own the measurement framework for conversational AI products - defining KPIs across containment, resolution, deflection, intent recognition accuracy, conversation completion rates, drop-off, and sentiment. Build and maintain dashboards that surface conversation health metrics, user behavioral patterns, and performance trends across voice and chat channels. Provide UX insights via analysis of conversation transcripts, dialogue flows, and session data to identify friction points, misunderstood intents, fallback loops, and abandonment patterns. Partner with the Applied AI team to establish baseline performance benchmarks and define what success looks like for each conversational initiative. Translate raw interaction data into clear, actionable recommendations for conversation designers, product managers, and ML teams. UX Research & User Insight Conduct mixed-methods research - combining quantitative conversation analytics with qualitative usability testing, session replays, and user feedback - to build a complete picture of user experience quality. Map conversation flows against actual user journeys to identify where AI-guided paths diverge from user expectations or intent. Support the design and analysis of A/B tests on conversational flows, dialogue variations, and escalation triggers. Synthesize research findings into user journey documentation, annotated flow diagrams, and prioritized insight reports for product and design stakeholders. Cross-functional Collaboration & Communication Serve as the data-informed voice in conversation design reviews - bringing evidence from real user interactions to design critiques, sprint reviews, and roadmap discussions. Present insights and performance reports to senior stakeholders in clear, non-technical language that connects conversational metrics to business outcomes (conversion rates, customer satisfaction, operational cost savings). Partner with engineering to ensure proper logging, tagging, and event tracking are in place to support robust conversation analytics. Collaborate with the broader product organization to align conversational AI metrics with company-wide data governance and reporting standards. Continuous Improvement Establish a regular cadence of conversation audits - systematically reviewing transcript samples to surface emergent issues before they appear in aggregate metrics. Monitor model and flow performance post-deployment, flagging regression or drift and coordinating response with the AI and engineering teams. Stay current with trends in conversational AI, NLP evaluation methods, and UX analytics tooling - and identify where new capabilities can strengthen the team’s analytical practice. Current Initiatives (as of 2026) The Applied AI team is currently focused on the following - these represent the starting portfolio, not the boundaries of the role: Voice - Measuring and improving the user experience of an AI-powered voice bot, including turn-level analysis, intent accuracy, and escalation/hand-off optimization. Conversational Chat - Tracking and improving chatbot performance across the homepage, landing pages, and key customer journeys from a UX and outcomes lens. Internal Knowledge Base Tooling - Supporting adoption and usability measurement for AI-powered internal tools through employee feedback loops and query-level analytics. Qualifications: Education/Experience 4+ years of experience in UX analytics, product analytics, or a related data-focused role, with at least 1–2 years working on conversational AI, chatbot, or voice AI products. Required Skills/Abilities Demonstrated ability to analyze conversation-level data - transcript review, intent classification accuracy, fallback rates, CSAT/sentiment scoring - and turning findings into design or product recommendations. Proficiency with analytics and BI tools (e.g., Tableau, Looker, Power BI, or similar) for building dashboards and self-serve reporting. Comfortable working with SQL to query conversation logs and event data; Python or R a plus but not required. Strong understanding of UX research methods - usability testing, user journey mapping, qualitative coding of transcripts, and mixed-methods synthesis. Familiarity with core conversational AI concepts - intents, entities, dialogue management, NLU/NLP, fallback handling, escalation logic - sufficient to contribute meaningfully to cross-functional discussions without owning the engineering. Excellent communication skills; able to distill complex data into clear narratives for both technical teammates and non-technical stakeholders. Comfortably operating with ambiguity and managing multiple workstreams simultaneously in a fast-moving environment. Preferred: Experience with conversational analytics or NLP platforms (e.g., Qualtrics Conversational Analytics, Cognigy.AI, Google CCAI Insights, Gong, or similar). Familiarity with voice AI or IVR platforms (LiveKit, Sierra, Parloa, Replicant, or similar) from a measurement and QA perspective. Direct experience with LLM-based products, prompt evaluation, or retrieval-augmented generation (RAG) quality assessment. Background in fintech, financial services, or debt relief / credit counseling. Experience with A/B testing and experimentation frameworks applied to conversational flows. Formal UX research certification or training (Nielsen Norman Group, UXQB, or similar). National Debt Relief Role Qualifications: Computer competency and ability to work with a computer. Prioritize multiple tasks and projects simultaneously. Exceptional written and verbal communication skills. Punctuality expected, ready to report to work on a consistent basis. Attain and maintain high performance expectations on a monthly basis. Work in a fast-paced, high-volume setting. Use and navigate multiple computer systems with exceptional multi-tasking skills. Remain calm and professional during difficult discussions. Take constructive feedback. Compensation Information: Our salary ranges are determined by role, level, and location. The range displayed on each job posting reflects the minimum and maximum target for each position across the US. Within the range, individual pay is determined by work location, job-related skills, experience, and relevant education or training. This good faith pay range is provided in compliance with NYC law and the laws of other jurisdictions that may require a salary range in job postings. The salary for this position is $114,500 - $131,500 annually. About National Debt Relief: National Debt Relief was founded in 2009 with the goal of helping an expanding number of consumers deal with overwhelming debt. We are one of the most-trusted and best-rated consumer debt relief providers in the United States. As a leading debt settlement organization, we have helped over 450,000 people settle over $10 billion of debt, while empowering them to lead a healthier financial lifestyle and feel free to live their best life. At National Debt Relief, we treat our clients like real people. Our purpose is to elevate, empower, and transform their lives. Rated A+ by the Better Business Bureau, our goal is to help individuals and families get out of debt with the least possible cost through conducting financial consultations, educating the consumer and recommending the appropriate solution. We become our clients' number one advocate to help them reestablish financial stability as quickly as possible. Want to learn more about who we are? Connect with us on social! Benefits: National Debt Relief is a team-oriented environment full of rewards and growth opportunities for our employees. We are dedicated to our employee's success and growth within the company, through our employee mentorship and leadership programs. Our extensive benefits package includes: Generous Medical, Dental, and Vision Benefits 401(k) with Company Match Paid Holidays, Volunteer Time Off, Sick Days, and Vacation 12 weeks Paid Parental Leave Pre-tax Transit Benefits No-Cost Life Insurance Benefits Voluntary Benefits Options ASPCA Pet Health Insurance Discount Wellness Incentive Program National Debt Relief is a certified Great Place to Work®! National Debt Relief is an equal opportunity employer and makes employment decisions without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, protected veteran status, disability status, or any other status protected by law. For information about our Employee Privacy Policy, please see here For information about our Applicant Terms, please see here #LI-REMOTE