Principal Agentic Engineer

ZeroFox protects organizations from external threats across the public attack surface, and we're building agentic AI into the core of how that work gets done. You'll define and build the production agentic systems that power ZeroFox's product: the architectures, tooling, and practices that make agents manage state across steps, handle failures without losing context, and know when to escalate to a human. This is deep agent systems work. You'll be solving the hardest problems in multi-step orchestration, reliability, and evaluation for non-deterministic systems operating on adversarial data. What you'll do Design and implement production agent architectures: state management, error handling, retry logic, graceful degradation, human-in-the-loop escalation Build evaluation and testing frameworks for non-deterministic agent workflows: offline tests, synthetic data generation, regression checks, and post-deploy monitoring Implement orchestration patterns: multi-agent coordination, tool-calling chains, memory management, context window optimization Define deployment and governance practices: agent versioning, rollback, behavioral telemetry, anomaly detection Instrument agents with tracing, logging, and observability that make production behavior debuggable Establish architectural standards and best practices for agentic development through design reviews, pairing, and mentorship Requirements Required qualifications 7+ years of software engineering experience building production systems, with hands-on experience designing and deploying LLM-based agents Deep knowledge of agent reliability patterns: state management, error boundaries, escalation logic, context management, tool-calling failure modes Experience with agent orchestration frameworks and understanding of tradeoffs between existing frameworks and building custom Strong backend engineering fundamentals: testing, monitoring, deployment, debugging, performance optimization Hands-on experience with retrieval-augmented systems and understanding of how retrieval quality affects agent behavior Experience building systems that handle adversarial or noisy input (cybersecurity, fraud detection, content moderation, or similar domains) Familiarity with cloud-based AI deployment, including observability, reliability, and cost considerations What success looks like Agents ship to production with defined reliability contracts and clear operational ownership Agent failures are detected and handled systematically through telemetry and evaluation pipelines, not customer reports The team has reusable patterns for state management, error handling, and escalation that new agent projects build on rather than reinvent Preferred qualifications Experience defining agent governance practices (versioning, behavioral telemetry, rollback) in a production environment Background in cybersecurity or adjacent domain with high-noise, adversarial data Experience with tool-use architectures and integration patterns for connecting agents to external systems Track record of raising engineering capability across a team through mentoring, pairing, or design review Benefits Competitive compensation Community-driven culture with employee events Regular catered lunches for in-office work; snacks, drinks available daily Generous time off Comprehensive health benefits & 401(k) plan Fun, modern workspace Respectful and nourishing work environment, where every opinion is heard and everyone is encouraged to be an active part of the organizational culture About us ZeroFox is on a mission to make the internet safer for all. Innovation is at our core – we are relentless in the pursuit of finding new ways to disrupt external cyber threats on the surface, deep, and dark web. ZeroFox offers the only unified cybersecurity platform combining advanced AI analytics, digital risk and privacy protection, full-spectrum threat intelligence, and a robust portfolio of breach, incident and takedown response capabilities to protect customers from growing threats across the external attack surface. It’s a great time to join us in the Fox Den – with fresh private equity funding, expanding investments in AI, a people-first culture, and centers of excellence around the world, we’re growing like never before. If you’re looking for a mission-oriented, customer-focused, collaborative team and ready to take the fight to the adversary, apply to join us in the Den today. Equal Opportunity We aim to build a team that represents a variety of backgrounds, perspectives, and skills. We embrace inclusion and ensure equal employment opportunity without discrimination or harassment based on race, color, religion, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity or expression, age, disability, national origin, marital or domestic/civil partnership status, genetic information, citizenship status, military or veteran status, or any other personal characteristic.