[Remote] Senior Data Engineer

Note: The job is a remote job and is open to candidates in USA. Avaya is an enterprise software leader that helps organizations forge unbreakable connections. They are seeking a Senior Data Engineer to build and scale a real-time and batch data platform for their enterprise contact center solution, focusing on ultra-low-latency decisioning and big-data analytics.ResponsibilitiesDesign, build, and operate low-latency streaming pipelines (Kafka, Spark Structured Streaming) and robust batch ETL/ELT on Databricks LakehouseEstablish reliable orchestration and dependency management (Airflow), with strong SLAs and on-call readiness for business-critical data flowsModel, optimize, and document curated datasets and interfaces that serve analytics, product features, and AI workloadsImplement data quality checks, observability, and backfills; drive root-cause analysis and incident preventionPartner with application teams (Go/Java), analytics, and ML/AI to ship data products into productionBuild and maintain datasets and services that power RAG pipelines and agentic AI workflows (tool-use/function calling)When Spark/Databricks isn’t optimal, design and operate custom processors/services in Go to meet strict latency or specialized transformation requirementsInstrument prompt/response and token usage telemetry to support LLMOps evaluation and cost optimization; provide datasets for labeling and golden setsImprove performance and cost (storage/compute), review code, and raise engineering standardsDesign data solutions aligned to enterprise security, privacy, and compliance requirements (e.g., SOC 2, ISO 27001, GDPR/CCPA as applicable), partnering with Security/LegalImplement RBAC/ABAC and least-privilege access; manage service principals, secrets, and key rotation; enforce encryption in transit and at restGovern sensitive data: classification, PII handling, masking/tokenization, retention/archival, lineage, and audit logging across pipelines and storageBuild observability for data security and quality; support incident response, access reviews, and audit readinessEmbed controls in CI/CD (policy checks, dependency vulnerability scanning) and ensure infra-as-code adheres to guardrailsPartner with security engineering on penetration tests, threat modeling, and red-team exercises; remediate findings and document controlsContribute to compliance audits (e.g., SOC 2/ISO 27001) with evidence collection and continuous control monitoring; support DPIAs/PIAs where requiredSkills6+ years building production-grade data pipelines at scale (streaming and batch)Deep proficiency in Python and SQL; strong Spark experience on Databricks (or similar)Advanced SQL: window functions, CTEs, partitioning/z-ordering, query planning and tuning in lakehouse environmentsHands-on with Kafka (or equivalent) and an orchestrator (Airflow preferred)Strong data modeling skills and performance tuning for low latency and high throughputProduction mindset: SLAs, monitoring, alerting, CI/CD, and on-call participationProficient using AI coding assistants (Cursor, Claude Code) as part of daily developmentProficiency building data services/processors in Go (or willingness to ramp quickly), and familiarity with alternative frameworks (e.g., Flink/Beam) is a plusExperience in multi-cloud or cloud migration (Azure plus either GCP or AWS)Exposure to building data for AI/RAG, LLM-powered features, and agentic AI patterns (tool-use/function calling, planning/execution, memory)Familiarity with LLMOps telemetry (prompt/response logs, token budgets) and agent evaluation pipelinesBackground in high-scale product engineering (vs. internal IT-only projects)Contact center or CRM data familiarity (nice-to-have, not required)Bachelor's or Master's in CS/EE/Math or similar; strong academic background and/or top-tier companiesBenefitsPerformance-related bonusBenefitsAnnual bonus that aligns with individual and company performanceCompany OverviewAvaya is a global leader in enterprise communications, hybrid cloud CCaaS and UC solutions for mission-critical, AI-agnostic workflows. It was founded in 2000, and is headquartered in Morristown, New Jersey, USA, with a workforce of 5001-10000 employees. Its website is http://www.avaya.com.