[Remote] Gen AI - Data Automation Engineer

Note: The job is a remote job and is open to candidates in USA. Scalence L.L.C. is a Fortune 500 technology, engineering, and science solutions leader addressing complex challenges in various sectors. They are seeking a Data Automation Engineer to design and implement AI-driven automation solutions across AWS and Azure environments, focusing on building scalable data pipelines and automating processes for analytics and customer engagement.ResponsibilitiesDesign and maintain data pipelines in AWS using S3, RDS/SQL Server, Glue, Lambda, EMR, DynamoDB, and Step FunctionsDevelop ETL/ELT processes to move data from multiple data systems including DynamoDB → SQL Server (AWS) and between AWS ↔ Azure SQL systemsIntegrate AWS Connect CRM data into the enterprise data pipeline for analytics and operational reportingEngineer, enhance ingestion pipelines with Apache Spark, Flume, Kafka for real-time and batch processing into Apache Solr, AWS Open Search platformsLeverage Generative AI services and Frameworks (AWS Bedrock, Amazon Q, Azure OpenAI, Hugging Face, LangChain) to: Create automated processes for vector generation and embeddings from unstructured data. Automate data quality checks, metadata tagging, and lineage tracking. Enhance ingestion/ETL with LLM-assisted transformation and anomaly detection. Build conversational BI interfaces that allow natural language access to Solr and SQL dataDevelop AI-powered copilots for pipeline monitoring and automated troubleshootingImplement SQL Server stored procedures, indexing, query optimization, profiling, and execution plan tuning to maximize performanceApply CI/CD best practices using GitHub, Jenkins, or Azure DevOps for both data pipelines and GenAI model integrationEnsure security and compliance through IAM, KMS encryption, VPC isolation, RBAC, and firewallsSupport Agile DevOps processes with sprint-based delivery of pipeline and AI-enabled featuresSkillsBS in Computer Science or related field with 2+ years of data engineering, automation experiencesHands-on experience with SQL, SSIS, Python, Spark, Bash, Power shell, AWS/Azure CLIsExperience with AWS services like S3, RDS/SQL Server, Glue, Lambda, EMR, DynamoDBFamiliarity with Apache Flume, Kafka, Solr for large-scale data ingestion and searchFamiliarity with LLM, Gen AI frameworks using AWS Bedrock, Azure OpenAI or open source platform, toolsExperience with integrating REST API calls in data pipelines and workflowsFamiliarity with JIRA, GitHub / Azure DevOps / Jenkins for SDLC and CI/CD automationStrong troubleshooting and performance optimization skills in SQL, Spark or other data engineering solutionsExperience operationalizing Generative AI (GenAI Ops) pipelines, including model deployment, monitoring, retraining, and lifecycle management for LLMs and AI-enabled data workflowsGood communication and presentation skillsAbility to obtain Federal government Public Trust clearanceCertifications: AWS Data Engineer, AWS AI/ML Specialty, Azure AI Engineer, Databricks certified Data EngineerExperience implementing RAG pipelines, embeddings, and vector search with Solr, OpenSearch, FAISS, Pinecone, or Pgvector/SQL server vector typesExperience with GenAI powered coding tools such as Claude Code, OpenAI Codex, VS CodeExperience with multi-cloud data integration (AWS ↔ Azure SQL)Familiarity with Client BizTalk and SSIS for SQL Server ETL workflowsKnowledge of data lineage/governance tools (Purview, Unity Catalog, AWS Glue Catalog)Familiarity with Infrastructure-as-Code (Terraform/CloudFormation, Bicep) for automated deploymentsExperience with compliance frameworks (FedRAMP, PCI-DSS, HIPAA)Company OverviewIn today’s dynamic and competitive market, success hinges on mastering three key areas: Data Intelligence, Business Resilience, and Digital Experience. It was founded in undefined, and is headquartered in Morristown, New Jersey, US, with a workforce of 501-1000 employees. Its website is https://www.scalence.com/.