[Remote] Data Automation Engineer

Note: The job is a remote job and is open to candidates in USA. Aptonet is seeking a Data Automation Engineer to design and implement innovative data automation solutions for an enterprise-scale Microsoft Azure-based analytics and reporting platform. The role involves building scalable data pipelines, automating workflows, and supporting AI/ML initiatives while collaborating with various stakeholders to optimize cloud-based data environments.ResponsibilitiesDesign, develop, and maintain high-performance data pipelines using:Azure Data FactoryAzure Synapse PipelinesApache Spark NotebooksPythonSQLStored ProceduresTranslate business requirements into scalable data engineering and AI-driven solutionsContinuously improve automation tools for reliability, scalability, and adaptabilityResearch and implement AI/ML and Generative AI solutions to automate data processes and eliminate workflow bottlenecksCollaborate with implementation specialists, engineering teams, and customers to develop data-driven solutionsDesign and implement data ingestion, transformation, integration, and processing solutionsSupport advanced analytics, reporting, visualization, and AI/ML initiativesImplement:Data migrationData qualityData integrityMetadata managementData security functionsMonitor, troubleshoot, and optimize data pipeline performanceExecute ETL performance testing and validate benchmark resultsAnalyze:Pipeline runtimeThroughputLatencyResource utilizationParticipate in performance testing for:Azure Data Factory (ADF)Azure SynapseDatabricksSupport performance tuning activities including:Query optimizationPartitioningIndexingValidate data consistency and completeness after performance testingCollaborate with DevOps and infrastructure teams on compute, memory, and scaling optimizationDocument test results, findings, and recommendationsSupport Agile DevOps processes including Program Increment planningMaintain strict versioning and configuration control processesSkills2+ years of experience with two or more of the following: SQL, T-SQL, MDX/DAX, Python, PySparkExperience designing and building ETL and data engineering solutionsExperience with: Azure Data Lake Services, Azure Synapse Analytics, Azure Data Factory, Integration RuntimeExperience with Microsoft data and BI technologies including: SQL Server, Stored Procedures, SSIS, SSRS, SSAS (Cubes), Power BIExperience automating data processes using: Azure CLI, AWS CLI, Bash, PowerShellExperience with: Azure DevOps Repos, GitHub, Pipeline versioning, Release managementExperience supporting: Production environments, Development environments, Testing environments, Integration environmentsKnowledge of Agile development methodologiesStrong analytical, troubleshooting, and problem-solving skillsAbility to support multiple projects simultaneouslyStrong communication and collaboration skillsBachelor's degree in: Computer Science, Related technical field2+ years of relevant professional experienceU.S. Citizenship requiredAbility to successfully obtain and maintain a Public Trust clearanceDemonstrated commitment to continuous learning and professional developmentGenerative AI development experienceGenerative AI for Data Analytics experienceMicrosoft certifications including: Azure Fundamentals, Azure Data Engineer, Power BI, Azure AI, AWS Certified Data Engineer certificationExperience with: Databricks, REST APIs, Docker, Enterprise ETL toolsetsPerformance tuning experience including: Indexing, Execution plans, Query analytics, Data profilingKnowledge of: Data encryption, Cloud virtual networks, Routing, Firewalls, Log Analytics, Monitoring toolsExperience with: ARM templates, Bicep templates, RBAC access controlsData lineage and impact analysis experience using: Microsoft Purview, Synapse Pipeline TracingCompany OverviewAptonet delivers innovative IoT, operations research, mobility & web solutions along with staffing services with a focus on professionalism. It was founded in 2004, and is headquartered in Atlanta, Georgia, USA, with a workforce of 201-500 employees. Its website is https://www.aptonet.com.