[Remote] Data Automation Engineer

Note: The job is a remote job and is open to candidates in USA. AllStates Consulting Services is seeking a Data Automation Engineer to design and implement innovative data automation solutions for an enterprise-scale Microsoft Azure-based analytics and reporting platform. The role involves building scalable data pipelines, automating workflows, and supporting AI/ML initiatives while collaborating with various stakeholders to optimize cloud-based data environments.ResponsibilitiesDesign, develop, and maintain high-performance data pipelines using:Azure Data FactoryAzure Synapse PipelinesApache Spark NotebooksPythonSQLStored ProceduresTranslate business requirements into scalable data engineering and AI-driven solutionsContinuously improve automation tools for reliability, scalability, and adaptabilityResearch and implement AI/ML and Generative AI solutions to automate data processes and eliminate workflow bottlenecksCollaborate with implementation specialists, engineering teams, and customers to develop data-driven solutionsDesign and implement data ingestion, transformation, integration, and processing solutionsSupport advanced analytics, reporting, visualization, and AI/ML initiativesImplement:Data migrationData qualityData integrityMetadata managementData security functionsMonitor, troubleshoot, and optimize data pipeline performanceExecute ETL performance testing and validate benchmark resultsAnalyze:Pipeline runtimeThroughputLatencyResource utilizationParticipate in performance testing for:Azure Data Factory (ADF)Azure SynapseDatabricksSupport performance tuning activities including:Query optimizationPartitioningIndexingValidate data consistency and completeness after performance testingCollaborate with DevOps and infrastructure teams on compute, memory, and scaling optimizationDocument test results, findings, and recommendationsSupport Agile DevOps processes including Program Increment planningMaintain strict versioning and configuration control processesSkills2+ years of experience with two or more of the following: SQL, T-SQL, MDX/DAX, Python, PySparkExperience designing and building ETL and data engineering solutionsExperience with Azure Data Lake ServicesExperience with Azure Synapse AnalyticsExperience with Azure Data FactoryExperience with Integration RuntimeExperience with Microsoft data and BI technologies including: SQL Server, Stored Procedures, SSIS, SSRS, SSAS (Cubes), Power BIExperience automating data processes using: Azure CLI, AWS CLI, Bash, PowerShellExperience with Azure DevOps ReposExperience with GitHubExperience with Pipeline versioningExperience with Release managementExperience supporting Production environmentsExperience supporting Development environmentsExperience supporting Testing environmentsExperience supporting Integration environmentsKnowledge of Agile development methodologiesStrong analytical, troubleshooting, and problem-solving skillsAbility to support multiple projects simultaneouslyStrong communication and collaboration skillsBachelor's degree in: Computer Science, Related technical field2+ years of relevant professional experienceU.S. Citizenship requiredAbility to successfully obtain and maintain a Public Trust clearanceDemonstrated commitment to continuous learning and professional developmentGenerative AI development experienceGenerative AI for Data Analytics experienceMicrosoft certifications including: Azure Fundamentals, Azure Data Engineer, Power BI, Azure AIAWS Certified Data Engineer certificationExperience with DatabricksExperience with REST APIsExperience with DockerExperience with Enterprise ETL toolsetsPerformance tuning experience including: Indexing, Execution plans, Query analytics, Data profilingKnowledge of: Data encryption, Cloud virtual networks, Routing, Firewalls, Log Analytics, Monitoring toolsExperience with ARM templatesExperience with Bicep templatesExperience with RBAC access controlsData lineage and impact analysis experience using: Microsoft Purview, Synapse Pipeline TracingBenefitsHealth insuranceHealth savings accountDental insuranceVision insuranceFlexible spending accountsLife insuranceRetirement planCompany OverviewAllStates Consulting Services offers technical staffing solutions and IT professional services. It was founded in 1945, and is headquartered in Plano, Texas, USA, with a workforce of 201-500 employees. Its website is https://www.allstatesconsulting.net/.