Senior Data Engineer – Azure Databricks / Cybersecurity Data Platform
Senior Data Engineer – Azure Databricks / Cybersecurity Data Platform
• Location: Charlotte, NC (Remote – Must Work EST Time Zone)
• Employment Type: Full-Time
Job Description
Key Responsibilities
• Design, develop, test, and support cloud-based data pipelines and data ingestion solutions
• Build and maintain the Cybersecurity Data Lakehouse (CyberDW) for centralized security data
• Develop ETL/ELT pipelines for ingesting and transforming cybersecurity data sources
• Implement data quality (DQ) rules, data lineage tracking, and governance standards
• Optimize data pipelines for performance, scalability, and reliability
• Work with data architecture teams to design logical and physical data models
• Collaborate with data owners and stakeholders to define technical requirements and key metrics
• Develop data views and datasets for downstream analytics and reporting
• Monitor and troubleshoot data pipeline performance and resolve data quality issues
• Participate in development planning, task assignments, and technical design discussions
• Mentor junior engineers and support team knowledge sharing
• Ensure batch processing and job scheduling standards are followed
Required Qualifications
• Bachelor’s degree in Computer Science, Data Engineering, Information Security, or related field
• 10+ years of experience in data engineering, data architecture, or data development roles
• 5+ years of experience working with Azure data platforms, including:
• Azure Databricks
• Azure Data Factory
• Azure Data Lake Storage
• Azure Functions
• Azure Monitor / Log Analytics
• Strong experience with ETL/ELT pipelines and large-scale data processing
• Proficiency in SQL, T-SQL, and database development (SQL Server or Oracle)
• Programming experience with Python, Java, or similar server-side languages
• Experience with CI/CD and DevOps tools such as Git, Jenkins, or Azure DevOps
• Experience working with REST APIs and microservices architecture
• Strong problem-solving and analytical skills
Preferred Qualifications
• Experience with data governance and metadata tools such as Collibra or IBM Infosphere
• Experience working with enterprise cybersecurity data (Identity Management, Vulnerability Management, Incident Response, etc.)
• Familiarity with security frameworks such as NIST or CIS
• Experience integrating data from enterprise platforms such as:
• ServiceNow
• CrowdStrike
• Microsoft Purview
• Azure Active Directory
• Knowledge of IBM Tivoli Workload Scheduler
• Experience with Power BI for data visualization
• Experience with SIEM platforms such as Microsoft Sentinel
Job Type: Full-time
Application Question(s):
• What type of work authorization do you have? A visa, green card, or U.S. citizen status?
• Do you have 10+ years of experience in IT development, data engineering, data governance, or data architecture roles?
• Do you have at least 5 years of experience working with Azure data services such as Azure Databricks, Azure Data Factory, Azure Functions, or Azure Data Lake Storage?
• Are you proficient in SQL development (SQL Server, Oracle, PL/SQL, T-SQL) including stored procedures?
• Do you have programming experience using Python, Java, or a similar high-level programming language?
• Do you have experience with CI/CD and DevOps tools such as Git, Jenkins, or Azure DevOps, as well as building ETL/ELT data pipelines?
• Do you have experience working with enterprise cybersecurity or security-related data and security frameworks such as NIST, CIS, or similar standards?
• Do you have experience working with REST APIs, microservices architecture, and enterprise platforms such as ServiceNow or Azure Active Directory?
Education:
• Bachelor's (Required)
Location:
• Charlotte, NC (Required)
Work Location: Remote
Apply tot his job
Apply To this Job
• Location: Charlotte, NC (Remote – Must Work EST Time Zone)
• Employment Type: Full-Time
Job Description
Key Responsibilities
• Design, develop, test, and support cloud-based data pipelines and data ingestion solutions
• Build and maintain the Cybersecurity Data Lakehouse (CyberDW) for centralized security data
• Develop ETL/ELT pipelines for ingesting and transforming cybersecurity data sources
• Implement data quality (DQ) rules, data lineage tracking, and governance standards
• Optimize data pipelines for performance, scalability, and reliability
• Work with data architecture teams to design logical and physical data models
• Collaborate with data owners and stakeholders to define technical requirements and key metrics
• Develop data views and datasets for downstream analytics and reporting
• Monitor and troubleshoot data pipeline performance and resolve data quality issues
• Participate in development planning, task assignments, and technical design discussions
• Mentor junior engineers and support team knowledge sharing
• Ensure batch processing and job scheduling standards are followed
Required Qualifications
• Bachelor’s degree in Computer Science, Data Engineering, Information Security, or related field
• 10+ years of experience in data engineering, data architecture, or data development roles
• 5+ years of experience working with Azure data platforms, including:
• Azure Databricks
• Azure Data Factory
• Azure Data Lake Storage
• Azure Functions
• Azure Monitor / Log Analytics
• Strong experience with ETL/ELT pipelines and large-scale data processing
• Proficiency in SQL, T-SQL, and database development (SQL Server or Oracle)
• Programming experience with Python, Java, or similar server-side languages
• Experience with CI/CD and DevOps tools such as Git, Jenkins, or Azure DevOps
• Experience working with REST APIs and microservices architecture
• Strong problem-solving and analytical skills
Preferred Qualifications
• Experience with data governance and metadata tools such as Collibra or IBM Infosphere
• Experience working with enterprise cybersecurity data (Identity Management, Vulnerability Management, Incident Response, etc.)
• Familiarity with security frameworks such as NIST or CIS
• Experience integrating data from enterprise platforms such as:
• ServiceNow
• CrowdStrike
• Microsoft Purview
• Azure Active Directory
• Knowledge of IBM Tivoli Workload Scheduler
• Experience with Power BI for data visualization
• Experience with SIEM platforms such as Microsoft Sentinel
Job Type: Full-time
Application Question(s):
• What type of work authorization do you have? A visa, green card, or U.S. citizen status?
• Do you have 10+ years of experience in IT development, data engineering, data governance, or data architecture roles?
• Do you have at least 5 years of experience working with Azure data services such as Azure Databricks, Azure Data Factory, Azure Functions, or Azure Data Lake Storage?
• Are you proficient in SQL development (SQL Server, Oracle, PL/SQL, T-SQL) including stored procedures?
• Do you have programming experience using Python, Java, or a similar high-level programming language?
• Do you have experience with CI/CD and DevOps tools such as Git, Jenkins, or Azure DevOps, as well as building ETL/ELT data pipelines?
• Do you have experience working with enterprise cybersecurity or security-related data and security frameworks such as NIST, CIS, or similar standards?
• Do you have experience working with REST APIs, microservices architecture, and enterprise platforms such as ServiceNow or Azure Active Directory?
Education:
• Bachelor's (Required)
Location:
• Charlotte, NC (Required)
Work Location: Remote
Apply tot his job
Apply To this Job