Databricks SME
Key Responsibilities:
Architecture & Platform Design
Design enterprise Databricks Lakehouse architectures aligned with the Databricks Well-Architected Framework
Define reference architectures for batch, streaming, analytics, and ML workloads
Select and standardize cluster, compute, and workspace architectures
Design multi-workspace strategies (dev/test/prod, shared vs. isolated)
Ensure architectures meet scalability, availability, and performance requirements
Well-Architected Framework Alignment
Apply Databricks best practices across all pillars, including:
Security & Governance (Unity Catalog, IAM, data access controls)
Reliability & Resilience (job retries, checkpointing, failure isolation)
Performance Efficiency (cluster sizing, autoscaling, caching)
Cost Optimization (compute policies, workload separation, monitoring)
Operational Excellence (monitoring, automation, CI/CD, runbooks)
Implementation & Engineering
Lead Databricks workspace, cluster, and Unity Catalog implementations
Implement Delta Lake, Delta Live Tables (DLT), and Structured Streaming
Build and optimize ETL/ELT pipelines using Spark and SQL
Integrate Databricks with cloud services (S3/ADLS/GCS, IAM, Key Vault, networking)
Establish CI/CD pipelines for notebooks, jobs, and infrastructure
Security, Governance & Compliance
Implement Unity Catalog for centralized governance
Define data classification, lineage, and audit strategies
Enforce least-privilege access and secure networking patterns
Support compliance requirements (HIPAA, SOC 2, PCI, GDPR as applicable)
Operations & Optimization
Monitor platform health, performance, and cost
Troubleshoot production issues across jobs, clusters, and data pipelines
Perform workload tuning and cost-performance optimization
Define SLOs, alerts, and operational metrics
Collaboration & Leadership
Partner with Data Engineering, Analytics, ML, Platform, and Security teams
Translate business requirements into technical architectures
Provide architectural guidance and technical mentorship
Communicate risks, tradeoffs, and recommendations to leadership
Required Qualifications:
Experience
7+ years in data engineering, analytics, or platform architecture
3–5+ years hands-on Databricks experience in production environments
Proven experience applying the Databricks Well-Architected Framework
Experience designing cloud-native lakehouse architectures
Experience supporting mission-critical data platforms
Technical Skills
Databricks Lakehouse Platform
Apache Spark (PySpark / Scala / Spark SQL)
Delta Lake, Delta Live Tables, Structured Streaming
Unity Catalog (governance, lineage, access controls)
Cloud platforms: AWS, Azure, or GCP
Infrastructure as Code (Terraform strongly preferred)
CI/CD tools (GitHub Actions, Azure DevOps, GitLab, etc.)
Data formats and protocols (Parquet, JSON, Avro)
Certifications Required:
Databricks Certified Data Engineer Professional
Databricks Certified Professional Architect (or equivalent advanced certification)
Preferred / Additional Certifications
AWS Certified Solutions Architect (Associate or Professional)
Azure Solutions Architect Expert
Google Professional Data Engineer
Databricks Machine Learning Professional
Snowflake or other cloud data platform certifications
Soft Skills
Strong architectural decision-making and documentation skills
Excellent communication with technical and non-technical stakeholders
Ability to lead design reviews and architecture governance forums
Strong troubleshooting and performance-tuning mindset
Nice-to-Have Experience
MLflow and MLOps architectures
Real-time analytics and streaming pipelines
Multi-region or cross-account data architectures
Consulting or MSP delivery experience
Architecture & Platform Design
Design enterprise Databricks Lakehouse architectures aligned with the Databricks Well-Architected Framework
Define reference architectures for batch, streaming, analytics, and ML workloads
Select and standardize cluster, compute, and workspace architectures
Design multi-workspace strategies (dev/test/prod, shared vs. isolated)
Ensure architectures meet scalability, availability, and performance requirements
Well-Architected Framework Alignment
Apply Databricks best practices across all pillars, including:
Security & Governance (Unity Catalog, IAM, data access controls)
Reliability & Resilience (job retries, checkpointing, failure isolation)
Performance Efficiency (cluster sizing, autoscaling, caching)
Cost Optimization (compute policies, workload separation, monitoring)
Operational Excellence (monitoring, automation, CI/CD, runbooks)
Implementation & Engineering
Lead Databricks workspace, cluster, and Unity Catalog implementations
Implement Delta Lake, Delta Live Tables (DLT), and Structured Streaming
Build and optimize ETL/ELT pipelines using Spark and SQL
Integrate Databricks with cloud services (S3/ADLS/GCS, IAM, Key Vault, networking)
Establish CI/CD pipelines for notebooks, jobs, and infrastructure
Security, Governance & Compliance
Implement Unity Catalog for centralized governance
Define data classification, lineage, and audit strategies
Enforce least-privilege access and secure networking patterns
Support compliance requirements (HIPAA, SOC 2, PCI, GDPR as applicable)
Operations & Optimization
Monitor platform health, performance, and cost
Troubleshoot production issues across jobs, clusters, and data pipelines
Perform workload tuning and cost-performance optimization
Define SLOs, alerts, and operational metrics
Collaboration & Leadership
Partner with Data Engineering, Analytics, ML, Platform, and Security teams
Translate business requirements into technical architectures
Provide architectural guidance and technical mentorship
Communicate risks, tradeoffs, and recommendations to leadership
Required Qualifications:
Experience
7+ years in data engineering, analytics, or platform architecture
3–5+ years hands-on Databricks experience in production environments
Proven experience applying the Databricks Well-Architected Framework
Experience designing cloud-native lakehouse architectures
Experience supporting mission-critical data platforms
Technical Skills
Databricks Lakehouse Platform
Apache Spark (PySpark / Scala / Spark SQL)
Delta Lake, Delta Live Tables, Structured Streaming
Unity Catalog (governance, lineage, access controls)
Cloud platforms: AWS, Azure, or GCP
Infrastructure as Code (Terraform strongly preferred)
CI/CD tools (GitHub Actions, Azure DevOps, GitLab, etc.)
Data formats and protocols (Parquet, JSON, Avro)
Certifications Required:
Databricks Certified Data Engineer Professional
Databricks Certified Professional Architect (or equivalent advanced certification)
Preferred / Additional Certifications
AWS Certified Solutions Architect (Associate or Professional)
Azure Solutions Architect Expert
Google Professional Data Engineer
Databricks Machine Learning Professional
Snowflake or other cloud data platform certifications
Soft Skills
Strong architectural decision-making and documentation skills
Excellent communication with technical and non-technical stakeholders
Ability to lead design reviews and architecture governance forums
Strong troubleshooting and performance-tuning mindset
Nice-to-Have Experience
MLflow and MLOps architectures
Real-time analytics and streaming pipelines
Multi-region or cross-account data architectures
Consulting or MSP delivery experience