Databricks Data Warehouse Engineer for BI/AI Integration (US-Based)

Databricks Data Warehouse Engineer for BI/AI Integration (US-Based) Project Summary We are Saddlehill (www.saddlehill.com), one of the East Coast's premier agritourism destinations, operating a winery, farmhouse kitchen, and a rapidly growing e-commerce brand. We are seeking an expert, ongoing Databricks/Data Warehouse Engineer to design, build, and maintain our central data warehouse. This long-term partnership is critical to unify data from our disparate operational systems to power all future Business Intelligence (BI) and Artificial Intelligence (AI) initiatives across the company.This is an ongoing, long-term partnership focused on building a reliable, scalable data foundation, NOT a one-time setup. Key Integration Systems (Connecting via APIs)The core initial scope involves developing robust, fault-tolerant ETL/ELT pipelines to ingest data from the following key systems into the Databricks Lakehouse: Toast POS System (Restaurant/Tastings): Point-of-Sale (POS) and operations data. Shopify E-commerce: Sales, inventory, and customer data. QuickBooks Financials: Transactional and general ledger data.Tock Reservation System: Booking and guest data. Teamworks Scheduling System: Employee scheduling and labor data. ADP: Payroll & HR InnoVint Wine Management: Inventory and production data. The Ideal Talent (US-Based Only) We require a US-based, expert-level freelancer (no agencies) with significant, hands-on experience in cloud data warehousing and Databricks architecture. Location Requirement: Must be a U.S.-based individual freelancer. Preferably East Coast. Expertise: Deep, proven expertise with Databricks andAPI Mastery: Strong ability to connect to and extract data from various third-party SaaS APIsand manage authentication/error handling.Focus: Experience designing schemas and implementing best practices for BI readiness and future AI/ML consumption. Communication: Excellent written and verbal communication is essential for clear, ongoing collaboration with our leadership and BI team. Scope of Work & ResponsibilitiesThis role is focused on the complete lifecycle of our data foundation: Architecture & Design: Design and implement the optimal Lakehouse architecture within Databricks (e.g., Bronze, Silver, Gold layers) using bolthires Lake.API Integration & Pipeline Development: Develop and deploy robust, scheduled ETL/ELT pipelines to extract, transform, and load data from all specified source systems via their APIs. Implement effective error logging, monitoring, and reconciliation processes. Data Modeling: Create and maintain standardized data models (e.g., dimensional models) in the "Gold" layer to support self-service BI reporting and advanced analytics. Data Quality & Governance: Implement data quality checks and utilize Databricks Unity Catalog for metadata management, access control, and lineage tracking.Performance Optimization: Continuously monitor and optimize Databricks workspace and cluster performance to manage costs and ensure fast query execution. Ad-Hoc Data Requests: Assist our BI team with complex SQL or Python transformations as needed for advanced analytics projects. Requirements & Qualifications Mandatory: 3+ years of hands-on experience specifically with Databricks for Data Warehouse/Lakehouse build-outs. Expert Proficiency in Python and PySpark. Demonstrable experience integrating data from common e-commerce/retail platforms (Shopify, Toast, QuickBooks, etc.).Experience setting up data ingestion scheduling and orchestration (e.g., Databricks Workflows, Airflow, or similar). A strong portfolio or detailed case studies showcasing successful data warehousing projects. To Apply, Please Include: A brief introduction confirming you are a U.S.-based individual freelancer. A detailed overview of your experience with Databricks and its components (bolthires Lake, Unity Catalog, etc.). Describe your typical approach to building an API integration pipeline for a system like Shopify or Toast.Your hourly rate or suggested monthly retainer for ongoing, maintenance-focused work. We look forward to finding a reliable partner to build the future of our data-driven growth! Apply tot his job