Senior Data Engineer – Databricks & Cloud Data Engineering
Location: Basel, Switzerland (100% on-site, full-time)
Company: Swiss Himmel GmbH
Start: By agreement
Employment Duration: Permanent
About the Role
Swiss Himmel GmbH is on the lookout for a highly skilled Senior Data Engineer, possessing deep expertise in Databricks and Cloud Data Engineering. The ideal candidate will be responsible for designing, optimizing, and leading the development of scalable data pipelines and frameworks on the Azure Databricks Lakehouse platform. This role demands a solid background in metadata-driven ingestion frameworks, advanced data governance, and CI/CD automation within cloud-native environments. You will collaborate with cross-functional teams, playing a pivotal role in driving modernization initiatives for enterprise-grade data solutions.
Key Responsibilities
- Design, develop, and optimize scalable, cloud-native data pipelines and ingestion frameworks utilizing Azure Databricks, PySpark, and Delta Lake.
- Architect and implement metadata-driven ingestion and validation frameworks to streamline the onboarding of diverse data sources.
- Establish and uphold data governance and security policies with Unity Catalog and Attribute-Based Access Control (ABAC).
- Work collaboratively with Data Scientists, Architects, and DevOps teams to provide enterprise-grade solutions.
- Lead modernization efforts by migrating legacy ETL/ELT infrastructures to the Databricks Lakehouse platform.
- Implement robust data quality checks, monitoring frameworks, and validation processes to ensure the reliability of datasets.
- Drive CI/CD integration using Azure DevOps, Jenkins, and Databricks Asset Bundles.
- Provide technical leadership through code reviews, mentoring, and the enforcement of best practices.
- Partner with stakeholders to gather requirements and translate them into scalable, cost-efficient technical solutions.
- Setup and manage frameworks for monitoring, logging, and alerting, ensuring infrastructure health.
- Optimize compute and storage resources to balance cost efficiency and performance.
- Troubleshoot and resolve performance issues related to Databricks, Spark, and data pipelines.
- Stay current with emerging trends and technologies in data engineering, advocating for beneficial adoptions.
Candidate Requirements
- 15+ years of professional IT experience, with 8+ years in data engineering within cloud-native environments.
- Strong hands-on expertise in Databricks, Delta Lake, Azure Data Factory, Azure Data Lake, Python, PySpark, and SQL.
- Proven experience with metadata-driven frameworks, ETL/ELT workflows, and automation pipelines.
- Solid understanding of CI/CD practices with Azure DevOps, Jenkins, and Databricks Asset Bundles.
- Experience in implementing data governance and security frameworks (Unity Catalog, ABAC).
- Proficient in batch and real-time ingestion pipelines using tools such as Azure Data Factory and Kafka.
- Familiarity with Palantir Foundry and Databricks visualization tools (Matplotlib, dashboards) is an advantage.
- Demonstrated ability to troubleshoot, optimize, and scale large data processing workflows.
- Strong communication and stakeholder engagement skills, with a proven ability to mentor and lead teams.
- Relevant certifications such as Databricks Certified Data Engineer Associate, Databricks Spark Developer, or Microsoft Azure certifications are preferred.
What We Offer
- A culture of innovation and collaboration, dedicated to delivering lasting value for clients and employees.
- Continuous learning and training opportunities to expand your expertise.
- A flat, non-hierarchical structure, enabling direct interaction with senior partners and clients.
- A diverse and inclusive environment, where contributions are valued at every level.
Apply online using the form below. Please note that only applications matching the job profile will be considered.