Pricing For Talent
Login Free Trial Book a Demo

Lead Databricks Engineer

New

Skills

AWS Databricks Databricks Workflows Delta Lake Delta Live Tables Google Cloud Platform Microsoft Azure PySpark Spark SQL Unity Catalog

We are seeking a highly skilled Lead Databricks Engineer to own and oversee the end-to-end Databricks lakehouse architecture across major cloud platforms such as AWS, Azure, or GCP. This role involves designing and scaling batch and streaming pipelines using PySpark and Spark SQL, while defining ingestion, governance, and performance patterns for petabyte-scale data.

Key Responsibilities
  • Own end-to-end Databricks lakehouse architecture across cloud platforms.
  • Design and scale batch and streaming pipelines with PySpark and Spark SQL.
  • Define ingestion, governance, and performance patterns for large-scale data.
  • Lead Unity Catalog governance including access, lineage, and PII handling.
  • Mentor engineers and lead architecture reviews and engineering practices.
  • Collaborate with ML engineers to productionize models using MLflow.
Required Skills & Qualifications
  • 8+ years of experience in data engineering; 4+ years specifically on Databricks.
  • Deep expertise in Spark technologies (PySpark, Spark SQL) and performance tuning.
  • Experience with Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables.
  • Production cloud experience with AWS, Azure, or GCP, including networking, IAM, and storage.
  • Proficiency in Python and SQL; Scala is a plus.
  • Experience with CI/CD and DevOps tools such as Git, Terraform, dbx, or Asset Bundles.

No forms. Your profile is generated instantly.

Job Type: Remote

Salary: Not Disclosed

Experience: Entry

Duration: Months

Share this job:

Similar Jobs

Senior Full Stack Engineer

Posted 140 days ago

Develop and maintain full stack web and mobile applications.

Implement and automate robust backend API services.

Angular API Design AWS CI/CD

Senior Full Stack Engineer (ASP.NET Core, AWS)

Posted 140 days ago

Design and develop user-facing backend services and web applications.

Collaborate effectively with team members and stakeholders.

AWS Engineer Full stack Git

Software Engineer at Acquia

Posted 140 days ago

Develop and maintain customer-facing applications and microservices on Kubernetes

Collaborate with cross-functional teams to document product and development details

Ansible AWS Devops Engineer

Principal AI Engineer Role

Posted 140 days ago

Hire a remote Principal AI Engineer

Develop customer experience automation solutions

Ai Automation AWS Cloud Computing

Affirm Senior Backend Engineer

Posted 140 days ago

Lead a team of engineers through ambiguity to solve problems

Collaborate with various stakeholders in product development

Android AWS Code Review Distributed systems

Software Engineer II, Backend

Posted 140 days ago

Develop backend systems for credit decisioning.

Collaborate with team members and stakeholders.

Android AWS Backend Systems Code Review

Forward Deployed Software Engineer (Data)

Posted 140 days ago

Leading the technical implementation of AI-driven data solutions

Translating customer needs into technical requirements

AWS Communication Skills Engineer Microsoft Azure

Forward Deployed Software Engineer

Posted 140 days ago

Lead technical implementation and optimization of data platform

Serve as primary technical contact for key accounts

Airflow AWS Databricks Engineer

Staff AI Backend Engineer

Posted 140 days ago

Architect and maintain high-performance backend infrastructure

Integrate advanced AI and LLM technologies into data workflows

AWS Docker Engineer FastAPI

Staff Software Engineer Role

Posted 140 days ago

Lead backend development for AI-powered data solutions

Architect scalable systems and APIs for enterprise clients

Agile Methodologies Angular AWS CI/CD

Staff Software Engineer, AI Backend

Posted 140 days ago

Design and build scalable AI-driven backend systems

Integrate advanced language models into data workflows

AWS Docker FastAPI Google Cloud Platform

Staff Software Engineer

Posted 140 days ago

Revolutionize enterprise data operations through AI solutions.

Automate and accelerate data tasks for overworked data teams.

Ai Airflow Ansible Api Development

Data Lead: ETL & Analytics

Posted 140 days ago

Improve data infrastructure Optimize performance and accessibility Enable data-driven

g Collaborate with cross-functional teams Mentor and lead data

Airflow Analytics AWS BigQuery

Senior Node.js Angular Engineer

Posted 140 days ago

Deliver high-quality software using Node.js and Angular.

Lead and mentor engineering teams in Agile environments.

Angular AWS CI/CD Docker

Remote Senior Data Engineer

Posted 140 days ago

Hiring a remote Senior Data Engineer for Apollo

Full-time position in Poland

AWS BigQuery Cloud Data

Developer Advocacy

Posted 140 days ago

. Engage with developer community

. Provide technical support and guidance

AWS Git GraphQL JavaScript

Backend Engineer

Posted 140 days ago

Design and implement scalable backend endpoints

Mentor team members on best practices

Ansible Api Development AWS Docker

Senior Backend Engineer

Posted 140 days ago

Develop scalable backend solutions

Mentor team members

Angular Api Development Architecture AWS

Staff AI Engineer

Posted 140 days ago

Building advanced AI systems powered by LLMs and intelligent agents

Developing scalable backend systems

A/b Testing Api Integration Architecture AWS

Senior AI Engineer

Posted 140 days ago

Build and productionize advanced AI systems

Develop large language model platforms

Ab testing Ai Systems Api Integration Architecture

Staff AI Engineer Role

Posted 140 days ago

Build and productionize advanced AI systems

Develop scalable LLM-powered applications and agents

Ab testing A/b Testing Api Integration Architecture

Senior AI Engineer Role

Posted 140 days ago

Build and deploy scalable AI systems for production use.

Develop advanced multi-agent architectures and conversational AI.

Api Integration Architecture AWS Backend Development

Senior Backend Engineer Role

Posted 140 days ago

Design scalable backend solutions

Lead full software development lifecycle

Agile Methodologies Android Android development Apache Kafka

Staff ML Engineer, Apollo

Posted 140 days ago

Lead development of scalable ML systems

Advance Apollo's AI-native product features

Airflow Architecture Databricks Engineer

Senior ML Engineer, Remote

Posted 140 days ago

Design and productionize scalable machine learning systems

Personalize user experiences using data-driven models

Cloud Computer science Databricks Engineer

Senior ML Engineer II at Apollo

Posted 140 days ago

Build and productionize Machine Learning models for Apollo products

Optimize users' experience at all stages of their product journey

Airflow Ai Systems Cloud Computer science

Senior Frontend Engineer

Posted 140 days ago

Implement best practices

Drive quality and innovation

Android AWS CI/CD Cloud

Senior Fullstack Engineer

Posted 140 days ago

Work with Apollo’s 200M+ Leads data and build features on top of it

Improve data quality and build customer-facing products

Android API Design Automated Testing AWS

Sr. Fullstack Engineer at Apollo

Posted 140 days ago

Work on high-impact features using a diverse tech stack

Collaborate with cross-functional teams for product development

Elasticsearch Front end Google Cloud Platform JavaScript

ML Engineer on Apollo Team

Posted 140 days ago

Build and deploy ML models for Apollo products.

Enhance user experience through data-driven insights.

Airflow Cloud Computer science Databricks

Data Architect Role

Posted 140 days ago

Design scalable data architectures, Lead big data systems development, Integrate AI and

ion, Deploy production services, Drive thought

AWS Docker Gitlab Hadoop

Remote Backend Engineer Poland

Posted 140 days ago

Hire a remote backend engineer

Enhance team collaboration tools

Android AWS Docker Engineer

AI Software Engineer - Copilot & Amazon Q

Posted 140 days ago

Seeking an AI-forward Software Engineer with expertise in GitHub Copilot and Amazon Q.

Designing and delivering structured hands-on developer sprints focused on AI technologies.

Agile practices AWS CI/CD Devops

AI Developer Sprint Mentor

Posted 140 days ago

Lead hands-on AI developer sprints

Mentor and enable developers in generative AI

Devops Engineer Java JavaScript

Full Stack Tech Lead Role

Posted 140 days ago

Lead and architect full-stack applications

Mentor and guide engineering teams

AWS Cloud infrastructure Database Design Database Optimization

Principal Software Engineer at G-P

Posted 140 days ago

Provide technical leadership and hands-on development

Collaborate with stakeholders to meet business goals

Agile Methodologies Apis AWS Java

Principal Software Engineer Position

Posted 140 days ago

Lead software development team in technical design and project execution

Collaborate with stakeholders to meet evolving customer needs

Agile Methodologies Apis AWS Java

Senior Backend Developer Project

Posted 140 days ago

Develop scalable and high-performance applications using Node.js and NestJS.

Collaborate with multidisciplinary teams to design and implement robust solutions.

API Design Devops Docker Elasticsearch

Senior Analytics Engineer

Posted 140 days ago

Hiring a Senior Analytics Engineer remotely

Axios - Smart brevity

Airflow Analytics BigQuery Bi tools

Senior Software Engineer Remote

Posted 140 days ago

Build healthcare-focused software solutions

Integrate and manage complex data sources

Agile Methodologies API Design AWS Code Review

Principal Data Engineer

Posted 140 days ago

Set technical direction for data initiatives

Design and build scalable data pipelines

Architecture Databricks Data Compliance Data Modeling

Staff IAM/PAM Administrator Role

Posted 140 days ago

Support identity and privileged access operations.

Monitor IAM and PAM platforms.

Automation AWS Cloud Cyberark

Python Software Engineer

Posted 140 days ago

Design and develop applications on Python stack and .Net Core

Work with RESTful services and APIs at scale

AWS Git Jenkins Microsoft Azure

Principal Product Manager - AI Platform

Posted 140 days ago

Lead and drive product initiatives for AI and Data Product Platform

Adopt and implement data mesh framework across the organization

Agile Agile Development Databricks Financial Acumen

Java DevOps Engineer

Posted 140 days ago

Contribute to automation and cloud technologies development.

Ensure security and efficiency in solution delivery platforms.

Automation Devops Engineer Html

Senior Software Engineer - C#

Posted 140 days ago

Design, build, and maintain efficient, reusable code

Participate in Agile development process and ceremonies

Agile Development AWS Css Engineer