Data Engineer Lead

New

Skills

AI Integration Best Practices Data Processing Entity Resolution Healthcare Data Infrastructure Optimization Machine Learning Mentoring PySpark Spark

As a Staff Data Engineer at Emerald, you will lead the design and implementation of Spark and PySpark pipelines focused on entity resolution and healthcare data processing. You will take ownership of automatching, identity mapping, deduplication, and enrichment workflows, ensuring high-quality data processing from various sources.

Key Responsibilities
  • Lead Spark/PySpark pipelines for entity resolution and healthcare data processing.
  • Own automatching, identity mapping, deduplication, and enrichment workflows.
  • Build scalable processing frameworks for PubMed, clinical trials, ct.gov, and other data sources.
  • Drive infrastructure optimization to improve throughput, runtime, observability, and cost efficiency.
  • Partner with AI/ML teams to integrate matching models into EMERALD and improve precision and recall.
  • Lead complex technical initiatives from architecture through deployment; mentor engineers and promote best practices.
Required Skills & Qualifications
  • Experience with Spark and PySpark.
  • Strong understanding of entity resolution techniques.
  • Proficiency in data processing frameworks.
  • Knowledge of healthcare data sources and standards.
  • Experience with infrastructure optimization strategies.
  • Familiarity with AI/ML model integration.
  • Strong mentoring and leadership skills.
  • Ability to drive complex technical initiatives.
  • Excellent problem-solving skills.
  • Proficient in best engineering practices.

No forms. Your profile is generated instantly.

Job Type: Remote

Salary: Not Disclosed

Experience: Entry

Duration: Months

Share this job:

Similar Jobs

Principal AI Engineer Role

Posted 81 days ago

Hire a remote Principal AI Engineer

Develop customer experience automation solutions

Ai Automation AWS Cloud Computing

ML Engineer - AdTech

Posted 81 days ago

Design and implement ML systems|Apply optimization strategies|Collaborate with teams|Analyze data

r user behavior|Develop data

C++ Data Analysis Java Machine Learning

Staff Software Engineer, Tax

Posted 81 days ago

Lead and scale tax engineering systems at Airbnb

Collaborate cross-functionally on global platform initiatives

Apis Architecture Backend Development Cloud Platforms

Staff Software Engineer - Biztech

Posted 81 days ago

Solving challenging and unique problems in Global Tax Engineering at Airbnb

Promoting sustainable engineering practices and well-being in the work environment

Architecture Backend Development Engineer Fintech

Compliance Engineering Manager

Posted 81 days ago

Lead a team focused on compliance with global financial regulations in the Payments space

Collaborate extensively with cross-functional teams to ensure compliance is integrated throughout the platform

Architecture Communication Compliance Cross-functional Collaboration

Staff Software Engineer

Posted 81 days ago

Revolutionize enterprise data operations through AI solutions.

Automate and accelerate data tasks for overworked data teams.

Ai Airflow Ansible Api Development

Senior Data Scientist Role

Posted 81 days ago

Hire a remote Senior Data Scientist

Enhance product with data-driven insights

A/b Testing Big Data Communication Cross-functional Communication

Remote Senior Data Engineer

Posted 81 days ago

Hiring a remote Senior Data Engineer for Apollo

Full-time position in Poland

AWS BigQuery Cloud Data

Senior Backend Engineer

Posted 81 days ago

Develop scalable backend solutions

Mentor team members

Angular Api Development Architecture AWS

Senior AI Engineer Role

Posted 81 days ago

Build and deploy scalable AI systems for production use.

Develop advanced multi-agent architectures and conversational AI.

Api Integration Architecture AWS Backend Development

Senior Backend Engineer Role

Posted 81 days ago

Design scalable backend solutions

Lead full software development lifecycle

Agile Methodologies Android Android development Apache Kafka

Staff ML Engineer, Apollo

Posted 81 days ago

Lead development of scalable ML systems

Advance Apollo's AI-native product features

Airflow Architecture Databricks Engineer

Senior ML Engineer, Remote

Posted 81 days ago

Design and productionize scalable machine learning systems

Personalize user experiences using data-driven models

Cloud Computer science Databricks Engineer

Senior ML Engineer II at Apollo

Posted 81 days ago

Build and productionize Machine Learning models for Apollo products

Optimize users' experience at all stages of their product journey

Airflow Ai Systems Cloud Computer science

Senior Product Manager

Posted 81 days ago

Build vision, strategy, and roadmap for new product line

Incorporate data analysis & research for product decisions

Ab testing Agile Agile Methodology Analytical Skills

Data Architect Role

Posted 81 days ago

Design scalable data architectures, Lead big data systems development, Integrate AI and

ion, Deploy production services, Drive thought

AWS Docker Gitlab Hadoop

Enterprise Account Executive

Posted 81 days ago

Manage key enterprise accounts effectively.

Drive revenue growth through strategic planning.

Account Executive Account manager Client Relationship Management Cloud Computing

AI Team Engineering Lead

Posted 81 days ago

. Lead and manage the AI Team effectively

. Drive innovation in AI technologies

Big Data Cloud Computing Data Science Java

Applied AI Engineer

Posted 81 days ago

Guide customers through product journey

Build custom demos and prototypes

Ai Tools Engineer JavaScript LLMs

Senior AI Product Manager

Posted 81 days ago

Drive AI quality in collaboration products

Lead remote cross-functional teams

Agile Methodologies Ai Data Analysis Machine Learning

AI Senior Design Manager

Posted 81 days ago

Lead AI-focused design initiatives

Manage and mentor remote design teams

Ai Collaboration Tools Cross-functional Collaboration Machine Learning

Senior Product Designer

Posted 81 days ago

Design exceptional products from start to finish.

Iterate actively with prototypes.

Ai Communication Communication Skills Design Systems

Staff Data Scientist Lead

Posted 81 days ago

Provide technical leadership in data science

Develop and implement advanced analytics models

Big Data Hadoop hive Machine Learning

Senior Data Scientist

Posted 81 days ago

Drive cross-functional data initiatives

Collaborate with various teams to uncover insights

Analytics Big Data Bi tools Data Mining

Senior Data Scientist Project

Posted 81 days ago

Drive data-driven strategies for product development.

Enhance business and customer impact through data analysis.

Big Data Hadoop Machine Learning Numpy

Creative Lead - Woo

Posted 81 days ago

Lead design teams and drive brand campaigns

Enhance user experiences through creative strategies

Brand Design Campaign Management Communication Community engagement

Full Stack Tech Lead Role

Posted 81 days ago

Lead and architect full-stack applications

Mentor and guide engineering teams

AWS Cloud infrastructure Database Design Database Optimization

Principal Data Engineer

Posted 81 days ago

Set technical direction for data initiatives

Design and build scalable data pipelines

Architecture Databricks Data Compliance Data Modeling

Compliance Analyst Lead

Posted 81 days ago

Ensure high-quality onboarding for charities and campaigns

Safeguard JustGiving from financial crime risks

Analyst Compliance Data Analytics Documentation

Principal Mobile App Engineer

Posted 81 days ago

Contribute to software application design and development

Optimize performance of critical components

Apis Architecture Communication Skills Debugging

Sr. Principal - HRIS

Posted 81 days ago

Lead HR transformation through Workday optimization

Guide cross-functional teams in complex Workday changes

Ai Automation Data Analytics Documentation

Senior Product Manager - Finance

Posted 81 days ago

Drive the strategy, roadmap, and delivery of the Financial Models product.

Serve as a subject matter expert for financial modeling within the Financial Platform.

Accounting Agile Agile Development Cross-functional Collaboration

Senior Product Manager-Finance

Posted 81 days ago

Seeking an experienced Product Manager with a focus on Finance & Accounting.

Driving the strategy, roadmap, and delivery of Financial Models within a Financial Platform.

Agile Cross-functional Collaboration Cybersecurity Finance & Accounting

Principal Product Manager - AI Platform

Posted 81 days ago

Lead and drive product initiatives for AI and Data Product Platform

Adopt and implement data mesh framework across the organization

Agile Agile Development Databricks Financial Acumen

Principal Product Manager

Posted 81 days ago

Lead multiple product initiatives from concept to delivery.

Drive data business transformation and enhance customer outcomes.

Agile Agile Development Analytics Databricks

Senior Data Architect for Blackbaud

Posted 81 days ago

Lead data strategy and architecture for Blackbaud

Design breakthrough products in Data Intelligence

Big Data Databricks Data Modeling Machine Learning

AI Sales Specialist Role

Posted 81 days ago

Drive adoption of AI-powered solutions

Support and enable sales strategies

Analytics Communication Machine Learning Natural Language Processing

Enterprise Data Architect Role

Posted 81 days ago

Design and oversee enterprise-wide data architecture

Develop strategies for data platforms and analytics

AWS Big Data Databricks Data Modeling

Principal Data Engineer Role

Posted 81 days ago

Modernize and optimize legacy data warehouse systems

Design and implement scalable data pipelines

Architecture AWS Databricks Devops

Principal Product Manager, Data

Posted 81 days ago

Lead strategic product initiatives for data and AI platforms

Drive organizational adoption of data mesh framework

Agile Agile Methodologies Databricks Financial Acumen

AI/ML Senior Engineer

Posted 81 days ago

Design and implement scalable AI/ML models

Optimize and maintain enterprise AI solutions

AWS Data Modeling Deep Learning Engineer

Software Engineer II

Posted 81 days ago

Design and develop a cloud native API first platform for a Security Knowledge Platform™

Drive modernization efforts towards a reliable platform

Agile Ci/cd Pipelines Cloud Cloud-native architecture

Senior Software Engineer

Posted 81 days ago

Design and develop cloud-native API first platform for patented data and AI-powered Security Knowledge Platform

Build and maintain integrations connecting platform with customer systems, tools, and more

Agile Agile Development Ai Tools Android

Senior Software Engineering Manager

Posted 81 days ago

Lead and manage a highly skilled engineering team

Drive architectural evolution towards a high-performance ecosystem

Agile Ai Tools API Backend Development

Senior Manager, Software Engineering

Posted 81 days ago

Lead and grow a skilled full-stack engineering team.

Drive architectural initiatives for high-performance ecosystem.

Agile Ai Tools API Cloud Native

Cloud Field Engineering Manager

Posted 81 days ago

Lead and develop a field engineering team

Promote and solve customer problems with cloud technologies

AWS Ceph Cloud Devops

Junior Cloud Field Engineer

Posted 81 days ago

Assist global companies in adopting private cloud infrastructure

Design and implement cloud solutions using Linux Ubuntu, OpenStack, Kubernetes, etc.

AWS Ceph Cloud Cloud Computing