Model Evaluation Engineer

New

Skills

Benchmarking Pipelines Documentation Evaluation Datasets Evaluation Metrics Model Evaluation Python Qualitative Feedback Analysis SQL Stress Testing Systematic Experiments

We are seeking a Research Engineer focused on evaluations to take ownership of end-to-end and integration-level model evaluations. This role will involve assessing models across various metrics such as accuracy, latency, and feature-specific performance.

Key Responsibilities
  • Own end-to-end and integration-level model evaluation across accuracy, latency, and feature-specific metrics.
  • Build and maintain competitive benchmarking pipelines.
  • Design and run systematic experiments to measure the impact of model changes.
  • Onboard, curate, and maintain evaluation datasets.
  • Create evaluation subsets to stress-test specific capabilities and edge cases.
  • Define evaluation metrics for real-world performance.
  • Translate qualitative customer feedback into quantifiable evaluation criteria.
  • Work with customer-facing teams to understand pain points and convert them into research priorities.
  • Maintain clean evaluation pipelines and clear documentation.
  • Identify evaluation gaps proactively and propose solutions.
Required Skills & Qualifications
  • Strong understanding of ML fundamentals and ability to interpret results and debug issues without retraining from scratch.
  • Proficient in Python, including writing clean evaluation scripts and working with data pipelines.
  • Comfortable with SQL and cloud infrastructure.
  • Understanding of good evaluation metrics and ensuring statistical rigor.
  • Familiarity with voice agent stack, including VAD, ASR, turn detection, LLM, and TTS systems interaction.
  • Tinkerer mentality with a preference for shipping and iterating quickly.
  • Excellent communication skills to explain technical results and summarize findings.
  • Ownership mindset to proactively fill evaluation gaps.
  • Ability to work at least 3-4 hours overlapping with Eastern US Time Zone.

No forms. Your profile is generated instantly.

Job Type: Remote

Salary: Not Disclosed

Experience: Entry

Duration: Months

Share this job:

Similar Jobs

Model Evaluation Engineer

Posted 19 days ago

Conduct comprehensive model evaluations.

Establish and maintain benchmarking pipelines.

Benchmarking Pipelines Cloud Infrastructure Data Pipelines Evaluation Metrics

Model Evaluation Engineer

Posted 12 days ago

Oversee model evaluation across various metrics.

Build and maintain benchmarking pipelines.

Benchmarking Pipelines Cloud Infrastructure Customer Feedback Analysis Evaluation Datasets

Model Evaluation Engineer

Posted 14 days ago

Conduct comprehensive model evaluations.

Develop and maintain benchmarking pipelines.

Benchmarking Pipelines Cloud Infrastructure Data Pipelines Documentation

Model Evaluation Engineer

Posted 14 days ago

Lead end-to-end model evaluation processes.

Develop and maintain benchmarking pipelines.

Benchmarking Pipelines Cloud Infrastructure Dataset Curation Documentation

Model Evaluation Engineer

Posted 13 days ago

Conduct comprehensive model evaluations.

Build and maintain benchmarking pipelines.

Benchmarking Pipelines Cloud Infrastructure Customer Feedback Analysis Data Pipeline Management

Model Evaluation Engineer

Posted 8 days ago

Conduct comprehensive model evaluations.

Build and maintain benchmarking pipelines.

Benchmarking Pipelines Cloud Infrastructure Data Pipelines Evaluation Metrics

Model Evaluation Engineer

Posted 7 days ago

Oversee model evaluation processes.

Build and maintain benchmarking pipelines.

Benchmarking Pipelines Cloud Infrastructure Customer Feedback Analysis Data Pipelines

Model Evaluation Engineer

New

Own end-to-end model evaluation.

Build and maintain benchmarking pipelines.

Benchmarking Pipelines Cloud Infrastructure Data Curation Documentation Practices

Model Evaluation Engineer

New

Conduct end-to-end model evaluations.

Build competitive benchmarking pipelines.

Benchmarking Pipelines Cloud Infrastructure Data Pipelines Experimental Design

Model Evaluation Engineer

New

Conduct end-to-end model evaluations.

Build and maintain benchmarking pipelines.

Benchmarking Pipelines Cloud Infrastructure Data Pipelines Documentation

C# Backend Engineer Remote

Posted 30 days ago

Develop and maintain backend C# services and APIs

Collaborate remotely with a distributed team

Agile Agile Methodologies Api Development Back-end

C# Development Engineer

Posted 30 days ago

Contribute to back-end services used by company products

Collaborate with a global team

Agile Agile Methodologies Ai Tools Api Development

Back-End C# Developer

Posted 30 days ago

Enhance back-end services for millions of users

Collaborate with a global team on complex API implementations

Agile Agile Methodology Api Integration Back-end

Sys Admin Support

Posted 30 days ago

Support critical infrastructure Manage device tools Provide desktop/laptop support Maintain

None

AWS Documentation Linux macOS

SOX Technical Control Testing

Posted 30 days ago

Evaluate and test IT controls for SOX compliance.

Collaborate with technology teams to assess and improve control design.

Accounting Change Management Communication Documentation

Remote Developer Support Engineer

Posted 30 days ago

Provide remote technical support

Assist with Airtable APIs and integrations

Apis Customer Service Customer Support Developer Advocate

Transaction Monitoring Lead

Posted 30 days ago

Hiring a Transaction Monitoring Lead to enhance compliance capabilities

Optimizing TM rules and alert systems to reduce false positives

Automation Documentation Fraud Detection Monitoring

Transaction Monitoring Manager

Posted 30 days ago

Lead and scale transaction monitoring operations

Optimize and automate AML controls and alert logic

Automation Data Analysis Documentation Monitoring

Regional Compliance Officer

Posted 30 days ago

Lead regional compliance oversight for Airtm in India.

Implement and monitor AML/CFT measures for the region.

Anti-money Laundering Audit Communication Compliance

Blockchain Solutions Engineer

Posted 30 days ago

Engage with customers to understand blockchain data needs

Design and implement tailored data analytics solutions

Blockchain technology Data Analytics Data Security Documentation

Growth Engineer at Allium

Posted 30 days ago

Optimize internal customer support processes

Automate customer interactions for efficiency

Blockchain Collaboration Tools Documentation Engineer

Remote Customer Support Specialist

Posted 30 days ago

Provide customer support for AllTrails platform users.

Resolve customer issues and inquiries.

Communication Skills Customer Support Documentation Email Support

Compliance Program Director

Posted 30 days ago

Lead and shape Alma's compliance program

Ensure alignment with industry regulations

Communication Compliance Director Documentation

Marketing Operations Specialist

Posted 30 days ago

Execute and optimize marketing campaigns using Marketo

Maintain and improve operational campaign workflows

adobe Airtable Asana Communication

Anaplan Solution Architect India

Posted 30 days ago

Architect and deliver Anaplan planning solutions

Translate business requirements into multi-dimensional models

Agile Agile Methodology Anaplan Business Analytics

Anaplan Solution Architect

Posted 30 days ago

Architect, design, and deliver Anaplan planning solutions

Translate client requirements into sophisticated models

Agile Agile Methodology Anaplan Business Analytics

API Support Engineer Role

Posted 30 days ago

Provide first-line API support to customers

Troubleshoot and resolve integration issues

Api Documentation Cross-functional Collaboration Cross-functional Communication Customer success

Remote Technical Support Engineer

Posted 30 days ago

Provide remote technical support

Enhance customer satisfaction

Collaboration Communication Skills Customer Support Dns

Support & QA Engineer

Posted 30 days ago

Drive product quality through support insights

Proactively identify and resolve technical issues

Debugging Documentation Engineer QA

WordPress Playground Developer

Posted 30 days ago

Develop low-level PHP tools for WordPress Core

Read and implement web standards proposals

Community engagement Documentation Editing Front end

CRM Business Analyst

Posted 30 days ago

Bridging gap between non-profit needs and technical solutions

Gathering and analyzing user requirements

Analyst Automation Business Analyst Crm

Staff IAM/PAM Administrator Role

Posted 30 days ago

Support identity and privileged access operations.

Monitor IAM and PAM platforms.

Automation AWS Cloud Cyberark

Compliance Analyst Lead

Posted 30 days ago

Ensure high-quality onboarding for charities and campaigns

Safeguard JustGiving from financial crime risks

Analyst Compliance Data Analytics Documentation

Sr. Principal - HRIS

Posted 30 days ago

Lead HR transformation through Workday optimization

Guide cross-functional teams in complex Workday changes

Ai Automation Data Analytics Documentation

Customer Contact Data Analyst

Posted 30 days ago

Lead analysis and resolution of customer contact data changes.

Enhance customer data health through defined playbooks.

Analyst Automation Communication Crm

UX Engineer - Senior

Posted 30 days ago

Design valuable and usable solutions for user problems

Collaborate with product and engineering teams

Collaboration Communication Css Documentation

Senior Auditor - SOX Compliance

Posted 30 days ago

Lead Sarbanes-Oxley compliance initiatives within the Internal Audit team

Collaborate with various departments to ensure control awareness and compliance

Accounting Audit Data Analytics Documentation

Sr Manager - Customer Support

Posted 30 days ago

Lead team development and mentorship

Drive operational efficiency and performance metrics

Collaboration Communication Communication Skills Continuous improvement

Customer Success Implementation Consultant

Posted 30 days ago

Help non-profit organizations excel through product implementation and consulting

Maintain strong customer relationships

Communication Crm Customer Relations Database Management

Education Services Developer

Posted 30 days ago

Collaborate with internal teams to design and build virtual training sandbox environments

Develop and maintain database and lab environments to support training needs

Agile Angular Azure C#

Customer Support Staff

Posted 30 days ago

Provide high-quality technical support to B2B clients.

Troubleshoot software incidents and escalate complex issues.

Apis Cloud Communication Crm

Marketing Ops Specialist

Posted 30 days ago

Hiring a remote Marketing Operations Specialist

Full-time position

Asana Automation Communication Communication Tools

BBCRM Business Analyst

Posted 30 days ago

Bridge gap between non-profit needs and technical solutions

Gather and analyze user requirements

Analyst Automation Business Analyst Crm

Customer Success Services Consultant

Posted 30 days ago

Building and maintaining customer relationships

Leading and managing projects

Customer Relations Documentation Leadership Microsoft Excel

BBCRM Analyst Job

Posted 30 days ago

Analyze user requirements and document them effectively

Bridge gap between non-profit needs and technical solutions

Analyst Automation Business Analyst Client Communication

LMS Administrator

Posted 30 days ago

Collaborate with training team in LMS tasks

Execute LMS strategies and maintain data infrastructure

Agile Methodologies Automation Communication Communication Skills

Virtual Training Environment Developer

Posted 30 days ago

Design and maintain virtual training environments

Implement cloud-based architectures and DevOps practices

Agile C# Communication Documentation

Workday Integrations Specialist

Posted 30 days ago

Manage Workday integrations from design to support

Ensure data security and compliance in all integrations

Computer science Documentation Hris Security

LMS Administrator Role

Posted 30 days ago

Manage and optimize LMS content and structure

Ensure data accuracy and system quality

Agile Methodologies Automation Communication Css