AI QA Trainer

New

Skills

Adversarial Testing Artificial Intelligence Data Science Linguistics Machine Learning OpenAI Evals PyTest Quality Assurance RAG Evaluators Test Automation

We are seeking an AI QA Trainer for a freelance project focused on evaluating and improving large language models. This role involves interacting with AI systems to ensure their factual accuracy and logical soundness.

Key Responsibilities
  • Converse with the model using real-world prompts.
  • Verify factual accuracy and logical soundness of AI outputs.
  • Design and execute test plans and regression suites.
  • Build rubrics and establish pass/fail criteria.
  • Capture reproducible error traces and analyze root causes.
  • Suggest improvements to prompts, guardrails, and performance metrics.
Required Skills & Qualifications
  • Advanced degree in Computer Science, Data Science, Linguistics, or Statistics.
  • Experience in QA for ML/AI systems, including safety and red-team experience.
  • Proficiency in test automation frameworks (e.g., PyTest).
  • Hands-on experience with LLM evaluation tools (OpenAI Evals, RAG evaluators, W&B).
  • Strong skills in rubric design and adversarial testing.
  • Ability to perform regression testing at scale.
  • Excellent communication skills.

No forms. Your profile is generated instantly.

Job Type: Remote

Salary: Not Disclosed

Experience: Entry

Duration: Months

Share this job:

Similar Jobs

Senior Golang Developer Role

Posted 96 days ago

Develop cloud-based cyber protection solutions

Design and maintain high-load distributed services

Algorithms Architecture Cloud Services Data Structures

Principal AI Engineer Role

Posted 96 days ago

Hire a remote Principal AI Engineer

Develop customer experience automation solutions

Ai Automation AWS Cloud Computing

ML Engineer - AdTech

Posted 96 days ago

Design and implement ML systems|Apply optimization strategies|Collaborate with teams|Analyze data

r user behavior|Develop data

C++ Data Analysis Java Machine Learning

Staff Software Engineer

Posted 96 days ago

Revolutionize enterprise data operations through AI solutions.

Automate and accelerate data tasks for overworked data teams.

Ai Airflow Ansible Api Development

Remote QA Engineer

Posted 96 days ago

- Conduct thorough testing of Anagram's insurance billing software - Identify and report software

bugs and defects - Collaborate with the development team to ensure quality standards are met -

Agile Methodologies Communication Skills Database Management Problem-solving

Senior Data Scientist Role

Posted 96 days ago

Hire a remote Senior Data Scientist

Enhance product with data-driven insights

A/b Testing Big Data Communication Cross-functional Communication

Remote Quality Engineer

Posted 96 days ago

Hiring a remote Quality Engineer for Apollo

Full-time position in Poland

Collaboration Engineer Problem-solving QA

Senior Backend Engineer

Posted 96 days ago

Develop scalable backend solutions

Mentor team members

Angular Api Development Architecture AWS

Senior AI Engineer Role

Posted 96 days ago

Build and deploy scalable AI systems for production use.

Develop advanced multi-agent architectures and conversational AI.

Api Integration Architecture AWS Backend Development

Senior Backend Engineer Role

Posted 96 days ago

Design scalable backend solutions

Lead full software development lifecycle

Agile Methodologies Android Android development Apache Kafka

Staff ML Engineer, Apollo

Posted 96 days ago

Lead development of scalable ML systems

Advance Apollo's AI-native product features

Airflow Architecture Databricks Engineer

Senior ML Engineer, Remote

Posted 96 days ago

Design and productionize scalable machine learning systems

Personalize user experiences using data-driven models

Cloud Computer science Databricks Engineer

Senior ML Engineer II at Apollo

Posted 96 days ago

Build and productionize Machine Learning models for Apollo products

Optimize users' experience at all stages of their product journey

Airflow Ai Systems Cloud Computer science

Senior Product Manager

Posted 96 days ago

Build vision, strategy, and roadmap for new product line

Incorporate data analysis & research for product decisions

Ab testing Agile Agile Methodology Analytical Skills

(Senior)QA Automation Engineer

Posted 96 days ago

Develop and execute test plans efficiently, Contribute to automation tools development, Implement

ntinuous integration practices, Collaborate with cross-functional teams, Maintain high quality

Agile Appium CI/CD Java

Enterprise Account Executive

Posted 96 days ago

Manage key enterprise accounts effectively.

Drive revenue growth through strategic planning.

Account Executive Account manager Client Relationship Management Cloud Computing

AI Team Engineering Lead

Posted 96 days ago

. Lead and manage the AI Team effectively

. Drive innovation in AI technologies

Big Data Cloud Computing Data Science Java

Applied AI Engineer

Posted 96 days ago

Guide customers through product journey

Build custom demos and prototypes

Ai Tools Engineer JavaScript LLMs

Marketing Localization Manager

Posted 96 days ago

Hire a remote manager in Poland

Lead marketing localization projects

Cross-functional Collaboration Marketing Strategy Project Management Quality Assurance

Senior AI Product Manager

Posted 96 days ago

Drive AI quality in collaboration products

Lead remote cross-functional teams

Agile Methodologies Ai Data Analysis Machine Learning

AI Senior Design Manager

Posted 96 days ago

Lead AI-focused design initiatives

Manage and mentor remote design teams

Ai Collaboration Tools Cross-functional Collaboration Machine Learning

Staff Data Scientist Lead

Posted 96 days ago

Provide technical leadership in data science

Develop and implement advanced analytics models

Big Data Hadoop hive Machine Learning

Support & QA Engineer

Posted 96 days ago

Drive product quality through support insights

Proactively identify and resolve technical issues

Debugging Documentation Engineer QA

Senior Data Scientist

Posted 96 days ago

Drive cross-functional data initiatives

Collaborate with various teams to uncover insights

Analytics Big Data Bi tools Data Mining

Senior Data Scientist Project

Posted 96 days ago

Drive data-driven strategies for product development.

Enhance business and customer impact through data analysis.

Big Data Hadoop Machine Learning Numpy

Senior Analytics Engineer

Posted 96 days ago

Hiring a Senior Analytics Engineer remotely

Axios - Smart brevity

Airflow Analytics BigQuery Bi tools

Associate Director Production

Posted 96 days ago

Recruit a remote production leader

Deliver high-quality Axios Live events

Budget Management Communication Skills Content Strategy Project Management

Principal Mobile App Engineer

Posted 96 days ago

Contribute to software application design and development

Optimize performance of critical components

Apis Architecture Communication Skills Debugging

Sr. Principal - HRIS

Posted 96 days ago

Lead HR transformation through Workday optimization

Guide cross-functional teams in complex Workday changes

Ai Automation Data Analytics Documentation

Senior Product Manager - Finance

Posted 96 days ago

Drive the strategy, roadmap, and delivery of the Financial Models product.

Serve as a subject matter expert for financial modeling within the Financial Platform.

Accounting Agile Agile Development Cross-functional Collaboration

Senior Product Manager-Finance

Posted 96 days ago

Seeking an experienced Product Manager with a focus on Finance & Accounting.

Driving the strategy, roadmap, and delivery of Financial Models within a Financial Platform.

Agile Cross-functional Collaboration Cybersecurity Finance & Accounting

Principal Product Manager - AI Platform

Posted 96 days ago

Lead and drive product initiatives for AI and Data Product Platform

Adopt and implement data mesh framework across the organization

Agile Agile Development Databricks Financial Acumen

Principal Product Manager

Posted 96 days ago

Lead multiple product initiatives from concept to delivery.

Drive data business transformation and enhance customer outcomes.

Agile Agile Development Analytics Databricks

Senior Data Architect for Blackbaud

Posted 96 days ago

Lead data strategy and architecture for Blackbaud

Design breakthrough products in Data Intelligence

Big Data Databricks Data Modeling Machine Learning

AI Sales Specialist Role

Posted 96 days ago

Drive adoption of AI-powered solutions

Support and enable sales strategies

Analytics Communication Machine Learning Natural Language Processing

Enterprise Data Architect Role

Posted 96 days ago

Design and oversee enterprise-wide data architecture

Develop strategies for data platforms and analytics

AWS Big Data Databricks Data Modeling

Principal Product Manager, Data

Posted 96 days ago

Lead strategic product initiatives for data and AI platforms

Drive organizational adoption of data mesh framework

Agile Agile Methodologies Databricks Financial Acumen

AI/ML Senior Engineer

Posted 96 days ago

Design and implement scalable AI/ML models

Optimize and maintain enterprise AI solutions

AWS Data Modeling Deep Learning Engineer

Dive Travel Website Specialist

Posted 96 days ago

Grow website traffic and user engagement

Enhance site UX and design for conversions

Content Creation Content Marketing Copywriting Css

QA Analyst (English & Punjabi)

Posted 96 days ago

Ensure quality and conformance of Bobtail's services

Assist in tracking and documenting quality levels and goals/KPIs

Analytical Skills Communication Documentation English Language

Remote Data Research Assessor

Posted 96 days ago

Improve search engine relevance and quality

Support AI model development for global brands

Online Research Quality Assurance

Staff Engineer - Quality

Posted 96 days ago

Lead quality assurance efforts for software project.

Design and execute test plans for software quality.

Agile Methodologies Collaboration Communication Skills Engineer

Brooklyn Liquor Store Manager

Posted 96 days ago

Lead warehouse operations, Improve KPIs, Drive financial performance, Ensure compliance, Develop

m

Budgeting Compliance Conflict resolution Financial Management

Senior Software Engineer

Posted 96 days ago

Design and develop cloud-native API first platform for patented data and AI-powered Security Knowledge Platform

Build and maintain integrations connecting platform with customer systems, tools, and more

Agile Agile Development Ai Tools Android

Remote Software Engineer Support

Posted 96 days ago

Enhance customer support processes through software solutions.

Collaborate with cross-functional teams to address technical issues.

Agile Methodologies Api Integration Customer Support Database Management

Engineering Manager - Python and K8s

Posted 96 days ago

Build a world-class devops culture in corporate information systems

Transform IS team into an extension of product engineering capability

Agile Development Architecture Cloud Devops