AI QA Trainer

New

Skills

Adversarial Testing Artificial Intelligence Data Science Linguistics Machine Learning OpenAI Evals PyTest Quality Assurance RAG Evaluators Test Automation

We are seeking an AI QA Trainer for a freelance project focused on evaluating and improving large language models. This role involves interacting with AI systems to ensure their factual accuracy and logical soundness.

Key Responsibilities
  • Converse with the model using real-world prompts.
  • Verify factual accuracy and logical soundness of AI outputs.
  • Design and execute test plans and regression suites.
  • Build rubrics and establish pass/fail criteria.
  • Capture reproducible error traces and analyze root causes.
  • Suggest improvements to prompts, guardrails, and performance metrics.
Required Skills & Qualifications
  • Advanced degree in Computer Science, Data Science, Linguistics, or Statistics.
  • Experience in QA for ML/AI systems, including safety and red-team experience.
  • Proficiency in test automation frameworks (e.g., PyTest).
  • Hands-on experience with LLM evaluation tools (OpenAI Evals, RAG evaluators, W&B).
  • Strong skills in rubric design and adversarial testing.
  • Ability to perform regression testing at scale.
  • Excellent communication skills.

No forms. Your profile is generated instantly.

Job Type: Remote

Salary: Not Disclosed

Experience: Entry

Duration: Months

Share this job:

Similar Jobs

Senior Product Manager, AI

Posted 11 days ago

Hiring for a Senior Product Manager position focused on AI.

Remote work opportunity available across the United States.

Agile Methodologies Artificial Intelligence Cross-functional Collaboration Data Analysis

Junior AI Software Engineer

Posted 11 days ago

To recruit a Junior AI Software Engineer for Santander Argentina.

To engage candidates in innovative technology projects.

API Development Artificial Intelligence Cloud Solutions Code Review

Security Program Manager

Posted 10 days ago

Collaborate with sponsors and Security to assess security risks.

Act as DRI for key security initiatives.

Agile Methodology Artificial Intelligence Collaboration Tools Compliance Management

Machine Learning Engineering Manager

Posted 10 days ago

Lead and grow a team of ML engineers.

Define vision and roadmap for cart perception.

Artificial Intelligence Computer Vision Deep Learning GCP

Growth Engineer Role

Posted 10 days ago

Transform BTRST token visibility and value.

Design token-powered growth loops to incentivize user behavior.

AI-Native Tools Artificial Intelligence Behavioral Economics Cohort Analysis

Growth Engineering Lead

Posted 7 days ago

Transform BTRST into a key factor for user engagement.

Develop systems for token visibility and user education.

Artificial Intelligence Behavioral Economics Cohort Analysis Gamification

AI Agent Testing Specialist

Posted 7 days ago

Design evaluation scenarios for AI agents.

Create test cases to simulate human tasks.

Artificial Intelligence Data Annotation JavaScript JSON

Compliance Data Science Director

Posted 7 days ago

Lead AI initiatives in compliance.

Collaborate with product and compliance teams.

AML Artificial Intelligence Data Engineering Data Science

Senior Manager, Engineering

Posted 7 days ago

Lead and develop a high-performing team in FDE.

Ensure clear communication and quality in project outcomes.

Agile Methodologies Artificial Intelligence Communication Skills Customer Engagement

Channel Account Manager

Posted 7 days ago

Build and maintain partner relationships.

Recruit partners into PolyAI’s channel program.

Artificial Intelligence CCaaS Channel Partner Management Cross-Functional Collaboration

Corporate Strategy Director

Posted 7 days ago

Lead corporate strategy and execution.

Identify and leverage AI-driven opportunities.

Artificial Intelligence Corporate Strategy Cross-functional Leadership DevOps

Growth Engineering Lead

Posted 4 days ago

Transform BTRST token into a core engagement tool.

Design and implement growth systems within the product.

Artificial Intelligence Behavioral Economics Cohort Analysis Growth Hacking

Growth Engineering Lead

Posted 7 days ago

Transform token visibility to drive engagement.

Design and test token-powered growth loops.

AI-Native Tools Artificial Intelligence Behavioral Economics Cohort Analysis

Growth Engineering Lead

Posted 6 days ago

Transform the BTRST token into a core component of user engagement.

Design token-powered growth loops to drive marketplace behavior.

Artificial Intelligence Behavioral Economics Cohort Analysis Growth Hacking

Growth Engineering Lead

Posted 6 days ago

Transform BTRST token into a key part of user engagement.

Design systems to incentivize user behaviors and marketplace participation.

AI-Native Tool Utilization Artificial Intelligence Behavioral Economics Cohort Analysis

Growth Engineering Lead

Posted 5 days ago

Enhance the visibility and value of the BTRST token.

Create engaging and educational content about the token.

AI-native Product Building Artificial Intelligence Behavioral Economics Cohort Analysis

Growth Engineering Lead

Posted 5 days ago

Transform BTRST token into a core component of user engagement.

Design and implement token visibility and educational tools.

Artificial Intelligence Behavioral Economics Cohort Analysis Growth Hacking

AI Security Intern

New

Develop understanding of AI/ML and LLMs.

Research AI security threats and vulnerabilities.

Artificial Intelligence AWS Bedrock Blue Teaming Data Leakage

AI Engineering Lead

New

Lead AI initiatives from concept to production.

Define and implement AI architecture and workflows.

AI-native Workflows Architecture Review Artificial Intelligence Automation

Solutions Engineer Role

New

Develop and deliver technical presentations.

Communicate solution value to diverse audiences.

Application GRC Artificial Intelligence AWS Cloud Security

Solutions Engineer Role

New

Develop and deliver technical presentations.

Communicate solution value to audiences.

Application GRC Artificial Intelligence AWS Azure

Data Analyst

Posted 22 days ago

Operational Excellence Program Leadership Strategic Prioritization Cross-functional

aybook

Crm Cross-functional Collaboration Customer success Data Science

AI Team Engineering Lead

Posted 22 days ago

. Lead and manage the AI Team effectively

. Drive innovation in AI technologies

Big Data Cloud Computing Data Science Java

Senior Analytics Engineer

Posted 22 days ago

Hiring a Senior Analytics Engineer remotely

Axios - Smart brevity

Airflow Analytics BigQuery Bi tools

Platform Product Manager Role

Posted 22 days ago

Drive strategic platform product development

Enable scalable collaboration and sync solutions

Agile Methodologies Agile Methodology Ai Tools API Design

Senior Data Scientist Role

Posted 22 days ago

Hire a remote Senior Data Scientist

Enhance product with data-driven insights

A/b Testing Big Data Communication Cross-functional Communication

Senior Data Scientist

Posted 22 days ago

Drive cross-functional data initiatives

Collaborate with various teams to uncover insights

Analytics Big Data Bi tools Data Mining

Insights Data Scientist Role

Posted 22 days ago

Analyze user behavior to drive key outcomes

Improve data accessibility and literacy

Analytics Bi tools Communication Data Science

Engineering Manager at Branch International

Posted 22 days ago

Lead a team of engineers in developing and maintaining products and systems.

Recruit, grow, and empower a team of engineers.

Collaboration Data-driven decision making Data Science Engineer

Machine Learning Engineer Role

Posted 22 days ago

Develop machine learning models for credit decisions

Automate customer service with NLP/LLMs

Cloud Cloud Computing Data Pipelines Data Science

Staff Software Engineer (DevTools)

Posted 22 days ago

Lead development efforts on DVC product and ecosystem.

Engage with the community and support users.

Agile Data Science Engineer Git

Security Product Manager Role

Posted 22 days ago

Lead development of Security Data Fabric and Exposure Management platform

Unify and contextualize security signals using AI/data science

Ai/ml Cloud Platforms Cybersecurity Data Science

Cyber Threat Analyst

Posted 22 days ago

Continuous learning and adaptability to emerging threats, Efficiently analyze and understand

, Clear communication of technical findings, Focus on automation for workflow efficiency, Utilize

Algorithms Data Science Development Machine Learning

VP Analytics & AI Leadership

Posted 22 days ago

Lead and scale a global analytics team

Drive business impact through advanced analytics and AI

Ab testing A/b Testing Analytics Customer Experience

Khan Analytics Growth Insights

Posted 22 days ago

Develop metrics & KPIs for strategic decisions

Analyze user behavior for insights and improvements

Data Compliance Data Modeling Data Science Data Security

Compliance Data Analyst Role

Posted 22 days ago

Develop and automate compliance dashboards and reports

Support regulatory reporting and audit readiness

Airflow AWS Data Analysis Data Analyst

Senior React.js Full-stack Developer

Posted 22 days ago

Seeking a talented Senior Developer for a remote job with decent compensation

Connecting Senior Developers with startups in the US and Europe

Architecture AWS Azure C#

Remote Education AI Engineer

Posted 22 days ago

Hiring a remote Education AI Engineer 3

Full-time position in the United States

Ai Data Science Deep Learning Engineer

Staff Data Scientist - Operations

Posted 22 days ago

Utilize data science techniques to optimize operational processes.

Collaborate with cross-functional teams for data-driven decisions.

Algorithm Development Cross-functional Collaboration Data Data Analysis

Marketing Data Science Manager

Posted 22 days ago

Lead and manage the marketing data science team

Develop data-driven marketing strategies

Communication Data Analysis Data Science Leadership

Prompt Engineer

Posted 22 days ago

Craft, optimize, evaluate, and benchmark prompts for enhanced AI performance.

Work collaboratively with Customer Success and Data Science teams to develop effective AI solutions.

Data Science Data Visualization Engineer Linguistics

Data Science Manager

Posted 22 days ago

Lead and mentor a team of Data Scientists.

Define and execute AI/ML strategy aligned with business objectives.

Data Science Deep Learning Hadoop LLMs

AI Prompt Engineer

Posted 22 days ago

Craft, optimize, and evaluate prompts for enhanced AI performance.

Develop client-specific solutions using NLP and ML principles.

Agile Development Ai Frameworks Data Science Data Visualization

AI Data Science Manager

Posted 22 days ago

Lead and mentor a high-performing data science team

Architect and deploy advanced AI models for customer service

Data Science Deep Learning Hadoop LLMs

AI Engineer at Omada Health

Posted 22 days ago

. Develop AI solutions for personalized health programs. •

. Collaborate with cross-functional teams on AI projects. •

Data Science Deep Learning Machine Learning Natural Language Processing

Data Analytics Consultant

Posted 22 days ago

Drive value for clients through data analytics.

Specialize in tools like Alteryx and KNIME for solution development.

Analytics Data Science Data Warehousing Knime