AI Agent Testing Specialist

New

Skills

Artificial Intelligence Data Annotation JavaScript JSON Machine Learning Natural Language Processing Python Quality Assurance Software Testing YAML

Mindrift is seeking an AI Agent Testing Specialist to design structured evaluation scenarios for LLM-based agents. This freelance, remote opportunity allows you to leverage your IT background while contributing to innovative AI projects.

Key Responsibilities
  • Design structured test scenarios based on real-world tasks.
  • Define golden paths and acceptable agent behavior.
  • Annotate task steps, expected outputs, and edge cases.
  • Collaborate with developers to test scenarios and enhance clarity.
  • Review agent outputs and adapt tests as necessary.
Required Skills & Qualifications
  • Bachelor's/Master's Degree in Computer Science, Software Engineering, or related fields.
  • Background in QA, software testing, data analysis, or NLP annotation.
  • Strong understanding of test design principles.
  • Excellent written communication skills in English.
  • Familiarity with structured formats like JSON/YAML.
  • Basic experience with Python and JavaScript.
  • Curiosity about AI-generated content and agent behavior.
  • Ability to switch between tasks and adapt to complex guidelines.

No forms. Your profile is generated instantly.

Job Type: Remote

Salary: Not Disclosed

Experience: Entry

Duration: Months

Share this job:

Similar Jobs

Senior Product Manager, AI

Posted 4 days ago

Hiring for a Senior Product Manager position focused on AI.

Remote work opportunity available across the United States.

Agile Methodologies Artificial Intelligence Cross-functional Collaboration Data Analysis

Junior AI Software Engineer

Posted 4 days ago

To recruit a Junior AI Software Engineer for Santander Argentina.

To engage candidates in innovative technology projects.

API Development Artificial Intelligence Cloud Solutions Code Review

Security Program Manager

Posted 3 days ago

Collaborate with sponsors and Security to assess security risks.

Act as DRI for key security initiatives.

Agile Methodology Artificial Intelligence Collaboration Tools Compliance Management

Machine Learning Engineering Manager

Posted 3 days ago

Lead and grow a team of ML engineers.

Define vision and roadmap for cart perception.

Artificial Intelligence Computer Vision Deep Learning GCP

Growth Engineer Role

Posted 3 days ago

Transform BTRST token visibility and value.

Design token-powered growth loops to incentivize user behavior.

AI-Native Tools Artificial Intelligence Behavioral Economics Cohort Analysis

Growth Engineering Lead

New

Transform BTRST into a key factor for user engagement.

Develop systems for token visibility and user education.

Artificial Intelligence Behavioral Economics Cohort Analysis Gamification

Compliance Data Science Director

New

Lead AI initiatives in compliance.

Collaborate with product and compliance teams.

AML Artificial Intelligence Data Engineering Data Science

Senior Manager, Engineering

New

Lead and develop a high-performing team in FDE.

Ensure clear communication and quality in project outcomes.

Agile Methodologies Artificial Intelligence Communication Skills Customer Engagement

AI Trainer for Gaming

Posted 15 days ago

Improve large language model accuracy

Support immersive game content creation

Collaboration Communication Data Analysis Data Annotation

AI Trainer for Games

Posted 15 days ago

Enhance AI model accuracy and performance

Annotate and evaluate in-game content

Content Development Data Annotation English Fluency Remote Collaboration

AI Trainer Role

Posted 15 days ago

Enhance AI dialogue quality

Provide high-quality training data

Ai Ai training Collaboration Communication

AI Data Specialist Role

Posted 15 days ago

Transform education and productivity through AI

Ensure high-quality labeled data for model training

Analyst Collaboration Tools Communication Skills Data Annotation

AI Data Specialist

Posted 15 days ago

Generate accurately labeled data for AI models.

Support model training and evaluation.

Collaboration Tools Communication Skills Data Annotation Data Labeling

Generalist - Language AI Evaluation

Posted 15 days ago

Evaluate LLM-generated responses

Conduct fact-checking on model responses

Ai Analytical Thinking Content Writing Data Annotation

Investment Banking AI Tutor

Posted 15 days ago

Enhance AI models in Investment Banking domain

Develop and maintain annotation standards

Ai AI Model Training Data Annotation Finance

AI Math & Stats Expert

Posted 15 days ago

Elevate AI math model quality

Develop and document annotation standards

Ai Data Annotation Mathematics phd

Voice Engineer

Posted 15 days ago

. Develop and maintain voice recognition tools. •

. Collaborate with researchers and developers. •

Agile Methodologies Data Annotation Git Machine Learning

AI Content Analyst Role

Posted 15 days ago

Assess accuracy and relevance of digital content

Support AI training and development

Ai Analytical Thinking Data Annotation English Fluency

AI Data Annotation Specialist

Posted 15 days ago

Recruit global remote freelancers

Improve AI model accuracy

Communication Skills Data Annotation English Machine Learning

AI Data Quality Associate

Posted 15 days ago

Hire remote AI data expert

Improve AI data quality

Data Data Annotation Data Cleaning Machine Learning

Italian Content Evaluation Specialist

Posted 15 days ago

Evaluate and improve digital content for Italian users.

Ensure high-quality, culturally relevant content standards.

Cultural Awareness Data Annotation Quality Assurance Remote Collaboration

Appen Remote AI Data Jobs

Posted 15 days ago

Specialize in high-quality dataset provision for AI models

Support enterprise AI development

Data Annotation Data Collection Deep Learning Transcription

Remote Language Services Roles

Posted 15 days ago

Deliver remote language services

Enhance translation efficiency using technology

Data Annotation Machine Translation Multilingual Communication Process Automation

Internet Search Rater USA

Posted 15 days ago

Improve search engine quality

Analyze and rate online content

Analytical Thinking Basic Computer Skills Data Annotation Quality Assurance

Freelance AI Economics Trainer

Posted 15 days ago

Train generative AI using domain expertise

Evaluate and correct AI model responses

Analytical Skills Critical Thinking Data Annotation Economics

Remote Chemistry AI Tutor

Posted 15 days ago

Connect chemistry experts to AI projects

Improve AI model reasoning in chemistry

Critical Thinking Data Annotation Model Evaluation Remote Collaboration

Korean AI Content Annotator

Posted 15 days ago

Support AI development through data annotation

Evaluate and improve AI-generated content

Communication Skills Data Annotation English Language Proficiency Remote Collaboration

Remote Mathematics AI Tutor

Posted 15 days ago

Support AI model development with expert mathematics input

Evaluate and refine AI-generated mathematical responses

Data Annotation Mathematics Model Evaluation Prompt Engineering

Remote Electrical AI Tutor

Posted 15 days ago

Collaborate remotely on AI projects

Enhance generative AI with domain expertise

Analytical Thinking Data Annotation English Proficiency Generative AI

Earth Science AI Tutor

Posted 15 days ago

Enhance generative AI with earth science expertise

Create and refine AI training prompts

Analytical Skills Data Annotation Generative AI Instructional Design

Automotive AI Tutor Remote

Posted 15 days ago

Shape and improve AI models for automotive engineering.

Create and evaluate challenging prompts for AI training.

AI Model Training Data Annotation Generative AI Prompt Engineering

Freelance AI Data Annotator

Posted 15 days ago

Contribute to AI development through data annotation

Ensure quality and accuracy of AI-generated content

Communication Skills Critical Thinking Data Annotation Remote Collaboration

Mandarin AI Annotation Specialist

Posted 15 days ago

Support AI development through data annotation

Ensure accuracy and quality of AI-generated content

Communication Skills Data Annotation English Language Proficiency

Junior AI Data Annotator

Posted 15 days ago

Support AI job matching system improvement

Perform accurate data labeling and annotation

Data Annotation Quality Assurance

Polish Data Annotation Specialist

Posted 15 days ago

Curate high-quality Polish medical datasets

Support AI and NLP healthcare initiatives

Communication Data Annotation Machine Learning Medical Terminology

Chinese AI QA Annotator

Posted 15 days ago

Improve AI through quality data annotation

Ensure factual accuracy and appropriateness of content

Data Annotation Data Labeling Mandarin Quality Assurance

Korean QA Annotator Role

Posted 15 days ago

Review and label AI training data

Ensure quality and accuracy of annotations

Data Annotation Data Labeling Quality Assurance Remote Collaboration

Japanese QA Annotator Remote

Posted 15 days ago

Ensure quality and accuracy of AI training data

Review and classify content according to guidelines

Data Annotation Data Labeling Japanese Quality Assurance

AI Data Trainer Role

Posted 15 days ago

Develop AI training content

Evaluate and improve AI model responses

Analytical Skills Data Annotation English Language Proficiency Grammar And Syntax

Python AI Content Reviewer

Posted 15 days ago

Evaluate and improve AI-generated content

Assess accuracy and clarity of responses

Algorithms Data Annotation Data Structures Debugging

Italian QA Annotator Remote

Posted 15 days ago

Ensure quality control of AI-generated content

Review and label data based on guidelines

Data Annotation Data Labeling English Language Quality Assurance

Portuguese AI QA Annotator

Posted 15 days ago

Review and ensure AI data quality

Label and classify content for projects

Data Annotation Data Labeling Quality Assurance Remote Collaboration

AI QA Trainer Freelancer

Posted 15 days ago

Validate and improve AI agent evaluation frameworks

Review and refine complex task structures and policy logic

Analytical Thinking Data Annotation QA Quality Assurance

Spanish QA Annotator Role

Posted 15 days ago

Ensure high-quality AI content

Review and label data for AI training

Data Annotation Data Labeling English Language Proficiency Quality Assurance

AI QA Project

Posted 15 days ago

Validate and improve complex task structures, policy logic, and agent evaluation frameworks

Review and analyze evaluation tasks and scenarios for logic and realism

Analytical Thinking Communication Skills Data Annotation Documentation

AI Agent Testing Specialist

Posted 15 days ago

Design realistic evaluation scenarios for AI agents

Create structured test cases and define gold-standard behavior

Computer science Data Analytics Data Annotation Data Science

Python Engineer - AI Project

Posted 15 days ago

Developing and maintaining MCP-compatible evaluation servers

Implementing logic to check agent actions against scenario definitions

Apis Data Annotation Docker FastAPI

AI Research Evaluator - Chinese

Posted 15 days ago

Review and validate AI-generated responses in Chinese for accuracy and reasoning quality

Identify errors, inconsistencies, and areas for improvement in STEM-related datasets

Ai Data Annotation English Language Medical

AI Engineer - Language Models

Posted 15 days ago

Support training and evaluation of AI large language models

Review and critique Industrial Engineering content for accuracy and clarity

Ai Data Annotation Data Evaluation Matlab

AI Agent Testing Specialist

Posted 3 days ago

Design realistic evaluation scenarios for AI agents.

Create test cases simulating human tasks.

AI Ethics Data Analysis Data Annotation JavaScript