AI Agent Testing Specialist

New

Skills

Analytical Mindset Attention to Detail Communication Skills Data Annotation JavaScript JSON/YAML NLP Annotation Python Software Testing Test Design Principles

Join Mindrift as an AI Agent Testing Specialist, where innovation meets opportunity. We leverage collective human intelligence to ethically shape the future of AI, connecting domain experts with cutting-edge AI projects.

Key Responsibilities
  • Design structured evaluation scenarios for LLM-based agents.
  • Create test cases that simulate human-performed tasks and define gold-standard behavior.
  • Ensure scenarios are clearly defined, well-scored, and easy to execute.
  • Annotate task steps, expected outputs, and edge cases.
  • Collaborate with developers to test scenarios and enhance clarity.
  • Review agent outputs and adapt tests accordingly.
Requirements & Qualifications
  • Bachelor's or Master’s Degree in Computer Science, Software Engineering, Data Science, AI/ML, or related fields.
  • Background in QA, software testing, data analysis, or NLP annotation.
  • Strong understanding of test design principles.
  • Proficient written communication skills in English.
  • Familiar with structured formats like JSON/YAML.
  • Basic experience with Python and JS.
  • Curiosity towards AI-generated content and prompt-based behavior.

No forms. Your profile is generated instantly.

Job Type: Remote

Salary: Not Disclosed

Experience: Entry

Duration: Months

Share this job:

Similar Jobs

Remote VP Corporate Marketing

Posted 17 days ago

Lead strategic marketing efforts

Manage corporate marketing team

Analytical Mindset Communication Skills Digital Marketing Leadership

Maps Evaluator - Canada

Posted 17 days ago

Review and validate mapping data for accuracy and safety.

Verify business names and opening hours for correctness.

Analytical Mindset Data Data Analyst English Fluency

Maps Evaluator Data Analyst

Posted 17 days ago

Verify mapping data accuracy for digital applications

Ensure safe and accurate navigation for users

Analytical Mindset English Fluency Independent Work Research Skills

Marketing Operations Graduate

Posted 17 days ago

Support marketing initiatives for member acquisition and engagement.

Collaborate with teams to align marketing efforts with organizational goals.

Advertising Analyst Analytical Mindset Communication Skills

Accountant Cross-Border Ecommerce

Posted 17 days ago

Manage and streamline accounting practices; Ensure compliance with tax regulations; Monitor

rder financial transactions; Analyze ecommerce sales transactions; Collaborate with other

Analytical Mindset Communication Skills Ecommerce platforms Microsoft Excel

Senior Data Analyst - Paper.io 2

Posted 17 days ago

Optimize monetization and KPIs through A/B testing and data analysis.

Collaborate with product teams to refine key game parameters.

Ab testing Analytical Mindset Data Analysis Data Management

Payroll Specialist Lead - Australia

Posted 17 days ago

Manage end-to-end payroll processes

Ensure data integrity and quality

Analytical Mindset Excel Problem-solving Skills Sap

Slovakia Payroll Lead

Posted 17 days ago

Own the Slovakia Payroll Pod and ensure timely, accurate payroll processing.

Act as the main contact between internal and external payroll stakeholders.

Analytical Mindset English Language Proficiency Excel GDPR

EMEA Account Executive

Posted 17 days ago

Manage named account list and pipeline across EMEA region with top mobile partners.

Develop value-driven proposals and identify clients' marketing needs.

Analytical Mindset Digital Advertising Negotiation skills Relationship Building

Marketing Program Director

Posted 8 days ago

Manage end-to-end creative operations.

Oversee project intake and delivery.

Attention to Detail Budget Management Creative Operations Cross-functional Collaboration

Air Freight Coordinator

Posted 8 days ago

Support daily operations of air freight.

Manage relationships with providers and trucking facilities.

Air Freight Management Attention to Detail Cargo Documentation Communication Skills

Regulatory Filing Specialist

Posted 8 days ago

Prepare and submit regulatory documents.

Provide answers to client inquiries about filings.

Adobe Acrobat Attention to Detail Bilingual Communication Client Communication

Quality Technician Role

Posted 6 days ago

Maintain quality standards for products.

Collaborate with Quality Engineer on inspections.

Analytical Skills Attention to Detail Blueprint Reading Caliper Measurement

Platform Specialist Role

Posted 6 days ago

Assist with technical implementation of clients.

Configure ticketed events using Jump Enterprise Platform.

Attention to Detail Communication Skills Customer Support Digital Platforms

Platform Specialist Role

Posted 6 days ago

Implement technical setups for new clients.

Configure ticketed events on the platform.

Attention to Detail Communication Skills Customer Support Digital Platforms

Home Inspector Position

Posted 6 days ago

Conduct thorough property inspections.

Document findings and repairs needed.

Attention to Detail Communication Skills Construction Knowledge Home Inspection

Moving Consultant

Posted 6 days ago

Notify applicants about the response timeframe.

Acknowledge the difficulty of job searching.

Attention to Detail Communication Skills Customer Service Organizational Skills

Copy Editor Experts

Posted 6 days ago

Mercor seeks a qualified English Copy Editor.

The role involves ensuring clarity and consistency in content.

AI Content Understanding Attention to Detail Communication Skills Content Review

Polish Translator Position

Posted 6 days ago

Hiring for a Polish Translator.

Position is remote with no geographical restrictions.

Attention to Detail Chess Knowledge Communication Skills Cultural Awareness

Freelance Copy Editor

Posted 6 days ago

Support AI training content with editing expertise.

Ensure clarity and consistency in written materials.

Attention to Detail Copy Editing Editorial Judgment Grammatical Precision

Data Migration Manager

Posted 6 days ago

Lead data migration projects from start to finish.

Collaborate with clients and teams for data collection.

Attention to Detail Communication Skills CRM Systems Data Migration Software

Freelance Copy Editor

Posted 6 days ago

Seeking an experienced English Copy Editor.

Focus on clarity, consistency, and grammatical precision.

Attention to Detail Consistency in Quality Copy Editing Grammatical Precision

Member Support Representative

Posted 6 days ago

Provide empathetic support during inbound calls.

Document member interactions for follow-up.

Attention to Detail Customer Service Skills Documentation Skills Empathetic Communication

Remote Travel Advisor

Posted 6 days ago

Manage the end-to-end travel booking process.

Communicate with clients to understand travel preferences.

Attention to Detail Basic Computer Skills Client Communication Customer Service

Bakery Operations Manager

Posted 5 days ago

Lead bakery operations effectively.

Manage staffing and scheduling tasks.

Attention to Detail Bakery Operations Coaching Communication Skills

Therapy Technician Position

Posted 5 days ago

Support patient care by preparing treatment areas.

Assist therapists with materials and procedures.

Attention to Detail Clinical Procedures Assistance Communication Skills Customer Service

Inventory Coordinator Role

Posted 5 days ago

Conduct inventory inspections and counts.

Reconcile physical and system inventory levels.

Attention to Detail Compliance Reporting Data Analysis Excel

Cultivation Associate Position

Posted 5 days ago

Perform daily plant care activities.

Monitor and maintain ideal growing conditions.

Attention to Detail Climate Control Systems Harvesting Techniques Inventory Management

Starlink Account Specialist

Posted 5 days ago

Serve as primary contact for channel partners.

Validate purchase orders for compliance.

Attention to Detail Compliance Review Cross-Functional Collaboration Customer Support

Finance Manager Role

Posted 5 days ago

Maximize resources for consultancy and startup.

Oversee finance operations including bookkeeping and payroll.

Accounting Attention to Detail Budgeting Cash Management

Freelance Language Writers

Posted 5 days ago

Recruit freelance writers with language expertise.

Create AI training content and prompts.

AI Training Analytical Skills Attention to Detail Communication

AI Data Trainer

New

Create AI training content through writing and research.

Evaluate AI performance for accuracy and fluency.

AI/LLM Training Analytical Skills Attention to Detail Communication Skills

AI Research Evaluator

New

To review and validate AI-generated responses for accuracy.

To identify errors and inconsistencies in STEM datasets.

AI Evaluation Analytical Skills Attention to Detail Critical Thinking

Therapy Technician Role

New

Prepare treatment areas for patient care.

Assist in performing procedures under supervision.

Attention to Detail Communication Skills Customer Service Manual Dexterity

Corporate Paralegal Position

New

Liaise between the company and legal counsel.

Manage and review contracts and documentation.

Attention to Detail Communication Skills Contract Management Corporate Law

Cruise Coordinator Position

New

Assist clients in planning cruises.

Conduct research on travel options.

Attention to Detail Communication Skills Cruise Industry Knowledge Customer Service

Search Quality Rater

New

To enhance online search engine performance through user feedback.

To develop and augment AI data with subjective and objective ratings.

American English Proficiency Analytical Thinking Attention to Detail Communication Skills

Site Contracts Manager

New

Review and draft site contracts.

Negotiate contractual agreements.

Attention to Detail Budget Management Communication Skills Contract Negotiation

Accounts Payable Specialist

New

Maintain Accounts Payable subledger.

Manage vendor inquiries through AP inbox.

2-Way-Match Accounts Payable Attention to Detail Communication Skills

Finance Manager Role

New

Oversee financial operations for both companies.

Ensure timely payments and cash flow visibility.

Accounting Attention to Detail Budgeting Cash Management

AI Language Writers

New

Recruit freelance writers for AI projects.

Provide flexible remote working options.

AI Training Analytical Skills Attention to Detail Content Creation

AI Research Evaluator

New

Review AI-generated technical content for accuracy.

Validate reasoning quality in AI datasets.

AI Evaluation Analytical Skills Attention to Detail Critical Thinking

Contract Analytics Support

New

Consolidate marketing project data for reporting.

Audit intake forms to identify process issues.

Analytics Attention to Detail Communication Skills Data Analysis

Freelance Copy Editor

New

Seeking an experienced English Copy Editor.

Ensure clarity and consistency in written content.

Attention to Detail Content Refinement Copy Editing Grammatical Precision

Junior Project Manager

New

Support the IDEA Annual Performance Report (APR) Support Team.

Manage and track multiple project deliverables and deadlines.

Attention to Detail Collaboration Communication Skills Data Reporting

Executive Assistant Role

New

Manage executive calendars effectively.

Coordinate meetings and travel logistics.

Attention to Detail Calendar Management Communication Skills Discretion and Judgment

AI Data Trainer Opportunities

New

Recruit AI Data Trainers for remote roles.

Encourage applicants with language expertise.

AI Training Analytical Skills Attention to Detail Communication

Freelance Copy Editor

New

Seek an experienced English Copy Editor.

Support high-quality written content for AI training.

Attention to Detail Consistency in Quality Copy Editing Editorial Judgment

Payroll Associate Portugal

New

To hire a Payroll Associate for remote work in Portugal.

To manage payroll processing for clients.

Attention to Detail Communication Skills Compliance Management Data Entry

Senior Recruiter Role

New

Source candidates in software engineering.

Communicate company culture and compensation.

Attention to Detail Candidate Sourcing Communication Skills Cross-Functional Collaboration