caferemote
Clera

Expert Talent Pool, Evals and Benchmarks

Remote Worldwide Full-time $75 – $175 / hour Posted 3h ago
Apply now
Work from
Worldwide
Contract
Full-time
Salary
$75 – $175 / hour
Posted
22 November 2026
Apply by
21 January 2027

Join a network of senior evaluation experts supporting AI research projects. When projects match your experience, you may lead benchmark design, validation, and grading operations to help assess model performance rigorously.

Responsibilities

  • Design benchmarks from the ground up, defining scope, question types, difficulty levels, and scoring methods.
  • Write and review rubrics to promote consistent grading and strong agreement across evaluators.
  • Develop gold-standard answers and test sets that can withstand expert review.
  • Lead grading rounds by calibrating graders, measuring agreement, and resolving disputes.
  • Analyze model failures, identify root causes, and recommend improvements.
  • Design and conduct rigorous model comparison studies.
  • Validate benchmarks for contamination, ambiguous answers, and saturation.
  • Design user studies on how people rate and interact with model outputs.

Requirements

  • At least 5 years of experience in evaluation, testing, or assessment, or 5 years of domain expertise combined with hands-on evaluation work.
  • Experience designing a benchmark, test plan, exam, or assessment used by others.
  • Experience writing or reviewing rubrics and gold-standard answers, and reviewing or leading graders or question writers.
  • Proficiency with Python and evaluation tools such as Inspect, lm-eval-harness, HELM, or promptfoo.
  • Experience analyzing model or test failures and recommending fixes.
  • Background in machine learning evaluation, QA, test engineering, psychometrics, assessment design, UX research, data science, or subject matter expertise.
  • Clear written English communication. Experience validating benchmarks, measuring grader agreement, running model comparisons, or conducting user studies is relevant.

Salary

Compensation is $75 to $175 per hour, depending on project, field, and experience level.

Compensation & Benefits

Payment is based on hours worked.

Location

Fully remote. Projects may last from one week to several months.

Discussion

No comments yet. Ask a question or share a tip about this role.

Apply now

More jobs from Clera

View company →

Product Engineer (Consumer, Design Heavy)

Clera
Remote United States Full-time $200K – $350K / year

About the Role Own the full lifecycle of consumer products, from spotting user needs and shaping product direction to building and shipping polished…

3h ago Software Development

Freelance Engineer

Clera
Remote Worldwide Full-time

About the Role Work independently on software projects for a logistics technology company developing AI tools for freight operations. This role helps deliver…

3h ago Software Development

Expert Talent Pool, Agents and Reinforcement Learning (Associate, 0-5 Years)

Clera
Remote Worldwide Full-time $35 – $70 / hour

About the Role Join a paid talent pool supporting AI projects focused on agents and reinforcement learning. You will help build training environments and…

3h ago Data

Senior Member of Technical Staff

Clera
Remote United States Full-time $150K – $215K / year

About the Role Join a founding engineering team building AI-powered tools for law enforcement and the broader criminal justice system. In this high-autonomy…

4h ago Software Development

Similar remote jobs

Browse all →

Oracle PL/SQL Developer

Bright Vision Technologies
Remote United States Full-time $120K – $180K / year

Oracle PL/SQL Developer – Remote Bright Vision Technologies is a technology consulting and software development company delivering cloud, AI, data, and enterprise solutions across the…

3h ago Data

Senior Data Scientist (R14369)

Oportun
Remote India Full-time

Who We Are Oportun (Nasdaq: OPRT) is a mission-driven financial services company that puts its members' financial goals within reach. With intelligent borrowing, savings, and budgeting…

3h ago Data

AI for Data & Analytics Coach

Leland
Remote United States Contract

Leland is building the home for ambition in the age of AI. We connect people with the experts, programs, and communities they need to land the job, get into the school, or upskill for a…

3h ago Data

Azure Data Engineer ( EU REMOTE)

IRIUM Portugal
Remote Albania, Andorra, Austria, Belarus, Belgium, Bosnia and Herzegovina, Bulgaria, Croatia, Cyprus, Czechia, Denmark, Estonia, Faroe Islands, Finland, France, Germany, Gibraltar, Greece, Guernsey, Holy See (Vatican City State), Hungary, Iceland, Ireland, Full-time €2K – €3K / month

At IRIUM, we want you to always pursue your dreams! Here, get ready to achieve your goals while enjoying the journey. We are currently looking for an Azure Data Engineer to join our team!…

3h ago Data

Applied Scientist (LLM)

Squad
Remote Ukraine Contract

Team Summary Our distributed team is looking for an experienced Applied Scientist with a strong background in Large Language models to develop high-performance Generative AI features across…

3h ago Data

Engineering Manager, Data Feeds

Chainlink Labs
Remote Canada, United States Full-time $129K – $304K / year

About Chainlink Labs Chainlink Labs is one of the primary contributing developers of Chainlink, the standard for onchain finance enabling high-quality data, secure cross-chain…

3h ago Data

Get remote jobs the moment they are posted

Choose your categories and skills. We push matching jobs to your device. No email, ever.