- Work from
- Worldwide
- Contract
- Full-time
- Salary
- $75 – $175 / hour
- Posted
- 16 November 2026
- Apply by
- 15 January 2027
You will lead benchmark design, validation, and grading operations when projects align with your expertise, helping produce rigorous and reliable assessments of AI models.
About the role
Join a network of senior evaluation experts supporting AI research projects.
Responsibilities
- Design benchmarks, including their scope, question types, difficulty levels, and scoring methods.
- Write and review rubrics to support consistent grading.
- Develop gold-standard answers and test sets that hold up to expert review.
- Lead grading rounds by calibrating graders, measuring agreement, and resolving disputes.
- Analyze model failures, identify root causes, and recommend improvements.
- Design and conduct rigorous model comparison studies.
- Validate benchmarks for contamination, ambiguity, and saturation.
- Design user studies on how people rate and interact with model outputs.
Requirements
- At least 5 years of experience in evaluation, testing, or assessment, or at least 5 years of domain expertise combined with hands-on evaluation work.
- Experience designing a benchmark, test plan, exam, or assessment used by others.
- Experience writing or reviewing rubrics and gold-standard answers, and reviewing or leading graders or question writers.
- Proficiency in Python and evaluation tools such as Inspect, lm-eval-harness, HELM, or promptfoo.
- Experience analyzing model or test failures and recommending fixes.
- Background in ML evaluation, QA, test engineering, psychometrics, assessment design, UX research, data science, or deep subject matter expertise.
- Clear written English communication.
- Experience validating benchmarks, measuring grader agreement, conducting model comparisons, or running user studies is relevant.
Salary
Compensation is $75 to $175 per hour, depending on project, field, and experience level. Compensation Range: $156K - $364K
Compensation & Benefits
Pay is based on hours worked.
Location
Fully remote. Projects may last from one week to several months.
N
C
L
W
D
T
Discussion
No comments yet. Ask a question or share a tip about this role.