- Work from
- Worldwide
- Contract
- Full-time
- Salary
- $75 – $175 / hour
- Posted
- 22 November 2026
- Apply by
- 21 January 2027
Join a network of senior evaluation experts supporting AI research projects. When projects match your experience, you may lead benchmark design, validation, and grading operations to help assess model performance rigorously.
Responsibilities
- Design benchmarks from the ground up, defining scope, question types, difficulty levels, and scoring methods.
- Write and review rubrics to promote consistent grading and strong agreement across evaluators.
- Develop gold-standard answers and test sets that can withstand expert review.
- Lead grading rounds by calibrating graders, measuring agreement, and resolving disputes.
- Analyze model failures, identify root causes, and recommend improvements.
- Design and conduct rigorous model comparison studies.
- Validate benchmarks for contamination, ambiguous answers, and saturation.
- Design user studies on how people rate and interact with model outputs.
Requirements
- At least 5 years of experience in evaluation, testing, or assessment, or 5 years of domain expertise combined with hands-on evaluation work.
- Experience designing a benchmark, test plan, exam, or assessment used by others.
- Experience writing or reviewing rubrics and gold-standard answers, and reviewing or leading graders or question writers.
- Proficiency with Python and evaluation tools such as Inspect, lm-eval-harness, HELM, or promptfoo.
- Experience analyzing model or test failures and recommending fixes.
- Background in machine learning evaluation, QA, test engineering, psychometrics, assessment design, UX research, data science, or subject matter expertise.
- Clear written English communication. Experience validating benchmarks, measuring grader agreement, running model comparisons, or conducting user studies is relevant.
Salary
Compensation is $75 to $175 per hour, depending on project, field, and experience level.
Compensation & Benefits
Payment is based on hours worked.
Location
Fully remote. Projects may last from one week to several months.
B
O
L
I
S
C
Discussion
No comments yet. Ask a question or share a tip about this role.