The repository includes 72 Chinese scenario candidates: the 24-case pilot plus 48 new AI-authored challenges awaiting human review.
YOUR OWN STUDY
Bring an annotation packet.
Open a JSON packet supplied by your study coordinator. Progress stays in this browser until you export it.
Use assigned pseudonyms. The packet contains anonymous response IDs.
AUTOMATIC SCREENING + HUMAN VALIDATION
Run models. Inspect the ranking.
Generate one response per model and scenario. A judge checks every pair in both orders. Use the automatic result to screen candidates, then collect human judgments.
1. Choose your models
This baseline preset exercises the workflow. Enter exact model IDs for your experiment.
Set the API key in your terminal environment. This page only creates a configuration file; it does not receive keys or launch API calls.
2. Preview, then execute
Save the configuration in the repository folder. In Windows PowerShell, run these commands from that folder using the installed .venv environment:
PREVIEW · NO API CALLSEXECUTE · USES YOUR API ACCOUNT
The request cap includes retries and persists across restarts. It is a request count, not a dollar budget. Add --resume to continue the same run.
3. Read both kinds of evidence
AUTOMATIC REPORT
Import report.json
Open Results → Import report and select studies/model-run/report.json. Inspect the accepted-pair count and excluded comparisons alongside the estimates.
AUDIT TRAIL
Check the decisions
judge/observations.json stores each displayed pair, raw judge output and provider metadata. Reversed checks count as one vote only when all decisions agree.
HUMAN VALIDATION
Let people compare
Distribute the three packets in human-study/public to three independent raters. Their results are evaluated separately. A judge from a candidate's own model family can introduce preference bias.
THE ANNOTATION DESK
Read the situation. Make the call.
0 / 0
judgments saved in this browser
情境 / SITUATION
你的任务 / TASK
事实与限制 / Facts & constraints
AVERSION A
BVERSION B
在这个场景中,你更愿意使用哪一个?
Judge usefulness in this situation. A tie is a valid choice.
Flag a problem passage (optional)
Select text in A or B, then choose a category. These optional flags record noticeable issues (severity 2).
PACKET COMPLETE
Thank you. Every judgment matters.
Please export your judgments and return the file to your coordinator.
Export preview / 导出内容
Download prepared. If a file does not appear, copy all the text below into a UTF-8 file named and return it to your coordinator.
HUMAN STUDY COORDINATION
Know what has come back.
No collection snapshot loaded.
Analysis readiness
A connected comparison graph permits a fit. Completion and connectivity do not establish a winner, statistical separation or participant validity.
Update from returned files
Save participants' JSONL exports in your local returns directory, then run the collection command against the original frozen study. Import the new snapshot here.
This page displays a saved snapshot. Browser saves and newly received files do not update these counts automatically. No files or ratings are uploaded by this page.
Assigned raters
Assignment
Returned / planned
Remaining
Status
Coverage by genre
Genre
Fully rated / planned pairs
Returned / planned judgments
Comparisons still needing ratings
Scenario
Models
Returned / required
Waiting for
Input files and interpretation
File
Rows
Assignments
SHA-256 prefix
CHECK THE MEASUREMENT
How much do the judges agree?
Compare judges on the same response texts. Inspect each decision in both orders. Human annotations provide a separate reference when available.
Load a judge audit produced by the compare-judges command.
Consistency and human agreement
Judge
Preference consistent
Actions consistent
Accepted pairs
Human agreement
Consistency counts use pairs with two valid, non-abstaining checks. Accepted pairs must agree on preference and both actions. Human agreement uses stable preferences and an eligible human majority; its denominator is shown.
Inspect the evidence
EVIDENCE, WITH ITS LIMITS
A report you can interrogate.
System estimates
Direct use is a separate outcome
System
Ability · 95% interval
Direct use
Win / tie / loss
Actions observed
Head-to-head comparisons
Model estimates at neutral display position; ties count as half a preference.
Evidence checks
How much do exclusions matter?
The diagnostic includes one stable preference per pair even if action labels disagree. These are fitted orders, not significance tests. No action scores are added.
Left / right system
Left wins / ties / right wins
Excluded / planned
Left score range over all planned pairs
The range assigns all excluded outcomes first to the right, then to the left, with observed ties worth half a point. It shows how exclusions could affect the score; it is not a confidence interval.
Context matters
Slice summaries are descriptive and do not adjust for different opponent mixes.