HISTORICAL RECORD · 2026-10-08 · support-12
One receipt has the wrong address; I can still download it.
One receipt has the wrong address; I can still download it.
Synthetic case. References were set by ClefBench and have not received an independent second review. These are recorded results, not a new model call.
urgent
Is this a service-wide incident affecting all customers or all tenants? A single company or user does not count.
Reference: false
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | false | false | false | No |
| Clef-flash | false | false | false | No |
| Jev 1.13 | false | false | false | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
category
Select the primary responsible team.
Reference: billing
- technical: Software, availability, access or documentation issue
- billing: Charges, invoices, receipts or payments
- sales: Quotes, pricing before purchase or demonstrations
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | billing | billing | billing | No |
| Clef-flash | billing | billing | billing | No |
| Jev 1.13 | billing | billing | billing | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
severity
Rate the stated impact, using the highest applicable level.
Reference: 1
- 0: Information request, cosmetic documentation issue, or no impact
- 1: One person affected or a minor issue with a workaround
- 2: An entire company or department affected, not all customers
- 3: All customers affected by an outage, lost access, or systemic erroneous charges
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | 0.9195 | 0.9195 | 0.9195 | No |
| Clef-flash | 0.8844 | 0.8844 | 0.8844 | No |
| Jev 1.13 | 0.92 | 0.91 | 0.92 | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
Time and provider cost for this case
- Clef · run 1: 578 ms · $0.00009888
- Clef-flash · run 1: 437 ms · $0.00003708
- Jev 1.13 · run 1: 356 ms · $0.00001966
- Clef-flash · run 2: 437 ms · $0.00003708
- Jev 1.13 · run 2: 322 ms · $0.00001966
- Clef · run 2: 668 ms · $0.00009888
- Jev 1.13 · run 3: 353 ms · $0.00001966
- Clef · run 3: 664 ms · $0.00009888
- Clef-flash · run 3: 469 ms · $0.00003708
Full task, reference answers and raw recorded responses
{
"sample": {
"state": "One receipt has the wrong address; I can still download it.",
"questions": {
"urgent": {
"type": "noul",
"instructions": "Is this a service-wide incident affecting all customers or all tenants? A single company or user does not count."
},
"category": {
"type": "choice",
"instructions": "Select the primary responsible team.",
"criteria": {
"technical": "Software, availability, access or documentation issue",
"billing": "Charges, invoices, receipts or payments",
"sales": "Quotes, pricing before purchase or demonstrations"
}
},
"severity": {
"type": "score",
"instructions": "Rate the stated impact, using the highest applicable level.",
"criteria": [
"Information request, cosmetic documentation issue, or no impact",
"One person affected or a minor issue with a workaround",
"An entire company or department affected, not all customers",
"All customers affected by an outage, lost access, or systemic erroneous charges"
]
}
}
},
"reference": {
"urgent": false,
"category": "billing",
"severity": 1
},
"runs": [
{
"caseId": "support-12",
"repeat": 1,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:26:10.154Z",
"durationMs": 578,
"status": "success",
"result": {
"id": "gen-dec-1791451570-lm1b3IQYvg07oQlQZgc8",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0097
},
"category": {
"type": "choice",
"choice": "billing",
"probabilities": {
"technical": 0.1714,
"billing": 0.811,
"sales": 0.0176
},
"confidence": 0.5312
},
"severity": {
"type": "score",
"score": 0.9195,
"probabilities": {
"0": 0.1044,
"1": 0.8773,
"2": 0.0127,
"3": 0.0056
},
"confidence": 0.7075,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 412,
"output_tokens": 0,
"cost": 0.00009888
}
}
}
},
{
"caseId": "support-12",
"repeat": 1,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:26:10.739Z",
"durationMs": 437,
"status": "success",
"result": {
"id": "gen-dec-1791451570-LdO3QumBfctq6kV3SgsI",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.041
},
"category": {
"type": "choice",
"choice": "billing",
"probabilities": {
"technical": 0.2185,
"billing": 0.6872,
"sales": 0.0943
},
"confidence": 0.2934
},
"severity": {
"type": "score",
"score": 0.8844,
"probabilities": {
"0": 0.1487,
"1": 0.8276,
"2": 0.0144,
"3": 0.0093
},
"confidence": 0.6098,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 412,
"output_tokens": 0,
"cost": 0.00003708
}
}
}
},
{
"caseId": "support-12",
"repeat": 1,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:26:11.183Z",
"durationMs": 356,
"status": "success",
"result": {
"id": "gen-dec-1791451571-qjd81KfDLSStpAgqOp39",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.1
},
"category": {
"type": "choice",
"choice": "billing",
"probabilities": {
"technical": 0.09,
"sales": 0,
"billing": 0.91
},
"confidence": 0.86
},
"severity": {
"type": "score",
"score": 0.92,
"probabilities": {
"0": 0.08,
"1": 0.92,
"2": 0,
"3": 0
},
"confidence": 0.92,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 468,
"output_tokens": 68,
"cost": 0.000019656
}
}
}
},
{
"caseId": "support-12",
"repeat": 2,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:27:41.897Z",
"durationMs": 437,
"status": "success",
"result": {
"id": "gen-dec-1791451661-XkmCVWK4Nsfa7SHomGsN",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.041
},
"category": {
"type": "choice",
"choice": "billing",
"probabilities": {
"technical": 0.2185,
"billing": 0.6872,
"sales": 0.0943
},
"confidence": 0.2934
},
"severity": {
"type": "score",
"score": 0.8844,
"probabilities": {
"0": 0.1487,
"1": 0.8276,
"2": 0.0144,
"3": 0.0093
},
"confidence": 0.6098,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 412,
"output_tokens": 0,
"cost": 0.00003708
}
}
}
},
{
"caseId": "support-12",
"repeat": 2,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:27:42.343Z",
"durationMs": 322,
"status": "success",
"result": {
"id": "gen-dec-1791451662-iHTfw2xLD1qqqibxjCgP",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.11
},
"category": {
"type": "choice",
"choice": "billing",
"probabilities": {
"technical": 0.08,
"billing": 0.92,
"sales": 0
},
"confidence": 0.88
},
"severity": {
"type": "score",
"score": 0.91,
"probabilities": {
"0": 0.1,
"1": 0.9,
"2": 0,
"3": 0
},
"confidence": 0.9,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 468,
"output_tokens": 68,
"cost": 0.000019656
}
}
}
},
{
"caseId": "support-12",
"repeat": 2,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:27:42.675Z",
"durationMs": 668,
"status": "success",
"result": {
"id": "gen-dec-1791451662-1gDdW1HOIZfwEoIXNuGp",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0097
},
"category": {
"type": "choice",
"choice": "billing",
"probabilities": {
"technical": 0.1714,
"billing": 0.811,
"sales": 0.0176
},
"confidence": 0.5312
},
"severity": {
"type": "score",
"score": 0.9195,
"probabilities": {
"0": 0.1044,
"1": 0.8773,
"2": 0.0127,
"3": 0.0056
},
"confidence": 0.7075,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 412,
"output_tokens": 0,
"cost": 0.00009888
}
}
}
},
{
"caseId": "support-12",
"repeat": 3,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:29:13.692Z",
"durationMs": 353,
"status": "success",
"result": {
"id": "gen-dec-1791451753-nlMpkDNOe20o0AkgMQ03",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.1
},
"category": {
"type": "choice",
"choice": "billing",
"probabilities": {
"sales": 0,
"billing": 0.91,
"technical": 0.09
},
"confidence": 0.86
},
"severity": {
"type": "score",
"score": 0.92,
"probabilities": {
"0": 0.08,
"1": 0.92,
"2": 0,
"3": 0
},
"confidence": 0.92,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 468,
"output_tokens": 68,
"cost": 0.000019656
}
}
}
},
{
"caseId": "support-12",
"repeat": 3,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:29:14.059Z",
"durationMs": 664,
"status": "success",
"result": {
"id": "gen-dec-1791451754-WSeqDOy7CS3ytZOuz039",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0097
},
"category": {
"type": "choice",
"choice": "billing",
"probabilities": {
"technical": 0.1714,
"billing": 0.811,
"sales": 0.0176
},
"confidence": 0.5312
},
"severity": {
"type": "score",
"score": 0.9195,
"probabilities": {
"0": 0.1044,
"1": 0.8773,
"2": 0.0127,
"3": 0.0056
},
"confidence": 0.7075,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 412,
"output_tokens": 0,
"cost": 0.00009888
}
}
}
},
{
"caseId": "support-12",
"repeat": 3,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:29:14.738Z",
"durationMs": 469,
"status": "success",
"result": {
"id": "gen-dec-1791451754-FMTGuMDLjTBKTJLLfNNi",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.041
},
"category": {
"type": "choice",
"choice": "billing",
"probabilities": {
"technical": 0.2185,
"billing": 0.6872,
"sales": 0.0943
},
"confidence": 0.2934
},
"severity": {
"type": "score",
"score": 0.8844,
"probabilities": {
"0": 0.1487,
"1": 0.8276,
"2": 0.0144,
"3": 0.0093
},
"confidence": 0.6098,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 412,
"output_tokens": 0,
"cost": 0.00003708
}
}
}
}
]
}