HISTORICAL RECORD · 2026-10-08 · support-15
Procurement needs an enterprise quote by tomorrow, but operations are normal.
Procurement needs an enterprise quote by tomorrow, but operations are normal.
Synthetic case. References were set by ClefBench and have not received an independent second review. These are recorded results, not a new model call.
urgent
Is this a service-wide incident affecting all customers or all tenants? A single company or user does not count.
Reference: false
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | false | false | false | No |
| Clef-flash | false | false | false | No |
| Jev 1.13 | false | false | false | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
category
Select the primary responsible team.
Reference: sales
- technical: Software, availability, access or documentation issue
- billing: Charges, invoices, receipts or payments
- sales: Quotes, pricing before purchase or demonstrations
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | sales | sales | sales | No |
| Clef-flash | sales | sales | sales | No |
| Jev 1.13 | sales | sales | sales | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
severity
Rate the stated impact, using the highest applicable level.
Reference: 0
- 0: Information request, cosmetic documentation issue, or no impact
- 1: One person affected or a minor issue with a workaround
- 2: An entire company or department affected, not all customers
- 3: All customers affected by an outage, lost access, or systemic erroneous charges
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | 1.2162Differs from reference | 1.2162Differs from reference | 1.2162Differs from reference | No |
| Clef-flash | 0.9487Differs from reference | 0.9487Differs from reference | 0.9487Differs from reference | No |
| Jev 1.13 | 0.54Differs from reference | 0.51Differs from reference | 0.57Differs from reference | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
Time and provider cost for this case
- Clef · run 1: 517 ms · $0.00009912
- Clef-flash · run 1: 415 ms · $0.00003717
- Jev 1.13 · run 1: 353 ms · $0.00001974
- Clef-flash · run 2: 451 ms · $0.00003717
- Jev 1.13 · run 2: 340 ms · $0.00001974
- Clef · run 2: 577 ms · $0.00009912
- Jev 1.13 · run 3: 421 ms · $0.00001974
- Clef · run 3: 547 ms · $0.00009912
- Clef-flash · run 3: 425 ms · $0.00003717
Full task, reference answers and raw recorded responses
{
"sample": {
"state": "Procurement needs an enterprise quote by tomorrow, but operations are normal.",
"questions": {
"urgent": {
"type": "noul",
"instructions": "Is this a service-wide incident affecting all customers or all tenants? A single company or user does not count."
},
"category": {
"type": "choice",
"instructions": "Select the primary responsible team.",
"criteria": {
"technical": "Software, availability, access or documentation issue",
"billing": "Charges, invoices, receipts or payments",
"sales": "Quotes, pricing before purchase or demonstrations"
}
},
"severity": {
"type": "score",
"instructions": "Rate the stated impact, using the highest applicable level.",
"criteria": [
"Information request, cosmetic documentation issue, or no impact",
"One person affected or a minor issue with a workaround",
"An entire company or department affected, not all customers",
"All customers affected by an outage, lost access, or systemic erroneous charges"
]
}
}
},
"reference": {
"urgent": false,
"category": "sales",
"severity": 0
},
"runs": [
{
"caseId": "support-15",
"repeat": 1,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:26:14.729Z",
"durationMs": 517,
"status": "success",
"result": {
"id": "gen-dec-1791451574-oNJk5j55xumnKyfENXAn",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0143
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"technical": 0.039,
"billing": 0.0111,
"sales": 0.9499
},
"confidence": 0.8558
},
"severity": {
"type": "score",
"score": 1.2162,
"probabilities": {
"0": 0.1785,
"1": 0.4341,
"2": 0.3801,
"3": 0.0073
},
"confidence": 0.1532,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 413,
"output_tokens": 0,
"cost": 0.00009912
}
}
}
},
{
"caseId": "support-15",
"repeat": 1,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:26:15.255Z",
"durationMs": 415,
"status": "success",
"result": {
"id": "gen-dec-1791451575-40bpKWhd3WBLdXpDHn7l",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0279
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"technical": 0.0216,
"billing": 0.0562,
"sales": 0.9222
},
"confidence": 0.7812
},
"severity": {
"type": "score",
"score": 0.9487,
"probabilities": {
"0": 0.3076,
"1": 0.4535,
"2": 0.2214,
"3": 0.0175
},
"confidence": 0.1328,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 413,
"output_tokens": 0,
"cost": 0.00003717
}
}
}
},
{
"caseId": "support-15",
"repeat": 1,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:26:15.679Z",
"durationMs": 353,
"status": "success",
"result": {
"id": "gen-dec-1791451575-EZZfsoi3psYDj0SuEK0z",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.06
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"technical": 0,
"sales": 1,
"billing": 0
},
"confidence": 1
},
"severity": {
"type": "score",
"score": 0.54,
"probabilities": {
"0": 0.68,
"1": 0.11,
"2": 0.21,
"3": 0
},
"confidence": 0.47,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 470,
"output_tokens": 68,
"cost": 0.00001974
}
}
}
},
{
"caseId": "support-15",
"repeat": 2,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:27:46.353Z",
"durationMs": 451,
"status": "success",
"result": {
"id": "gen-dec-1791451666-H9pXo9jkk93oE9WSeJib",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0279
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"technical": 0.0216,
"billing": 0.0562,
"sales": 0.9222
},
"confidence": 0.7812
},
"severity": {
"type": "score",
"score": 0.9487,
"probabilities": {
"0": 0.3076,
"1": 0.4535,
"2": 0.2214,
"3": 0.0175
},
"confidence": 0.1328,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 413,
"output_tokens": 0,
"cost": 0.00003717
}
}
}
},
{
"caseId": "support-15",
"repeat": 2,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:27:46.813Z",
"durationMs": 340,
"status": "success",
"result": {
"id": "gen-dec-1791451666-rHWuK3MhyLFueal0wSAY",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.07
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"technical": 0,
"billing": 0,
"sales": 1
},
"confidence": 1
},
"severity": {
"type": "score",
"score": 0.51,
"probabilities": {
"0": 0.69,
"1": 0.11,
"2": 0.2,
"3": 0
},
"confidence": 0.49,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 470,
"output_tokens": 68,
"cost": 0.00001974
}
}
}
},
{
"caseId": "support-15",
"repeat": 2,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:27:47.162Z",
"durationMs": 577,
"status": "success",
"result": {
"id": "gen-dec-1791451667-NYXMWKT9ZqUmAsESnSuU",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0143
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"technical": 0.039,
"billing": 0.0111,
"sales": 0.9499
},
"confidence": 0.8558
},
"severity": {
"type": "score",
"score": 1.2162,
"probabilities": {
"0": 0.1785,
"1": 0.4341,
"2": 0.3801,
"3": 0.0073
},
"confidence": 0.1532,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 413,
"output_tokens": 0,
"cost": 0.00009912
}
}
}
},
{
"caseId": "support-15",
"repeat": 3,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:29:18.300Z",
"durationMs": 421,
"status": "success",
"result": {
"id": "gen-dec-1791451758-RLRSlrnFNKavNyLs7Wta",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.06
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"sales": 1,
"billing": 0,
"technical": 0
},
"confidence": 1
},
"severity": {
"type": "score",
"score": 0.57,
"probabilities": {
"0": 0.66,
"1": 0.12,
"2": 0.22,
"3": 0
},
"confidence": 0.44,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 470,
"output_tokens": 68,
"cost": 0.00001974
}
}
}
},
{
"caseId": "support-15",
"repeat": 3,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:29:18.735Z",
"durationMs": 547,
"status": "success",
"result": {
"id": "gen-dec-1791451758-lfxzvCrL8shpco29lZRd",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0143
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"technical": 0.039,
"billing": 0.0111,
"sales": 0.9499
},
"confidence": 0.8558
},
"severity": {
"type": "score",
"score": 1.2162,
"probabilities": {
"0": 0.1785,
"1": 0.4341,
"2": 0.3801,
"3": 0.0073
},
"confidence": 0.1532,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 413,
"output_tokens": 0,
"cost": 0.00009912
}
}
}
},
{
"caseId": "support-15",
"repeat": 3,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:29:19.297Z",
"durationMs": 425,
"status": "success",
"result": {
"id": "gen-dec-1791451759-ijMgjhrsKcH10X13iTGY",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0279
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"technical": 0.0216,
"billing": 0.0562,
"sales": 0.9222
},
"confidence": 0.7812
},
"severity": {
"type": "score",
"score": 0.9487,
"probabilities": {
"0": 0.3076,
"1": 0.4535,
"2": 0.2214,
"3": 0.0175
},
"confidence": 0.1328,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 413,
"output_tokens": 0,
"cost": 0.00003717
}
}
}
}
]
}