HISTORICAL RECORD · 2026-10-08 · support-07
Can we arrange a demonstration next week?
Can we arrange a demonstration next week?
Synthetic case. References were set by ClefBench and have not received an independent second review. These are recorded results, not a new model call.
urgent
Is this a service-wide incident affecting all customers or all tenants? A single company or user does not count.
Reference: false
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | false | false | false | No |
| Clef-flash | trueDiffers from reference | trueDiffers from reference | trueDiffers from reference | No |
| Jev 1.13 | false | false | false | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
category
Select the primary responsible team.
Reference: sales
- technical: Software, availability, access or documentation issue
- billing: Charges, invoices, receipts or payments
- sales: Quotes, pricing before purchase or demonstrations
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | sales | sales | sales | No |
| Clef-flash | sales | sales | sales | No |
| Jev 1.13 | sales | sales | sales | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
severity
Rate the stated impact, using the highest applicable level.
Reference: 0
- 0: Information request, cosmetic documentation issue, or no impact
- 1: One person affected or a minor issue with a workaround
- 2: An entire company or department affected, not all customers
- 3: All customers affected by an outage, lost access, or systemic erroneous charges
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | 0.1706 | 0.1706 | 0.1706 | No |
| Clef-flash | 0.1514 | 0.1514 | 0.1514 | No |
| Jev 1.13 | 0.02 | 0.02 | 0.02 | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
Time and provider cost for this case
- Clef · run 1: 527 ms · $0.00009768
- Clef-flash · run 1: 452 ms · $0.00003663
- Jev 1.13 · run 1: 370 ms · $0.00001945
- Clef-flash · run 2: 439 ms · $0.00003663
- Jev 1.13 · run 2: 352 ms · $0.00001945
- Clef · run 2: 676 ms · $0.00009768
- Jev 1.13 · run 3: 332 ms · $0.00001945
- Clef · run 3: 620 ms · $0.00009768
- Clef-flash · run 3: 480 ms · $0.00003663
Full task, reference answers and raw recorded responses
{
"sample": {
"state": "Can we arrange a demonstration next week?",
"questions": {
"urgent": {
"type": "noul",
"instructions": "Is this a service-wide incident affecting all customers or all tenants? A single company or user does not count."
},
"category": {
"type": "choice",
"instructions": "Select the primary responsible team.",
"criteria": {
"technical": "Software, availability, access or documentation issue",
"billing": "Charges, invoices, receipts or payments",
"sales": "Quotes, pricing before purchase or demonstrations"
}
},
"severity": {
"type": "score",
"instructions": "Rate the stated impact, using the highest applicable level.",
"criteria": [
"Information request, cosmetic documentation issue, or no impact",
"One person affected or a minor issue with a workaround",
"An entire company or department affected, not all customers",
"All customers affected by an outage, lost access, or systemic erroneous charges"
]
}
}
},
"reference": {
"urgent": false,
"category": "sales",
"severity": 0
},
"runs": [
{
"caseId": "support-07",
"repeat": 1,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:26:02.520Z",
"durationMs": 527,
"status": "success",
"result": {
"id": "gen-dec-1791451562-C5Ixr4Qi7lG1spYPgG4s",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0068
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"technical": 0.0232,
"billing": 0.005,
"sales": 0.9718
},
"confidence": 0.9173
},
"severity": {
"type": "score",
"score": 0.1706,
"probabilities": {
"0": 0.8547,
"1": 0.1258,
"2": 0.0136,
"3": 0.0059
},
"confidence": 0.6622,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 407,
"output_tokens": 0,
"cost": 0.00009768
}
}
}
},
{
"caseId": "support-07",
"repeat": 1,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:26:03.049Z",
"durationMs": 452,
"status": "success",
"result": {
"id": "gen-dec-1791451563-WPm5gK9kVERAEbbEYFN2",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.5263
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"technical": 0.0427,
"billing": 0.0105,
"sales": 0.9468
},
"confidence": 0.8475
},
"severity": {
"type": "score",
"score": 0.1514,
"probabilities": {
"0": 0.8827,
"1": 0.0902,
"2": 0.02,
"3": 0.0071
},
"confidence": 0.717,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 407,
"output_tokens": 0,
"cost": 0.00003663
}
}
}
},
{
"caseId": "support-07",
"repeat": 1,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:26:03.509Z",
"durationMs": 370,
"status": "success",
"result": {
"id": "gen-dec-1791451563-2kjB4tpqHCWeMYuCut1S",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.06
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"technical": 0,
"billing": 0,
"sales": 1
},
"confidence": 1
},
"severity": {
"type": "score",
"score": 0.02,
"probabilities": {
"0": 0.99,
"1": 0.01,
"2": 0,
"3": 0
},
"confidence": 0.99,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 463,
"output_tokens": 68,
"cost": 0.000019446
}
}
}
},
{
"caseId": "support-07",
"repeat": 2,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:27:34.672Z",
"durationMs": 439,
"status": "success",
"result": {
"id": "gen-dec-1791451654-3JVp8BqlelasR2XVMKss",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.5263
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"technical": 0.0427,
"billing": 0.0105,
"sales": 0.9468
},
"confidence": 0.8475
},
"severity": {
"type": "score",
"score": 0.1514,
"probabilities": {
"0": 0.8827,
"1": 0.0902,
"2": 0.02,
"3": 0.0071
},
"confidence": 0.717,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 407,
"output_tokens": 0,
"cost": 0.00003663
}
}
}
},
{
"caseId": "support-07",
"repeat": 2,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:27:35.122Z",
"durationMs": 352,
"status": "success",
"result": {
"id": "gen-dec-1791451655-UhjyUrqZxeBI4ySAlUWM",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.07
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"sales": 1,
"technical": 0,
"billing": 0
},
"confidence": 1
},
"severity": {
"type": "score",
"score": 0.02,
"probabilities": {
"0": 0.99,
"1": 0.01,
"2": 0,
"3": 0
},
"confidence": 0.99,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 463,
"output_tokens": 68,
"cost": 0.000019446
}
}
}
},
{
"caseId": "support-07",
"repeat": 2,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:27:35.486Z",
"durationMs": 676,
"status": "success",
"result": {
"id": "gen-dec-1791451655-lA4k55FYTTuZC9pV9lmz",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0068
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"technical": 0.0232,
"billing": 0.005,
"sales": 0.9718
},
"confidence": 0.9173
},
"severity": {
"type": "score",
"score": 0.1706,
"probabilities": {
"0": 0.8547,
"1": 0.1258,
"2": 0.0136,
"3": 0.0059
},
"confidence": 0.6622,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 407,
"output_tokens": 0,
"cost": 0.00009768
}
}
}
},
{
"caseId": "support-07",
"repeat": 3,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:29:06.227Z",
"durationMs": 332,
"status": "success",
"result": {
"id": "gen-dec-1791451746-ndvLUSMwFtr1XHzj2TGk",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.06
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"sales": 1,
"technical": 0,
"billing": 0
},
"confidence": 1
},
"severity": {
"type": "score",
"score": 0.02,
"probabilities": {
"0": 0.99,
"1": 0.01,
"2": 0,
"3": 0
},
"confidence": 0.99,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 463,
"output_tokens": 68,
"cost": 0.000019446
}
}
}
},
{
"caseId": "support-07",
"repeat": 3,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:29:06.574Z",
"durationMs": 620,
"status": "success",
"result": {
"id": "gen-dec-1791451746-hH006Qp1Gamwpb7efMZB",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0068
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"technical": 0.0232,
"billing": 0.005,
"sales": 0.9718
},
"confidence": 0.9173
},
"severity": {
"type": "score",
"score": 0.1706,
"probabilities": {
"0": 0.8547,
"1": 0.1258,
"2": 0.0136,
"3": 0.0059
},
"confidence": 0.6622,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 407,
"output_tokens": 0,
"cost": 0.00009768
}
}
}
},
{
"caseId": "support-07",
"repeat": 3,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:29:07.209Z",
"durationMs": 480,
"status": "success",
"result": {
"id": "gen-dec-1791451747-UfoNLOs5tJJ5S4Mag2E5",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.5263
},
"category": {
"type": "choice",
"choice": "sales",
"probabilities": {
"technical": 0.0427,
"billing": 0.0105,
"sales": 0.9468
},
"confidence": 0.8475
},
"severity": {
"type": "score",
"score": 0.1514,
"probabilities": {
"0": 0.8827,
"1": 0.0902,
"2": 0.02,
"3": 0.0071
},
"confidence": 0.717,
"legend": {
"0": "Information request, cosmetic documentation issue, or no impact",
"1": "One person affected or a minor issue with a workaround",
"2": "An entire company or department affected, not all customers",
"3": "All customers affected by an outage, lost access, or systemic erroneous charges"
}
}
},
"usage": {
"input_tokens": 407,
"output_tokens": 0,
"cost": 0.00003663
}
}
}
}
]
}