HISTORICAL RECORD · 2026-10-08 · risk-08
All regional deliveries are delayed for a day; no physical danger is present.
All regional deliveries are delayed for a day; no physical danger is present.
Synthetic case. References were set by ClefBench and have not received an independent second review. These are recorded results, not a new model call.
urgent
Does this situation meet level 2 or 3 on the severity rubric and therefore require escalation?
Reference: true
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | falseDiffers from reference | falseDiffers from reference | falseDiffers from reference | No |
| Clef-flash | falseDiffers from reference | falseDiffers from reference | falseDiffers from reference | No |
| Jev 1.13 | falseDiffers from reference | falseDiffers from reference | falseDiffers from reference | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
category
Select the main risk domain.
Reference: operations
- security: Digital access, data or cybersecurity
- safety: Physical harm, fire, building or chemical safety
- operations: Availability, logistics or business workflow
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | operations | operations | operations | No |
| Clef-flash | operations | operations | operations | No |
| Jev 1.13 | operations | operations | operations | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
severity
Rate the risk using only stated evidence.
Reference: 2
- 0: No unresolved risk or impact
- 1: Contained minor issue, blocked attempt, or safe workaround
- 2: Serious unresolved vulnerability, potential physical harm, or a regional workflow blocked
- 3: Active data theft or destruction, immediate threat to life, or global outage with no workaround
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | 1.6327 | 1.6327 | 1.6327 | No |
| Clef-flash | 1.0734Differs from reference | 1.0734Differs from reference | 1.0734Differs from reference | No |
| Jev 1.13 | 1.5 | 1.49Differs from reference | 1.5 | Yes |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
Time and provider cost for this case
- Clef · run 1: 825 ms · $0.00009840
- Clef-flash · run 1: 454 ms · $0.00003690
- Jev 1.13 · run 1: 372 ms · $0.00001961
- Clef-flash · run 2: 425 ms · $0.00003690
- Jev 1.13 · run 2: 353 ms · $0.00001961
- Clef · run 2: 637 ms · $0.00009840
- Jev 1.13 · run 3: 324 ms · $0.00001961
- Clef · run 3: 871 ms · $0.00009840
- Clef-flash · run 3: 462 ms · $0.00003690
Full task, reference answers and raw recorded responses
{
"sample": {
"state": "All regional deliveries are delayed for a day; no physical danger is present.",
"questions": {
"urgent": {
"type": "noul",
"instructions": "Does this situation meet level 2 or 3 on the severity rubric and therefore require escalation?"
},
"category": {
"type": "choice",
"instructions": "Select the main risk domain.",
"criteria": {
"security": "Digital access, data or cybersecurity",
"safety": "Physical harm, fire, building or chemical safety",
"operations": "Availability, logistics or business workflow"
}
},
"severity": {
"type": "score",
"instructions": "Rate the risk using only stated evidence.",
"criteria": [
"No unresolved risk or impact",
"Contained minor issue, blocked attempt, or safe workaround",
"Serious unresolved vulnerability, potential physical harm, or a regional workflow blocked",
"Active data theft or destruction, immediate threat to life, or global outage with no workaround"
]
}
}
},
"reference": {
"urgent": true,
"category": "operations",
"severity": 2
},
"runs": [
{
"caseId": "risk-08",
"repeat": 1,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:27:05.185Z",
"durationMs": 825,
"status": "success",
"result": {
"id": "gen-dec-1791451625-dNvreIbmhKWdgN80IhFg",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0578
},
"category": {
"type": "choice",
"choice": "operations",
"probabilities": {
"security": 0.0097,
"safety": 0.0078,
"operations": 0.9825
},
"confidence": 0.9482
},
"severity": {
"type": "score",
"score": 1.6327,
"probabilities": {
"0": 0.0246,
"1": 0.3345,
"2": 0.6245,
"3": 0.0164
},
"confidence": 0.3369,
"legend": {
"0": "No unresolved risk or impact",
"1": "Contained minor issue, blocked attempt, or safe workaround",
"2": "Serious unresolved vulnerability, potential physical harm, or a regional workflow blocked",
"3": "Active data theft or destruction, immediate threat to life, or global outage with no workaround"
}
}
},
"usage": {
"input_tokens": 410,
"output_tokens": 0,
"cost": 0.0000984
}
}
}
},
{
"caseId": "risk-08",
"repeat": 1,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:27:06.019Z",
"durationMs": 454,
"status": "success",
"result": {
"id": "gen-dec-1791451626-PwguSHrLUpoXvOhgdBwq",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0465
},
"category": {
"type": "choice",
"choice": "operations",
"probabilities": {
"security": 0.0047,
"safety": 0.0058,
"operations": 0.9895
},
"confidence": 0.9686
},
"severity": {
"type": "score",
"score": 1.0734,
"probabilities": {
"0": 0.0974,
"1": 0.7523,
"2": 0.1299,
"3": 0.0204
},
"confidence": 0.457,
"legend": {
"0": "No unresolved risk or impact",
"1": "Contained minor issue, blocked attempt, or safe workaround",
"2": "Serious unresolved vulnerability, potential physical harm, or a regional workflow blocked",
"3": "Active data theft or destruction, immediate threat to life, or global outage with no workaround"
}
}
},
"usage": {
"input_tokens": 410,
"output_tokens": 0,
"cost": 0.0000369
}
}
}
},
{
"caseId": "risk-08",
"repeat": 1,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:27:06.483Z",
"durationMs": 372,
"status": "success",
"result": {
"id": "gen-dec-1791451626-uh7EKvw7nt7TesuDqziC",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.18
},
"category": {
"type": "choice",
"choice": "operations",
"probabilities": {
"safety": 0,
"operations": 1,
"security": 0
},
"confidence": 1
},
"severity": {
"type": "score",
"score": 1.5,
"probabilities": {
"0": 0.04,
"1": 0.42,
"2": 0.54,
"3": 0
},
"confidence": 0.5,
"legend": {
"0": "No unresolved risk or impact",
"1": "Contained minor issue, blocked attempt, or safe workaround",
"2": "Serious unresolved vulnerability, potential physical harm, or a regional workflow blocked",
"3": "Active data theft or destruction, immediate threat to life, or global outage with no workaround"
}
}
},
"usage": {
"input_tokens": 467,
"output_tokens": 70,
"cost": 0.000019614
}
}
}
},
{
"caseId": "risk-08",
"repeat": 2,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:28:35.334Z",
"durationMs": 425,
"status": "success",
"result": {
"id": "gen-dec-1791451715-i9kvHoL2j04BKBbaacxo",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0465
},
"category": {
"type": "choice",
"choice": "operations",
"probabilities": {
"security": 0.0047,
"safety": 0.0058,
"operations": 0.9895
},
"confidence": 0.9686
},
"severity": {
"type": "score",
"score": 1.0734,
"probabilities": {
"0": 0.0974,
"1": 0.7523,
"2": 0.1299,
"3": 0.0204
},
"confidence": 0.457,
"legend": {
"0": "No unresolved risk or impact",
"1": "Contained minor issue, blocked attempt, or safe workaround",
"2": "Serious unresolved vulnerability, potential physical harm, or a regional workflow blocked",
"3": "Active data theft or destruction, immediate threat to life, or global outage with no workaround"
}
}
},
"usage": {
"input_tokens": 410,
"output_tokens": 0,
"cost": 0.0000369
}
}
}
},
{
"caseId": "risk-08",
"repeat": 2,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:28:35.773Z",
"durationMs": 353,
"status": "success",
"result": {
"id": "gen-dec-1791451715-2oG8bx1ikvf152Obx2c4",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.18
},
"category": {
"type": "choice",
"choice": "operations",
"probabilities": {
"security": 0,
"safety": 0,
"operations": 1
},
"confidence": 1
},
"severity": {
"type": "score",
"score": 1.49,
"probabilities": {
"0": 0.03,
"1": 0.46,
"2": 0.51,
"3": 0
},
"confidence": 0.48,
"legend": {
"0": "No unresolved risk or impact",
"1": "Contained minor issue, blocked attempt, or safe workaround",
"2": "Serious unresolved vulnerability, potential physical harm, or a regional workflow blocked",
"3": "Active data theft or destruction, immediate threat to life, or global outage with no workaround"
}
}
},
"usage": {
"input_tokens": 467,
"output_tokens": 70,
"cost": 0.000019614
}
}
}
},
{
"caseId": "risk-08",
"repeat": 2,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:28:36.139Z",
"durationMs": 637,
"status": "success",
"result": {
"id": "gen-dec-1791451716-cawOSwXGETgKnMMhtCtR",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0578
},
"category": {
"type": "choice",
"choice": "operations",
"probabilities": {
"security": 0.0097,
"safety": 0.0078,
"operations": 0.9825
},
"confidence": 0.9482
},
"severity": {
"type": "score",
"score": 1.6327,
"probabilities": {
"0": 0.0246,
"1": 0.3345,
"2": 0.6245,
"3": 0.0164
},
"confidence": 0.3369,
"legend": {
"0": "No unresolved risk or impact",
"1": "Contained minor issue, blocked attempt, or safe workaround",
"2": "Serious unresolved vulnerability, potential physical harm, or a regional workflow blocked",
"3": "Active data theft or destruction, immediate threat to life, or global outage with no workaround"
}
}
},
"usage": {
"input_tokens": 410,
"output_tokens": 0,
"cost": 0.0000984
}
}
}
},
{
"caseId": "risk-08",
"repeat": 3,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:30:07.324Z",
"durationMs": 324,
"status": "success",
"result": {
"id": "gen-dec-1791451807-9YtYBjd56eDL9Zp3cnBv",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.18
},
"category": {
"type": "choice",
"choice": "operations",
"probabilities": {
"operations": 1,
"safety": 0,
"security": 0
},
"confidence": 1
},
"severity": {
"type": "score",
"score": 1.5,
"probabilities": {
"0": 0.02,
"1": 0.46,
"2": 0.52,
"3": 0
},
"confidence": 0.5,
"legend": {
"0": "No unresolved risk or impact",
"1": "Contained minor issue, blocked attempt, or safe workaround",
"2": "Serious unresolved vulnerability, potential physical harm, or a regional workflow blocked",
"3": "Active data theft or destruction, immediate threat to life, or global outage with no workaround"
}
}
},
"usage": {
"input_tokens": 467,
"output_tokens": 70,
"cost": 0.000019614
}
}
}
},
{
"caseId": "risk-08",
"repeat": 3,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:30:07.665Z",
"durationMs": 871,
"status": "success",
"result": {
"id": "gen-dec-1791451807-aQpkU6txbJBiNT1XTV8e",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0578
},
"category": {
"type": "choice",
"choice": "operations",
"probabilities": {
"security": 0.0097,
"safety": 0.0078,
"operations": 0.9825
},
"confidence": 0.9482
},
"severity": {
"type": "score",
"score": 1.6327,
"probabilities": {
"0": 0.0246,
"1": 0.3345,
"2": 0.6245,
"3": 0.0164
},
"confidence": 0.3369,
"legend": {
"0": "No unresolved risk or impact",
"1": "Contained minor issue, blocked attempt, or safe workaround",
"2": "Serious unresolved vulnerability, potential physical harm, or a regional workflow blocked",
"3": "Active data theft or destruction, immediate threat to life, or global outage with no workaround"
}
}
},
"usage": {
"input_tokens": 410,
"output_tokens": 0,
"cost": 0.0000984
}
}
}
},
{
"caseId": "risk-08",
"repeat": 3,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:30:08.553Z",
"durationMs": 462,
"status": "success",
"result": {
"id": "gen-dec-1791451808-Ah4EI8mKV7Ov7BZ62Zs4",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0465
},
"category": {
"type": "choice",
"choice": "operations",
"probabilities": {
"security": 0.0047,
"safety": 0.0058,
"operations": 0.9895
},
"confidence": 0.9686
},
"severity": {
"type": "score",
"score": 1.0734,
"probabilities": {
"0": 0.0974,
"1": 0.7523,
"2": 0.1299,
"3": 0.0204
},
"confidence": 0.457,
"legend": {
"0": "No unresolved risk or impact",
"1": "Contained minor issue, blocked attempt, or safe workaround",
"2": "Serious unresolved vulnerability, potential physical harm, or a regional workflow blocked",
"3": "Active data theft or destruction, immediate threat to life, or global outage with no workaround"
}
}
},
"usage": {
"input_tokens": 410,
"output_tokens": 0,
"cost": 0.0000369
}
}
}
}
]
}