HISTORICAL RECORD · 2026-10-08 · invoice-08
Cleaning contractor requests $800 to a newly substituted bank account confirmed fraudulent.
Cleaning contractor requests $800 to a newly substituted bank account confirmed fraudulent.
Synthetic case. References were set by ClefBench and have not received an independent second review. These are recorded results, not a new model call.
urgent
Does this invoice need review because any required field is missing, supplier or banking details are unverified, or fraud is confirmed?
Reference: true
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | true | true | true | No |
| Clef-flash | true | true | true | No |
| Jev 1.13 | true | true | true | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
category
Classify the purchased item.
Reference: services
- hardware: Physical computers, printers, monitors or servers
- software: Software licenses, applications or cloud storage subscriptions
- services: Professional or contractor work
- travel: Transport, accommodation or taxi expenses
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | services | services | services | No |
| Clef-flash | services | services | services | No |
| Jev 1.13 | services | services | services | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
severity
Rate invoice risk from the stated evidence.
Reference: 3
- 0: Complete invoice with no reported issue
- 1: A required invoice field is missing
- 2: Supplier identity or banking details are unverified
- 3: Fraud, forgery or fabrication is confirmed
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | 2.9735 | 2.9735 | 2.9735 | No |
| Clef-flash | 2.8726 | 2.8726 | 2.8726 | No |
| Jev 1.13 | 3 | 3 | 3 | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
Time and provider cost for this case
- Clef · run 1: 880 ms · $0.00010032
- Clef-flash · run 1: 461 ms · $0.00003762
- Jev 1.13 · run 1: 362 ms · $0.00001987
- Clef-flash · run 2: 456 ms · $0.00003762
- Jev 1.13 · run 2: 387 ms · $0.00001987
- Clef · run 2: 683 ms · $0.00010032
- Jev 1.13 · run 3: 376 ms · $0.00001987
- Clef · run 3: 631 ms · $0.00010032
- Clef-flash · run 3: 433 ms · $0.00003762
Full task, reference answers and raw recorded responses
{
"sample": {
"state": "Cleaning contractor requests $800 to a newly substituted bank account confirmed fraudulent.",
"questions": {
"urgent": {
"type": "noul",
"instructions": "Does this invoice need review because any required field is missing, supplier or banking details are unverified, or fraud is confirmed?"
},
"category": {
"type": "choice",
"instructions": "Classify the purchased item.",
"criteria": {
"hardware": "Physical computers, printers, monitors or servers",
"software": "Software licenses, applications or cloud storage subscriptions",
"services": "Professional or contractor work",
"travel": "Transport, accommodation or taxi expenses"
}
},
"severity": {
"type": "score",
"instructions": "Rate invoice risk from the stated evidence.",
"criteria": [
"Complete invoice with no reported issue",
"A required invoice field is missing",
"Supplier identity or banking details are unverified",
"Fraud, forgery or fabrication is confirmed"
]
}
}
},
"reference": {
"urgent": true,
"category": "services",
"severity": 3
},
"runs": [
{
"caseId": "invoice-08",
"repeat": 1,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:26:34.478Z",
"durationMs": 880,
"status": "success",
"result": {
"id": "gen-dec-1791451594-nICSRDK3QQmLr1RLeR2c",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.9868
},
"category": {
"type": "choice",
"choice": "services",
"probabilities": {
"hardware": 0.0076,
"software": 0.0058,
"services": 0.9778,
"travel": 0.0088
},
"confidence": 0.9416
},
"severity": {
"type": "score",
"score": 2.9735,
"probabilities": {
"0": 0.0035,
"1": 0.0034,
"2": 0.0093,
"3": 0.9838
},
"confidence": 0.9573,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 418,
"output_tokens": 0,
"cost": 0.00010032
}
}
}
},
{
"caseId": "invoice-08",
"repeat": 1,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:26:35.365Z",
"durationMs": 461,
"status": "success",
"result": {
"id": "gen-dec-1791451595-87w39b4yNMkN5SkHaFsN",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.9623
},
"category": {
"type": "choice",
"choice": "services",
"probabilities": {
"hardware": 0.0076,
"software": 0.0184,
"services": 0.9556,
"travel": 0.0184
},
"confidence": 0.8852
},
"severity": {
"type": "score",
"score": 2.8726,
"probabilities": {
"0": 0.0119,
"1": 0.0159,
"2": 0.06,
"3": 0.9122
},
"confidence": 0.7815,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 418,
"output_tokens": 0,
"cost": 0.00003762
}
}
}
},
{
"caseId": "invoice-08",
"repeat": 1,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:26:35.834Z",
"durationMs": 362,
"status": "success",
"result": {
"id": "gen-dec-1791451595-YSnxJJrAmHaMqverKCvr",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.98
},
"category": {
"type": "choice",
"choice": "services",
"probabilities": {
"services": 1,
"hardware": 0,
"software": 0,
"travel": 0
},
"confidence": 1
},
"severity": {
"type": "score",
"score": 3,
"probabilities": {
"0": 0,
"1": 0,
"2": 0,
"3": 1
},
"confidence": 1,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 473,
"output_tokens": 75,
"cost": 0.000019866
}
}
}
},
{
"caseId": "invoice-08",
"repeat": 2,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:28:06.053Z",
"durationMs": 456,
"status": "success",
"result": {
"id": "gen-dec-1791451686-HhYvKIYE5VuB3SlpiZ2b",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.9623
},
"category": {
"type": "choice",
"choice": "services",
"probabilities": {
"hardware": 0.0076,
"software": 0.0184,
"services": 0.9556,
"travel": 0.0184
},
"confidence": 0.8852
},
"severity": {
"type": "score",
"score": 2.8726,
"probabilities": {
"0": 0.0119,
"1": 0.0159,
"2": 0.06,
"3": 0.9122
},
"confidence": 0.7815,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 418,
"output_tokens": 0,
"cost": 0.00003762
}
}
}
},
{
"caseId": "invoice-08",
"repeat": 2,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:28:06.518Z",
"durationMs": 387,
"status": "success",
"result": {
"id": "gen-dec-1791451686-3FU2dpQfhM2kaFY1z8mG",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.97
},
"category": {
"type": "choice",
"choice": "services",
"probabilities": {
"services": 1,
"software": 0,
"hardware": 0,
"travel": 0
},
"confidence": 1
},
"severity": {
"type": "score",
"score": 3,
"probabilities": {
"0": 0,
"1": 0,
"2": 0,
"3": 1
},
"confidence": 1,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 473,
"output_tokens": 75,
"cost": 0.000019866
}
}
}
},
{
"caseId": "invoice-08",
"repeat": 2,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:28:06.920Z",
"durationMs": 683,
"status": "success",
"result": {
"id": "gen-dec-1791451686-t9mw8GdhMz2sBkoHM21a",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.9868
},
"category": {
"type": "choice",
"choice": "services",
"probabilities": {
"hardware": 0.0076,
"software": 0.0058,
"services": 0.9778,
"travel": 0.0088
},
"confidence": 0.9416
},
"severity": {
"type": "score",
"score": 2.9735,
"probabilities": {
"0": 0.0035,
"1": 0.0034,
"2": 0.0093,
"3": 0.9838
},
"confidence": 0.9573,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 418,
"output_tokens": 0,
"cost": 0.00010032
}
}
}
},
{
"caseId": "invoice-08",
"repeat": 3,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:29:36.536Z",
"durationMs": 376,
"status": "success",
"result": {
"id": "gen-dec-1791451776-5ZuCnLdbTMmwG59s0e80",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.97
},
"category": {
"type": "choice",
"choice": "services",
"probabilities": {
"hardware": 0,
"software": 0,
"services": 1,
"travel": 0
},
"confidence": 1
},
"severity": {
"type": "score",
"score": 3,
"probabilities": {
"0": 0,
"1": 0,
"2": 0,
"3": 1
},
"confidence": 1,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 473,
"output_tokens": 75,
"cost": 0.000019866
}
}
}
},
{
"caseId": "invoice-08",
"repeat": 3,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:29:36.928Z",
"durationMs": 631,
"status": "success",
"result": {
"id": "gen-dec-1791451776-8EPiRQs65jAICD31GgO3",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.9868
},
"category": {
"type": "choice",
"choice": "services",
"probabilities": {
"hardware": 0.0076,
"software": 0.0058,
"services": 0.9778,
"travel": 0.0088
},
"confidence": 0.9416
},
"severity": {
"type": "score",
"score": 2.9735,
"probabilities": {
"0": 0.0035,
"1": 0.0034,
"2": 0.0093,
"3": 0.9838
},
"confidence": 0.9573,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 418,
"output_tokens": 0,
"cost": 0.00010032
}
}
}
},
{
"caseId": "invoice-08",
"repeat": 3,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:29:37.575Z",
"durationMs": 433,
"status": "success",
"result": {
"id": "gen-dec-1791451777-R3bgTC7tYmpRjbwq9RKW",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.9623
},
"category": {
"type": "choice",
"choice": "services",
"probabilities": {
"hardware": 0.0076,
"software": 0.0184,
"services": 0.9556,
"travel": 0.0184
},
"confidence": 0.8852
},
"severity": {
"type": "score",
"score": 2.8726,
"probabilities": {
"0": 0.0119,
"1": 0.0159,
"2": 0.06,
"3": 0.9122
},
"confidence": 0.7815,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 418,
"output_tokens": 0,
"cost": 0.00003762
}
}
}
}
]
}