HISTORICAL RECORD · 2026-10-08 · invoice-01
A laptop purchase for $900
A laptop purchase for $900. Supplier, invoice number and tax details are present.
Synthetic case. References were set by ClefBench and have not received an independent second review. These are recorded results, not a new model call.
urgent
Does this invoice need review because any required field is missing, supplier or banking details are unverified, or fraud is confirmed?
Reference: false
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | false | false | false | No |
| Clef-flash | false | false | false | No |
| Jev 1.13 | false | false | false | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
category
Classify the purchased item.
Reference: hardware
- hardware: Physical computers, printers, monitors or servers
- software: Software licenses, applications or cloud storage subscriptions
- services: Professional or contractor work
- travel: Transport, accommodation or taxi expenses
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | hardware | hardware | hardware | No |
| Clef-flash | hardware | hardware | hardware | No |
| Jev 1.13 | hardware | hardware | hardware | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
severity
Rate invoice risk from the stated evidence.
Reference: 0
- 0: Complete invoice with no reported issue
- 1: A required invoice field is missing
- 2: Supplier identity or banking details are unverified
- 3: Fraud, forgery or fabrication is confirmed
| Model | Run 1 | Run 2 | Run 3 | Changed answer? |
|---|---|---|---|---|
| Clef | 0.1898 | 0.1898 | 0.1898 | No |
| Clef-flash | 0.841Differs from reference | 0.841Differs from reference | 0.841Differs from reference | No |
| Jev 1.13 | 0.63Differs from reference | 0.62Differs from reference | 0.61Differs from reference | No |
Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.
Time and provider cost for this case
- Clef · run 1: 646 ms · $0.00010104
- Clef-flash · run 1: 446 ms · $0.00003789
- Jev 1.13 · run 1: 346 ms · $0.00001999
- Clef-flash · run 2: 401 ms · $0.00003789
- Jev 1.13 · run 2: 316 ms · $0.00001999
- Clef · run 2: 567 ms · $0.00010104
- Jev 1.13 · run 3: 414 ms · $0.00001999
- Clef · run 3: 538 ms · $0.00010104
- Clef-flash · run 3: 395 ms · $0.00003789
Full task, reference answers and raw recorded responses
{
"sample": {
"state": "A laptop purchase for $900. Supplier, invoice number and tax details are present.",
"questions": {
"urgent": {
"type": "noul",
"instructions": "Does this invoice need review because any required field is missing, supplier or banking details are unverified, or fraud is confirmed?"
},
"category": {
"type": "choice",
"instructions": "Classify the purchased item.",
"criteria": {
"hardware": "Physical computers, printers, monitors or servers",
"software": "Software licenses, applications or cloud storage subscriptions",
"services": "Professional or contractor work",
"travel": "Transport, accommodation or taxi expenses"
}
},
"severity": {
"type": "score",
"instructions": "Rate invoice risk from the stated evidence.",
"criteria": [
"Complete invoice with no reported issue",
"A required invoice field is missing",
"Supplier identity or banking details are unverified",
"Fraud, forgery or fabrication is confirmed"
]
}
}
},
"reference": {
"urgent": false,
"category": "hardware",
"severity": 0
},
"runs": [
{
"caseId": "invoice-01",
"repeat": 1,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:26:24.200Z",
"durationMs": 646,
"status": "success",
"result": {
"id": "gen-dec-1791451584-r2VckNcfvKXCKq1Datfy",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0167
},
"category": {
"type": "choice",
"choice": "hardware",
"probabilities": {
"hardware": 0.9749,
"software": 0.012,
"services": 0.0081,
"travel": 0.005
},
"confidence": 0.9343
},
"severity": {
"type": "score",
"score": 0.1898,
"probabilities": {
"0": 0.8995,
"1": 0.0249,
"2": 0.0618,
"3": 0.0138
},
"confidence": 0.7517,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 421,
"output_tokens": 0,
"cost": 0.00010104
}
}
}
},
{
"caseId": "invoice-01",
"repeat": 1,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:26:24.854Z",
"durationMs": 446,
"status": "success",
"result": {
"id": "gen-dec-1791451584-V5bFXQ8AIAqohl7dhmF3",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.1934
},
"category": {
"type": "choice",
"choice": "hardware",
"probabilities": {
"hardware": 0.9834,
"software": 0.008,
"services": 0.0063,
"travel": 0.0023
},
"confidence": 0.9563
},
"severity": {
"type": "score",
"score": 0.841,
"probabilities": {
"0": 0.5368,
"1": 0.0999,
"2": 0.3488,
"3": 0.0145
},
"confidence": 0.2267,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 421,
"output_tokens": 0,
"cost": 0.00003789
}
}
}
},
{
"caseId": "invoice-01",
"repeat": 1,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:26:25.309Z",
"durationMs": 346,
"status": "success",
"result": {
"id": "gen-dec-1791451585-0E4bvAEVxKTeRAuoqo68",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.27
},
"category": {
"type": "choice",
"choice": "hardware",
"probabilities": {
"services": 0,
"hardware": 1,
"software": 0,
"travel": 0
},
"confidence": 1
},
"severity": {
"type": "score",
"score": 0.63,
"probabilities": {
"0": 0.66,
"1": 0.05,
"2": 0.29,
"3": 0
},
"confidence": 0.37,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 476,
"output_tokens": 75,
"cost": 0.000019992
}
}
}
},
{
"caseId": "invoice-01",
"repeat": 2,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:27:55.110Z",
"durationMs": 401,
"status": "success",
"result": {
"id": "gen-dec-1791451675-pMY2fdlR5OycUnIsbayF",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.1934
},
"category": {
"type": "choice",
"choice": "hardware",
"probabilities": {
"hardware": 0.9834,
"software": 0.008,
"services": 0.0063,
"travel": 0.0023
},
"confidence": 0.9563
},
"severity": {
"type": "score",
"score": 0.841,
"probabilities": {
"0": 0.5368,
"1": 0.0999,
"2": 0.3488,
"3": 0.0145
},
"confidence": 0.2267,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 421,
"output_tokens": 0,
"cost": 0.00003789
}
}
}
},
{
"caseId": "invoice-01",
"repeat": 2,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:27:55.517Z",
"durationMs": 316,
"status": "success",
"result": {
"id": "gen-dec-1791451675-i3elYUbVeYiimngoIxUb",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.33
},
"category": {
"type": "choice",
"choice": "hardware",
"probabilities": {
"services": 0,
"software": 0,
"hardware": 1,
"travel": 0
},
"confidence": 1
},
"severity": {
"type": "score",
"score": 0.62,
"probabilities": {
"0": 0.67,
"1": 0.04,
"2": 0.29,
"3": 0
},
"confidence": 0.38,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 476,
"output_tokens": 75,
"cost": 0.000019992
}
}
}
},
{
"caseId": "invoice-01",
"repeat": 2,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:27:55.837Z",
"durationMs": 567,
"status": "success",
"result": {
"id": "gen-dec-1791451675-A6hGlVzbqnhwWObnvoYz",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0167
},
"category": {
"type": "choice",
"choice": "hardware",
"probabilities": {
"hardware": 0.9749,
"software": 0.012,
"services": 0.0081,
"travel": 0.005
},
"confidence": 0.9343
},
"severity": {
"type": "score",
"score": 0.1898,
"probabilities": {
"0": 0.8995,
"1": 0.0249,
"2": 0.0618,
"3": 0.0138
},
"confidence": 0.7517,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 421,
"output_tokens": 0,
"cost": 0.00010104
}
}
}
},
{
"caseId": "invoice-01",
"repeat": 3,
"outcome": {
"model": "jev",
"measuredAt": "2026-10-08T09:29:26.847Z",
"durationMs": 414,
"status": "success",
"result": {
"id": "gen-dec-1791451766-LEhMggERPEMjwYz2WCWg",
"model": "typesafe/jev-1.13-20260917",
"provider": "TypeSafe",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.28
},
"category": {
"type": "choice",
"choice": "hardware",
"probabilities": {
"software": 0,
"services": 0,
"travel": 0,
"hardware": 1
},
"confidence": 1
},
"severity": {
"type": "score",
"score": 0.61,
"probabilities": {
"0": 0.67,
"1": 0.05,
"2": 0.28,
"3": 0
},
"confidence": 0.39,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 476,
"output_tokens": 75,
"cost": 0.000019992
}
}
}
},
{
"caseId": "invoice-01",
"repeat": 3,
"outcome": {
"model": "clef",
"measuredAt": "2026-10-08T09:29:27.275Z",
"durationMs": 538,
"status": "success",
"result": {
"id": "gen-dec-1791451767-ULmwzQQCwIq1IgmeDfF0",
"model": "cloudflare/clef",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.0167
},
"category": {
"type": "choice",
"choice": "hardware",
"probabilities": {
"hardware": 0.9749,
"software": 0.012,
"services": 0.0081,
"travel": 0.005
},
"confidence": 0.9343
},
"severity": {
"type": "score",
"score": 0.1898,
"probabilities": {
"0": 0.8995,
"1": 0.0249,
"2": 0.0618,
"3": 0.0138
},
"confidence": 0.7517,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 421,
"output_tokens": 0,
"cost": 0.00010104
}
}
}
},
{
"caseId": "invoice-01",
"repeat": 3,
"outcome": {
"model": "clef-flash",
"measuredAt": "2026-10-08T09:29:27.829Z",
"durationMs": 395,
"status": "success",
"result": {
"id": "gen-dec-1791451767-6TCZdmjJ6x2P26i9iscv",
"model": "cloudflare/clef-flash",
"provider": "Cloudflare",
"answers": {
"urgent": {
"type": "noul",
"noul": 0.1934
},
"category": {
"type": "choice",
"choice": "hardware",
"probabilities": {
"hardware": 0.9834,
"software": 0.008,
"services": 0.0063,
"travel": 0.0023
},
"confidence": 0.9563
},
"severity": {
"type": "score",
"score": 0.841,
"probabilities": {
"0": 0.5368,
"1": 0.0999,
"2": 0.3488,
"3": 0.0145
},
"confidence": 0.2267,
"legend": {
"0": "Complete invoice with no reported issue",
"1": "A required invoice field is missing",
"2": "Supplier identity or banking details are unverified",
"3": "Fraud, forgery or fabrication is confirmed"
}
}
},
"usage": {
"input_tokens": 421,
"output_tokens": 0,
"cost": 0.00003789
}
}
}
}
]
}