HISTORICAL RECORD · 2026-10-08 · invoice-16

Translation work $150 with complete verified supplier details.

Synthetic case. References were set by ClefBench and have not received an independent second review. These are recorded results, not a new model call.

urgent

Does this invoice need review because any required field is missing, supplier or banking details are unverified, or fraud is confirmed?

Reference: false

Model	Run 1	Run 2	Run 3	Changed answer?
Clef	false	false	false	No
Clef-flash	false	false	false	No
Jev 1.13	false	false	false	No

Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.

Model	Run 1	Run 2	Run 3	Changed answer?
Clef	services	services	services	No
Clef-flash	services	services	services	No
Jev 1.13	services	services	services	No

severity

Rate invoice risk from the stated evidence.

Reference: 0

0: Complete invoice with no reported issue
1: A required invoice field is missing
2: Supplier identity or banking details are unverified
3: Fraud, forgery or fabrication is confirmed

Model	Run 1	Run 2	Run 3	Changed answer?
Clef	0.1169	0.1169	0.1169	No
Clef-flash	0.5502Differs from reference	0.5502Differs from reference	0.5502Differs from reference	No
Jev 1.13	0.34	0.32	0.31	No

Time and provider cost for this case

Clef · run 1: 735 ms · $0.00009936
Clef-flash · run 1: 424 ms · $0.00003726
Jev 1.13 · run 1: 396 ms · $0.00001970
Clef-flash · run 2: 433 ms · $0.00003726
Jev 1.13 · run 2: 349 ms · $0.00001970
Clef · run 2: 917 ms · $0.00009936
Jev 1.13 · run 3: 366 ms · $0.00001970
Clef · run 3: 685 ms · $0.00009936
Clef-flash · run 3: 413 ms · $0.00003726

Full task, reference answers and raw recorded responses

{
  "sample": {
    "state": "Translation work $150 with complete verified supplier details.",
    "questions": {
      "urgent": {
        "type": "noul",
        "instructions": "Does this invoice need review because any required field is missing, supplier or banking details are unverified, or fraud is confirmed?"
      },
      "category": {
        "type": "choice",
        "instructions": "Classify the purchased item.",
        "criteria": {
          "hardware": "Physical computers, printers, monitors or servers",
          "software": "Software licenses, applications or cloud storage subscriptions",
          "services": "Professional or contractor work",
          "travel": "Transport, accommodation or taxi expenses"
        }
      },
      "severity": {
        "type": "score",
        "instructions": "Rate invoice risk from the stated evidence.",
        "criteria": [
          "Complete invoice with no reported issue",
          "A required invoice field is missing",
          "Supplier identity or banking details are unverified",
          "Fraud, forgery or fabrication is confirmed"
        ]
      }
    }
  },
  "reference": {
    "urgent": false,
    "category": "services",
    "severity": 0
  },
  "runs": [
    {
      "caseId": "invoice-16",
      "repeat": 1,
      "outcome": {
        "model": "clef",
        "measuredAt": "2026-10-08T09:26:47.012Z",
        "durationMs": 735,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451606-3o0eNaU5arrs80SNiIBQ",
          "model": "cloudflare/clef",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.0098
            },
            "category": {
              "type": "choice",
              "choice": "services",
              "probabilities": {
                "hardware": 0.0076,
                "software": 0.0128,
                "services": 0.9695,
                "travel": 0.0101
              },
              "confidence": 0.9205
            },
            "severity": {
              "type": "score",
              "score": 0.1169,
              "probabilities": {
                "0": 0.9338,
                "1": 0.0291,
                "2": 0.0236,
                "3": 0.0135
              },
              "confidence": 0.8313,
              "legend": {
                "0": "Complete invoice with no reported issue",
                "1": "A required invoice field is missing",
                "2": "Supplier identity or banking details are unverified",
                "3": "Fraud, forgery or fabrication is confirmed"
              }
            }
          },
          "usage": {
            "input_tokens": 414,
            "output_tokens": 0,
            "cost": 0.00009936
          }
        }
      }
    },
    {
      "caseId": "invoice-16",
      "repeat": 1,
      "outcome": {
        "model": "clef-flash",
        "measuredAt": "2026-10-08T09:26:47.755Z",
        "durationMs": 424,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451607-UYKkMSO65jsk7RPaj9KJ",
          "model": "cloudflare/clef-flash",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.1474
            },
            "category": {
              "type": "choice",
              "choice": "services",
              "probabilities": {
                "hardware": 0.0024,
                "software": 0.0106,
                "services": 0.9753,
                "travel": 0.0117
              },
              "confidence": 0.9353
            },
            "severity": {
              "type": "score",
              "score": 0.5502,
              "probabilities": {
                "0": 0.6845,
                "1": 0.1096,
                "2": 0.177,
                "3": 0.0289
              },
              "confidence": 0.3503,
              "legend": {
                "0": "Complete invoice with no reported issue",
                "1": "A required invoice field is missing",
                "2": "Supplier identity or banking details are unverified",
                "3": "Fraud, forgery or fabrication is confirmed"
              }
            }
          },
          "usage": {
            "input_tokens": 414,
            "output_tokens": 0,
            "cost": 0.00003726
          }
        }
      }
    },
    {
      "caseId": "invoice-16",
      "repeat": 1,
      "outcome": {
        "model": "jev",
        "measuredAt": "2026-10-08T09:26:48.188Z",
        "durationMs": 396,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451608-AmYYoZvPAPSTlKJ4HdG5",
          "model": "typesafe/jev-1.13-20260917",
          "provider": "TypeSafe",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.17
            },
            "category": {
              "type": "choice",
              "choice": "services",
              "probabilities": {
                "software": 0,
                "services": 1,
                "travel": 0,
                "hardware": 0
              },
              "confidence": 1
            },
            "severity": {
              "type": "score",
              "score": 0.34,
              "probabilities": {
                "0": 0.73,
                "1": 0.21,
                "2": 0.06,
                "3": 0
              },
              "confidence": 0.67,
              "legend": {
                "0": "Complete invoice with no reported issue",
                "1": "A required invoice field is missing",
                "2": "Supplier identity or banking details are unverified",
                "3": "Fraud, forgery or fabrication is confirmed"
              }
            }
          },
          "usage": {
            "input_tokens": 469,
            "output_tokens": 75,
            "cost": 0.000019698
          }
        }
      }
    },
    {
      "caseId": "invoice-16",
      "repeat": 2,
      "outcome": {
        "model": "clef-flash",
        "measuredAt": "2026-10-08T09:28:17.429Z",
        "durationMs": 433,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451697-izM3Uu9WnlI0z5I4KHzQ",
          "model": "cloudflare/clef-flash",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.1474
            },
            "category": {
              "type": "choice",
              "choice": "services",
              "probabilities": {
                "hardware": 0.0024,
                "software": 0.0106,
                "services": 0.9753,
                "travel": 0.0117
              },
              "confidence": 0.9353
            },
            "severity": {
              "type": "score",
              "score": 0.5502,
              "probabilities": {
                "0": 0.6845,
                "1": 0.1096,
                "2": 0.177,
                "3": 0.0289
              },
              "confidence": 0.3503,
              "legend": {
                "0": "Complete invoice with no reported issue",
                "1": "A required invoice field is missing",
                "2": "Supplier identity or banking details are unverified",
                "3": "Fraud, forgery or fabrication is confirmed"
              }
            }
          },
          "usage": {
            "input_tokens": 414,
            "output_tokens": 0,
            "cost": 0.00003726
          }
        }
      }
    },
    {
      "caseId": "invoice-16",
      "repeat": 2,
      "outcome": {
        "model": "jev",
        "measuredAt": "2026-10-08T09:28:17.875Z",
        "durationMs": 349,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451697-2Z5ei2GXMxk7zLDB4Xsa",
          "model": "typesafe/jev-1.13-20260917",
          "provider": "TypeSafe",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.17
            },
            "category": {
              "type": "choice",
              "choice": "services",
              "probabilities": {
                "hardware": 0,
                "travel": 0,
                "services": 1,
                "software": 0
              },
              "confidence": 1
            },
            "severity": {
              "type": "score",
              "score": 0.32,
              "probabilities": {
                "0": 0.75,
                "1": 0.19,
                "2": 0.06,
                "3": 0
              },
              "confidence": 0.69,
              "legend": {
                "0": "Complete invoice with no reported issue",
                "1": "A required invoice field is missing",
                "2": "Supplier identity or banking details are unverified",
                "3": "Fraud, forgery or fabrication is confirmed"
              }
            }
          },
          "usage": {
            "input_tokens": 469,
            "output_tokens": 75,
            "cost": 0.000019698
          }
        }
      }
    },
    {
      "caseId": "invoice-16",
      "repeat": 2,
      "outcome": {
        "model": "clef",
        "measuredAt": "2026-10-08T09:28:18.237Z",
        "durationMs": 917,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451698-s8e4kxSLwoQUMSbUeryF",
          "model": "cloudflare/clef",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.0098
            },
            "category": {
              "type": "choice",
              "choice": "services",
              "probabilities": {
                "hardware": 0.0076,
                "software": 0.0128,
                "services": 0.9695,
                "travel": 0.0101
              },
              "confidence": 0.9205
            },
            "severity": {
              "type": "score",
              "score": 0.1169,
              "probabilities": {
                "0": 0.9338,
                "1": 0.0291,
                "2": 0.0236,
                "3": 0.0135
              },
              "confidence": 0.8313,
              "legend": {
                "0": "Complete invoice with no reported issue",
                "1": "A required invoice field is missing",
                "2": "Supplier identity or banking details are unverified",
                "3": "Fraud, forgery or fabrication is confirmed"
              }
            }
          },
          "usage": {
            "input_tokens": 414,
            "output_tokens": 0,
            "cost": 0.00009936
          }
        }
      }
    },
    {
      "caseId": "invoice-16",
      "repeat": 3,
      "outcome": {
        "model": "jev",
        "measuredAt": "2026-10-08T09:29:48.629Z",
        "durationMs": 366,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451788-39wPMlnwor9t9EqgjNM1",
          "model": "typesafe/jev-1.13-20260917",
          "provider": "TypeSafe",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.16
            },
            "category": {
              "type": "choice",
              "choice": "services",
              "probabilities": {
                "hardware": 0,
                "software": 0,
                "services": 1,
                "travel": 0
              },
              "confidence": 1
            },
            "severity": {
              "type": "score",
              "score": 0.31,
              "probabilities": {
                "0": 0.76,
                "1": 0.18,
                "2": 0.06,
                "3": 0
              },
              "confidence": 0.7,
              "legend": {
                "0": "Complete invoice with no reported issue",
                "1": "A required invoice field is missing",
                "2": "Supplier identity or banking details are unverified",
                "3": "Fraud, forgery or fabrication is confirmed"
              }
            }
          },
          "usage": {
            "input_tokens": 469,
            "output_tokens": 75,
            "cost": 0.000019698
          }
        }
      }
    },
    {
      "caseId": "invoice-16",
      "repeat": 3,
      "outcome": {
        "model": "clef",
        "measuredAt": "2026-10-08T09:29:49.007Z",
        "durationMs": 685,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451788-9beIdzv618eSmCyOFWbw",
          "model": "cloudflare/clef",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.0098
            },
            "category": {
              "type": "choice",
              "choice": "services",
              "probabilities": {
                "hardware": 0.0076,
                "software": 0.0128,
                "services": 0.9695,
                "travel": 0.0101
              },
              "confidence": 0.9205
            },
            "severity": {
              "type": "score",
              "score": 0.1169,
              "probabilities": {
                "0": 0.9338,
                "1": 0.0291,
                "2": 0.0236,
                "3": 0.0135
              },
              "confidence": 0.8313,
              "legend": {
                "0": "Complete invoice with no reported issue",
                "1": "A required invoice field is missing",
                "2": "Supplier identity or banking details are unverified",
                "3": "Fraud, forgery or fabrication is confirmed"
              }
            }
          },
          "usage": {
            "input_tokens": 414,
            "output_tokens": 0,
            "cost": 0.00009936
          }
        }
      }
    },
    {
      "caseId": "invoice-16",
      "repeat": 3,
      "outcome": {
        "model": "clef-flash",
        "measuredAt": "2026-10-08T09:29:49.704Z",
        "durationMs": 413,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451789-8xMPvSi2c2C1Kfkc1YPj",
          "model": "cloudflare/clef-flash",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.1474
            },
            "category": {
              "type": "choice",
              "choice": "services",
              "probabilities": {
                "hardware": 0.0024,
                "software": 0.0106,
                "services": 0.9753,
                "travel": 0.0117
              },
              "confidence": 0.9353
            },
            "severity": {
              "type": "score",
              "score": 0.5502,
              "probabilities": {
                "0": 0.6845,
                "1": 0.1096,
                "2": 0.177,
                "3": 0.0289
              },
              "confidence": 0.3503,
              "legend": {
                "0": "Complete invoice with no reported issue",
                "1": "A required invoice field is missing",
                "2": "Supplier identity or banking details are unverified",
                "3": "Fraud, forgery or fabrication is confirmed"
              }
            }
          },
          "usage": {
            "input_tokens": 414,
            "output_tokens": 0,
            "cost": 0.00003726
          }
        }
      }
    }
  ]
}

Download the complete original results