HISTORICAL RECORD · 2026-10-08 · support-20

Everyone in our company sees incorrect invoice totals and cannot pay

Everyone in our company sees incorrect invoice totals and cannot pay. Other customers are fine.

Synthetic case. References were set by ClefBench and have not received an independent second review. These are recorded results, not a new model call.

urgent

Is this a service-wide incident affecting all customers or all tenants? A single company or user does not count.

Reference: false

Model	Run 1	Run 2	Run 3	Changed answer?
Clef	false	false	false	No
Clef-flash	false	false	false	No
Jev 1.13	false	false	false	No

Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.

Model	Run 1	Run 2	Run 3	Changed answer?
Clef	billing	billing	billing	No
Clef-flash	billing	billing	billing	No
Jev 1.13	billing	billing	billing	No

severity

Rate the stated impact, using the highest applicable level.

Reference: 2

0: Information request, cosmetic documentation issue, or no impact
1: One person affected or a minor issue with a workaround
2: An entire company or department affected, not all customers
3: All customers affected by an outage, lost access, or systemic erroneous charges

Model	Run 1	Run 2	Run 3	Changed answer?
Clef	2.0062	2.0062	2.0062	No
Clef-flash	1.9883	1.9883	1.9883	No
Jev 1.13	2	2	2	No

Time and provider cost for this case

Clef · run 1: 935 ms · $0.00009984
Clef-flash · run 1: 525 ms · $0.00003744
Jev 1.13 · run 1: 466 ms · $0.00001982
Clef-flash · run 2: 513 ms · $0.00003744
Jev 1.13 · run 2: 332 ms · $0.00001982
Clef · run 2: 625 ms · $0.00009984
Jev 1.13 · run 3: 371 ms · $0.00001982
Clef · run 3: 519 ms · $0.00009984
Clef-flash · run 3: 422 ms · $0.00003744

Full task, reference answers and raw recorded responses

{
  "sample": {
    "state": "Everyone in our company sees incorrect invoice totals and cannot pay. Other customers are fine.",
    "questions": {
      "urgent": {
        "type": "noul",
        "instructions": "Is this a service-wide incident affecting all customers or all tenants? A single company or user does not count."
      },
      "category": {
        "type": "choice",
        "instructions": "Select the primary responsible team.",
        "criteria": {
          "technical": "Software, availability, access or documentation issue",
          "billing": "Charges, invoices, receipts or payments",
          "sales": "Quotes, pricing before purchase or demonstrations"
        }
      },
      "severity": {
        "type": "score",
        "instructions": "Rate the stated impact, using the highest applicable level.",
        "criteria": [
          "Information request, cosmetic documentation issue, or no impact",
          "One person affected or a minor issue with a workaround",
          "An entire company or department affected, not all customers",
          "All customers affected by an outage, lost access, or systemic erroneous charges"
        ]
      }
    }
  },
  "reference": {
    "urgent": false,
    "category": "billing",
    "severity": 2
  },
  "runs": [
    {
      "caseId": "support-20",
      "repeat": 1,
      "outcome": {
        "model": "clef",
        "measuredAt": "2026-10-08T09:26:22.248Z",
        "durationMs": 935,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451582-Bi5yWPfNzOvFnTVqnEDz",
          "model": "cloudflare/clef",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.0163
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0.0367,
                "billing": 0.9564,
                "sales": 0.0069
              },
              "confidence": 0.8743
            },
            "severity": {
              "type": "score",
              "score": 2.0062,
              "probabilities": {
                "0": 0.0043,
                "1": 0.0089,
                "2": 0.9631,
                "3": 0.0237
              },
              "confidence": 0.9042,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 416,
            "output_tokens": 0,
            "cost": 0.00009984
          }
        }
      }
    },
    {
      "caseId": "support-20",
      "repeat": 1,
      "outcome": {
        "model": "clef-flash",
        "measuredAt": "2026-10-08T09:26:23.192Z",
        "durationMs": 525,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451583-CoIC6IGSr3IhM0Fkh4oA",
          "model": "cloudflare/clef-flash",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.1063
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0.0869,
                "billing": 0.8551,
                "sales": 0.058
              },
              "confidence": 0.6131
            },
            "severity": {
              "type": "score",
              "score": 1.9883,
              "probabilities": {
                "0": 0.0119,
                "1": 0.0213,
                "2": 0.9334,
                "3": 0.0334
              },
              "confidence": 0.8307,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 416,
            "output_tokens": 0,
            "cost": 0.00003744
          }
        }
      }
    },
    {
      "caseId": "support-20",
      "repeat": 1,
      "outcome": {
        "model": "jev",
        "measuredAt": "2026-10-08T09:26:23.726Z",
        "durationMs": 466,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451583-PTpk2MqD18m38wavFWnT",
          "model": "typesafe/jev-1.13-20260917",
          "provider": "TypeSafe",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.06
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0.15,
                "billing": 0.85,
                "sales": 0
              },
              "confidence": 0.77
            },
            "severity": {
              "type": "score",
              "score": 2,
              "probabilities": {
                "0": 0,
                "1": 0,
                "2": 1,
                "3": 0
              },
              "confidence": 1,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 472,
            "output_tokens": 68,
            "cost": 0.000019824
          }
        }
      }
    },
    {
      "caseId": "support-20",
      "repeat": 2,
      "outcome": {
        "model": "clef-flash",
        "measuredAt": "2026-10-08T09:27:53.610Z",
        "durationMs": 513,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451673-4gdLoSKCk9ihA9RqCVOE",
          "model": "cloudflare/clef-flash",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.1063
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0.0869,
                "billing": 0.8551,
                "sales": 0.058
              },
              "confidence": 0.6131
            },
            "severity": {
              "type": "score",
              "score": 1.9883,
              "probabilities": {
                "0": 0.0119,
                "1": 0.0213,
                "2": 0.9334,
                "3": 0.0334
              },
              "confidence": 0.8307,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 416,
            "output_tokens": 0,
            "cost": 0.00003744
          }
        }
      }
    },
    {
      "caseId": "support-20",
      "repeat": 2,
      "outcome": {
        "model": "jev",
        "measuredAt": "2026-10-08T09:27:54.132Z",
        "durationMs": 332,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451674-waVfbkU0yfXyox9zcqM4",
          "model": "typesafe/jev-1.13-20260917",
          "provider": "TypeSafe",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.05
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0.15,
                "billing": 0.85,
                "sales": 0
              },
              "confidence": 0.77
            },
            "severity": {
              "type": "score",
              "score": 2,
              "probabilities": {
                "0": 0,
                "1": 0,
                "2": 1,
                "3": 0
              },
              "confidence": 1,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 472,
            "output_tokens": 68,
            "cost": 0.000019824
          }
        }
      }
    },
    {
      "caseId": "support-20",
      "repeat": 2,
      "outcome": {
        "model": "clef",
        "measuredAt": "2026-10-08T09:27:54.475Z",
        "durationMs": 625,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451674-94M7Avr2wAwRbB4qSwLZ",
          "model": "cloudflare/clef",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.0163
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0.0367,
                "billing": 0.9564,
                "sales": 0.0069
              },
              "confidence": 0.8743
            },
            "severity": {
              "type": "score",
              "score": 2.0062,
              "probabilities": {
                "0": 0.0043,
                "1": 0.0089,
                "2": 0.9631,
                "3": 0.0237
              },
              "confidence": 0.9042,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 416,
            "output_tokens": 0,
            "cost": 0.00009984
          }
        }
      }
    },
    {
      "caseId": "support-20",
      "repeat": 3,
      "outcome": {
        "model": "jev",
        "measuredAt": "2026-10-08T09:29:25.487Z",
        "durationMs": 371,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451765-hG0Umj7K61ROhhL5mcYy",
          "model": "typesafe/jev-1.13-20260917",
          "provider": "TypeSafe",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.06
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0.14,
                "sales": 0,
                "billing": 0.86
              },
              "confidence": 0.79
            },
            "severity": {
              "type": "score",
              "score": 2,
              "probabilities": {
                "0": 0,
                "1": 0,
                "2": 1,
                "3": 0
              },
              "confidence": 1,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 472,
            "output_tokens": 68,
            "cost": 0.000019824
          }
        }
      }
    },
    {
      "caseId": "support-20",
      "repeat": 3,
      "outcome": {
        "model": "clef",
        "measuredAt": "2026-10-08T09:29:25.873Z",
        "durationMs": 519,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451765-hmaqdkx7BtdaauuKvIkQ",
          "model": "cloudflare/clef",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.0163
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0.0367,
                "billing": 0.9564,
                "sales": 0.0069
              },
              "confidence": 0.8743
            },
            "severity": {
              "type": "score",
              "score": 2.0062,
              "probabilities": {
                "0": 0.0043,
                "1": 0.0089,
                "2": 0.9631,
                "3": 0.0237
              },
              "confidence": 0.9042,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 416,
            "output_tokens": 0,
            "cost": 0.00009984
          }
        }
      }
    },
    {
      "caseId": "support-20",
      "repeat": 3,
      "outcome": {
        "model": "clef-flash",
        "measuredAt": "2026-10-08T09:29:26.409Z",
        "durationMs": 422,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451766-4KgfFhKzORNRijjHhrCv",
          "model": "cloudflare/clef-flash",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.1063
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0.0869,
                "billing": 0.8551,
                "sales": 0.058
              },
              "confidence": 0.6131
            },
            "severity": {
              "type": "score",
              "score": 1.9883,
              "probabilities": {
                "0": 0.0119,
                "1": 0.0213,
                "2": 0.9334,
                "3": 0.0334
              },
              "confidence": 0.8307,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 416,
            "output_tokens": 0,
            "cost": 0.00003744
          }
        }
      }
    }
  ]
}

Download the complete original results