HISTORICAL RECORD · 2026-10-08 · support-09

There is a small duplicate charge on my account; work is unaffected.

Synthetic case. References were set by ClefBench and have not received an independent second review. These are recorded results, not a new model call.

urgent

Is this a service-wide incident affecting all customers or all tenants? A single company or user does not count.

Reference: false

Model	Run 1	Run 2	Run 3	Changed answer?
Clef	false	false	false	No
Clef-flash	false	false	false	No
Jev 1.13	false	false	false	No

Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.

Model	Run 1	Run 2	Run 3	Changed answer?
Clef	billing	billing	billing	No
Clef-flash	billing	billing	billing	No
Jev 1.13	billing	billing	billing	No

severity

Rate the stated impact, using the highest applicable level.

Reference: 1

0: Information request, cosmetic documentation issue, or no impact
1: One person affected or a minor issue with a workaround
2: An entire company or department affected, not all customers
3: All customers affected by an outage, lost access, or systemic erroneous charges

Model	Run 1	Run 2	Run 3	Changed answer?
Clef	0.8514	0.8514	0.8514	No
Clef-flash	0.9827	0.9827	0.9827	No
Jev 1.13	0.91	0.91	0.92	No

Time and provider cost for this case

Clef · run 1: 956 ms · $0.00009912
Clef-flash · run 1: 407 ms · $0.00003717
Jev 1.13 · run 1: 347 ms · $0.00001970
Clef-flash · run 2: 422 ms · $0.00003717
Jev 1.13 · run 2: 365 ms · $0.00001970
Clef · run 2: 707 ms · $0.00009912
Jev 1.13 · run 3: 400 ms · $0.00001970
Clef · run 3: 495 ms · $0.00009912
Clef-flash · run 3: 426 ms · $0.00003717

Full task, reference answers and raw recorded responses

{
  "sample": {
    "state": "There is a small duplicate charge on my account; work is unaffected.",
    "questions": {
      "urgent": {
        "type": "noul",
        "instructions": "Is this a service-wide incident affecting all customers or all tenants? A single company or user does not count."
      },
      "category": {
        "type": "choice",
        "instructions": "Select the primary responsible team.",
        "criteria": {
          "technical": "Software, availability, access or documentation issue",
          "billing": "Charges, invoices, receipts or payments",
          "sales": "Quotes, pricing before purchase or demonstrations"
        }
      },
      "severity": {
        "type": "score",
        "instructions": "Rate the stated impact, using the highest applicable level.",
        "criteria": [
          "Information request, cosmetic documentation issue, or no impact",
          "One person affected or a minor issue with a workaround",
          "An entire company or department affected, not all customers",
          "All customers affected by an outage, lost access, or systemic erroneous charges"
        ]
      }
    }
  },
  "reference": {
    "urgent": false,
    "category": "billing",
    "severity": 1
  },
  "runs": [
    {
      "caseId": "support-09",
      "repeat": 1,
      "outcome": {
        "model": "clef",
        "measuredAt": "2026-10-08T09:26:05.519Z",
        "durationMs": 956,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451565-xpGN3tgGbJGhR92do9Yk",
          "model": "cloudflare/clef",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.0066
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0.0342,
                "billing": 0.9541,
                "sales": 0.0117
              },
              "confidence": 0.8673
            },
            "severity": {
              "type": "score",
              "score": 0.8514,
              "probabilities": {
                "0": 0.1824,
                "1": 0.7932,
                "2": 0.0151,
                "3": 0.0093
              },
              "confidence": 0.5504,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 413,
            "output_tokens": 0,
            "cost": 0.00009912
          }
        }
      }
    },
    {
      "caseId": "support-09",
      "repeat": 1,
      "outcome": {
        "model": "clef-flash",
        "measuredAt": "2026-10-08T09:26:06.483Z",
        "durationMs": 407,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451566-e3DbnEmz5F7K5Wp3oPWy",
          "model": "cloudflare/clef-flash",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.0242
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0.0424,
                "billing": 0.9378,
                "sales": 0.0198
              },
              "confidence": 0.8225
            },
            "severity": {
              "type": "score",
              "score": 0.9827,
              "probabilities": {
                "0": 0.1279,
                "1": 0.805,
                "2": 0.0236,
                "3": 0.0435
              },
              "confidence": 0.5558,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 413,
            "output_tokens": 0,
            "cost": 0.00003717
          }
        }
      }
    },
    {
      "caseId": "support-09",
      "repeat": 1,
      "outcome": {
        "model": "jev",
        "measuredAt": "2026-10-08T09:26:06.898Z",
        "durationMs": 347,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451566-zWexdto78sQG7hAxzUxB",
          "model": "typesafe/jev-1.13-20260917",
          "provider": "TypeSafe",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.06
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0,
                "billing": 1,
                "sales": 0
              },
              "confidence": 1
            },
            "severity": {
              "type": "score",
              "score": 0.91,
              "probabilities": {
                "0": 0.09,
                "1": 0.91,
                "2": 0,
                "3": 0
              },
              "confidence": 0.91,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 469,
            "output_tokens": 68,
            "cost": 0.000019698
          }
        }
      }
    },
    {
      "caseId": "support-09",
      "repeat": 2,
      "outcome": {
        "model": "clef-flash",
        "measuredAt": "2026-10-08T09:27:37.660Z",
        "durationMs": 422,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451657-mCT40PsVD7O8CsJv6MIl",
          "model": "cloudflare/clef-flash",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.0242
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0.0424,
                "billing": 0.9378,
                "sales": 0.0198
              },
              "confidence": 0.8225
            },
            "severity": {
              "type": "score",
              "score": 0.9827,
              "probabilities": {
                "0": 0.1279,
                "1": 0.805,
                "2": 0.0236,
                "3": 0.0435
              },
              "confidence": 0.5558,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 413,
            "output_tokens": 0,
            "cost": 0.00003717
          }
        }
      }
    },
    {
      "caseId": "support-09",
      "repeat": 2,
      "outcome": {
        "model": "jev",
        "measuredAt": "2026-10-08T09:27:38.094Z",
        "durationMs": 365,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451658-4z4HUPXRgSRSWGucdyeV",
          "model": "typesafe/jev-1.13-20260917",
          "provider": "TypeSafe",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.06
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "billing": 1,
                "sales": 0,
                "technical": 0
              },
              "confidence": 1
            },
            "severity": {
              "type": "score",
              "score": 0.91,
              "probabilities": {
                "0": 0.1,
                "1": 0.9,
                "2": 0,
                "3": 0
              },
              "confidence": 0.9,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 469,
            "output_tokens": 68,
            "cost": 0.000019698
          }
        }
      }
    },
    {
      "caseId": "support-09",
      "repeat": 2,
      "outcome": {
        "model": "clef",
        "measuredAt": "2026-10-08T09:27:38.471Z",
        "durationMs": 707,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451658-1DWbZ3ubrxjpmaZqSGBw",
          "model": "cloudflare/clef",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.0066
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0.0342,
                "billing": 0.9541,
                "sales": 0.0117
              },
              "confidence": 0.8673
            },
            "severity": {
              "type": "score",
              "score": 0.8514,
              "probabilities": {
                "0": 0.1824,
                "1": 0.7932,
                "2": 0.0151,
                "3": 0.0093
              },
              "confidence": 0.5504,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 413,
            "output_tokens": 0,
            "cost": 0.00009912
          }
        }
      }
    },
    {
      "caseId": "support-09",
      "repeat": 3,
      "outcome": {
        "model": "jev",
        "measuredAt": "2026-10-08T09:29:09.213Z",
        "durationMs": 400,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451749-u7AUCa7hpzFNFyXCkJ87",
          "model": "typesafe/jev-1.13-20260917",
          "provider": "TypeSafe",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.06
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0,
                "sales": 0,
                "billing": 1
              },
              "confidence": 1
            },
            "severity": {
              "type": "score",
              "score": 0.92,
              "probabilities": {
                "0": 0.08,
                "1": 0.92,
                "2": 0,
                "3": 0
              },
              "confidence": 0.92,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 469,
            "output_tokens": 68,
            "cost": 0.000019698
          }
        }
      }
    },
    {
      "caseId": "support-09",
      "repeat": 3,
      "outcome": {
        "model": "clef",
        "measuredAt": "2026-10-08T09:29:09.629Z",
        "durationMs": 495,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451749-wWFN0LDb6zz1AsaW3d8q",
          "model": "cloudflare/clef",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.0066
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0.0342,
                "billing": 0.9541,
                "sales": 0.0117
              },
              "confidence": 0.8673
            },
            "severity": {
              "type": "score",
              "score": 0.8514,
              "probabilities": {
                "0": 0.1824,
                "1": 0.7932,
                "2": 0.0151,
                "3": 0.0093
              },
              "confidence": 0.5504,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 413,
            "output_tokens": 0,
            "cost": 0.00009912
          }
        }
      }
    },
    {
      "caseId": "support-09",
      "repeat": 3,
      "outcome": {
        "model": "clef-flash",
        "measuredAt": "2026-10-08T09:29:10.139Z",
        "durationMs": 426,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451750-sWNHZnykylcvaH9VkoNe",
          "model": "cloudflare/clef-flash",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.0242
            },
            "category": {
              "type": "choice",
              "choice": "billing",
              "probabilities": {
                "technical": 0.0424,
                "billing": 0.9378,
                "sales": 0.0198
              },
              "confidence": 0.8225
            },
            "severity": {
              "type": "score",
              "score": 0.9827,
              "probabilities": {
                "0": 0.1279,
                "1": 0.805,
                "2": 0.0236,
                "3": 0.0435
              },
              "confidence": 0.5558,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 413,
            "output_tokens": 0,
            "cost": 0.00003717
          }
        }
      }
    }
  ]
}

Download the complete original results