HISTORICAL RECORD · 2026-10-08 · support-01

Our production API is down for every customer

Our production API is down for every customer. No requests succeed.

Synthetic case. References were set by ClefBench and have not received an independent second review. These are recorded results, not a new model call.

urgent

Is this a service-wide incident affecting all customers or all tenants? A single company or user does not count.

Reference: true

Model	Run 1	Run 2	Run 3	Changed answer?
Clef	true	true	true	No
Clef-flash	true	true	true	No
Jev 1.13	true	true	true	No

Yes/no uses a 50% threshold. Score values are weighted scale indices; error flags and stability use the nearest level. Use the rule and full distribution to judge whether a difference matters for your task.

Model	Run 1	Run 2	Run 3	Changed answer?
Clef	technical	technical	technical	No
Clef-flash	technical	technical	technical	No
Jev 1.13	technical	technical	technical	No

severity

Rate the stated impact, using the highest applicable level.

Reference: 3

0: Information request, cosmetic documentation issue, or no impact
1: One person affected or a minor issue with a workaround
2: An entire company or department affected, not all customers
3: All customers affected by an outage, lost access, or systemic erroneous charges

Model	Run 1	Run 2	Run 3	Changed answer?
Clef	2.9465	2.9465	2.9465	No
Clef-flash	2.8898	2.8898	2.8898	No
Jev 1.13	3	3	3	No

Time and provider cost for this case

Clef · run 1: 1306 ms · $0.00009888
Clef-flash · run 1: 494 ms · $0.00003708
Jev 1.13 · run 1: 343 ms · $0.00001966
Clef-flash · run 2: 419 ms · $0.00003708
Jev 1.13 · run 2: 389 ms · $0.00001966
Clef · run 2: 718 ms · $0.00009888
Jev 1.13 · run 3: 431 ms · $0.00001966
Clef · run 3: 637 ms · $0.00009888
Clef-flash · run 3: 399 ms · $0.00003708

Full task, reference answers and raw recorded responses

{
  "sample": {
    "state": "Our production API is down for every customer. No requests succeed.",
    "questions": {
      "urgent": {
        "type": "noul",
        "instructions": "Is this a service-wide incident affecting all customers or all tenants? A single company or user does not count."
      },
      "category": {
        "type": "choice",
        "instructions": "Select the primary responsible team.",
        "criteria": {
          "technical": "Software, availability, access or documentation issue",
          "billing": "Charges, invoices, receipts or payments",
          "sales": "Quotes, pricing before purchase or demonstrations"
        }
      },
      "severity": {
        "type": "score",
        "instructions": "Rate the stated impact, using the highest applicable level.",
        "criteria": [
          "Information request, cosmetic documentation issue, or no impact",
          "One person affected or a minor issue with a workaround",
          "An entire company or department affected, not all customers",
          "All customers affected by an outage, lost access, or systemic erroneous charges"
        ]
      }
    }
  },
  "reference": {
    "urgent": true,
    "category": "technical",
    "severity": 3
  },
  "runs": [
    {
      "caseId": "support-01",
      "repeat": 1,
      "outcome": {
        "model": "clef",
        "measuredAt": "2026-10-08T09:25:52.116Z",
        "durationMs": 1306,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451552-G1FOqTBtiV0c8syvQtq6",
          "model": "cloudflare/clef",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.9931
            },
            "category": {
              "type": "choice",
              "choice": "technical",
              "probabilities": {
                "technical": 0.9599,
                "billing": 0.0216,
                "sales": 0.0185
              },
              "confidence": 0.8832
            },
            "severity": {
              "type": "score",
              "score": 2.9465,
              "probabilities": {
                "0": 0.0066,
                "1": 0.0067,
                "2": 0.0202,
                "3": 0.9665
              },
              "confidence": 0.9127,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 412,
            "output_tokens": 0,
            "cost": 0.00009888
          }
        }
      }
    },
    {
      "caseId": "support-01",
      "repeat": 1,
      "outcome": {
        "model": "clef-flash",
        "measuredAt": "2026-10-08T09:25:53.429Z",
        "durationMs": 494,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451553-wtFTqL5SqlEP65oiAe0L",
          "model": "cloudflare/clef-flash",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.9444
            },
            "category": {
              "type": "choice",
              "choice": "technical",
              "probabilities": {
                "technical": 0.928,
                "billing": 0.0378,
                "sales": 0.0342
              },
              "confidence": 0.7958
            },
            "severity": {
              "type": "score",
              "score": 2.8898,
              "probabilities": {
                "0": 0.0132,
                "1": 0.0152,
                "2": 0.0402,
                "3": 0.9314
              },
              "confidence": 0.826,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 412,
            "output_tokens": 0,
            "cost": 0.00003708
          }
        }
      }
    },
    {
      "caseId": "support-01",
      "repeat": 1,
      "outcome": {
        "model": "jev",
        "measuredAt": "2026-10-08T09:25:53.929Z",
        "durationMs": 343,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451553-wMGwdHHtW0ll9zykJiwJ",
          "model": "typesafe/jev-1.13-20260917",
          "provider": "TypeSafe",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.95
            },
            "category": {
              "type": "choice",
              "choice": "technical",
              "probabilities": {
                "billing": 0,
                "sales": 0,
                "technical": 1
              },
              "confidence": 1
            },
            "severity": {
              "type": "score",
              "score": 3,
              "probabilities": {
                "0": 0,
                "1": 0,
                "2": 0,
                "3": 1
              },
              "confidence": 1,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 468,
            "output_tokens": 68,
            "cost": 0.000019656
          }
        }
      }
    },
    {
      "caseId": "support-01",
      "repeat": 2,
      "outcome": {
        "model": "clef-flash",
        "measuredAt": "2026-10-08T09:27:24.924Z",
        "durationMs": 419,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451644-tKvENONKWmjd27ZSp4aw",
          "model": "cloudflare/clef-flash",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.9444
            },
            "category": {
              "type": "choice",
              "choice": "technical",
              "probabilities": {
                "technical": 0.928,
                "billing": 0.0378,
                "sales": 0.0342
              },
              "confidence": 0.7958
            },
            "severity": {
              "type": "score",
              "score": 2.8898,
              "probabilities": {
                "0": 0.0132,
                "1": 0.0152,
                "2": 0.0402,
                "3": 0.9314
              },
              "confidence": 0.826,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 412,
            "output_tokens": 0,
            "cost": 0.00003708
          }
        }
      }
    },
    {
      "caseId": "support-01",
      "repeat": 2,
      "outcome": {
        "model": "jev",
        "measuredAt": "2026-10-08T09:27:25.355Z",
        "durationMs": 389,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451645-Lziwes4SLuUwiX7acamO",
          "model": "typesafe/jev-1.13-20260917",
          "provider": "TypeSafe",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.96
            },
            "category": {
              "type": "choice",
              "choice": "technical",
              "probabilities": {
                "technical": 1,
                "billing": 0,
                "sales": 0
              },
              "confidence": 1
            },
            "severity": {
              "type": "score",
              "score": 3,
              "probabilities": {
                "0": 0,
                "1": 0,
                "2": 0,
                "3": 1
              },
              "confidence": 1,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 468,
            "output_tokens": 68,
            "cost": 0.000019656
          }
        }
      }
    },
    {
      "caseId": "support-01",
      "repeat": 2,
      "outcome": {
        "model": "clef",
        "measuredAt": "2026-10-08T09:27:25.756Z",
        "durationMs": 718,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451645-CI9qm0BXsHpeenGWaeqy",
          "model": "cloudflare/clef",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.9931
            },
            "category": {
              "type": "choice",
              "choice": "technical",
              "probabilities": {
                "technical": 0.9599,
                "billing": 0.0216,
                "sales": 0.0185
              },
              "confidence": 0.8832
            },
            "severity": {
              "type": "score",
              "score": 2.9465,
              "probabilities": {
                "0": 0.0066,
                "1": 0.0067,
                "2": 0.0202,
                "3": 0.9665
              },
              "confidence": 0.9127,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 412,
            "output_tokens": 0,
            "cost": 0.00009888
          }
        }
      }
    },
    {
      "caseId": "support-01",
      "repeat": 3,
      "outcome": {
        "model": "jev",
        "measuredAt": "2026-10-08T09:28:57.397Z",
        "durationMs": 431,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451737-H75xVUK0Q2bprFjxDk70",
          "model": "typesafe/jev-1.13-20260917",
          "provider": "TypeSafe",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.96
            },
            "category": {
              "type": "choice",
              "choice": "technical",
              "probabilities": {
                "billing": 0,
                "sales": 0,
                "technical": 1
              },
              "confidence": 1
            },
            "severity": {
              "type": "score",
              "score": 3,
              "probabilities": {
                "0": 0,
                "1": 0,
                "2": 0,
                "3": 1
              },
              "confidence": 1,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 468,
            "output_tokens": 68,
            "cost": 0.000019656
          }
        }
      }
    },
    {
      "caseId": "support-01",
      "repeat": 3,
      "outcome": {
        "model": "clef",
        "measuredAt": "2026-10-08T09:28:57.843Z",
        "durationMs": 637,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451737-oE7EAptR9oWeMxcuwG1X",
          "model": "cloudflare/clef",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.9931
            },
            "category": {
              "type": "choice",
              "choice": "technical",
              "probabilities": {
                "technical": 0.9599,
                "billing": 0.0216,
                "sales": 0.0185
              },
              "confidence": 0.8832
            },
            "severity": {
              "type": "score",
              "score": 2.9465,
              "probabilities": {
                "0": 0.0066,
                "1": 0.0067,
                "2": 0.0202,
                "3": 0.9665
              },
              "confidence": 0.9127,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 412,
            "output_tokens": 0,
            "cost": 0.00009888
          }
        }
      }
    },
    {
      "caseId": "support-01",
      "repeat": 3,
      "outcome": {
        "model": "clef-flash",
        "measuredAt": "2026-10-08T09:28:58.504Z",
        "durationMs": 399,
        "status": "success",
        "result": {
          "id": "gen-dec-1791451738-99YeJKras2HN8WQZsrGA",
          "model": "cloudflare/clef-flash",
          "provider": "Cloudflare",
          "answers": {
            "urgent": {
              "type": "noul",
              "noul": 0.9444
            },
            "category": {
              "type": "choice",
              "choice": "technical",
              "probabilities": {
                "technical": 0.928,
                "billing": 0.0378,
                "sales": 0.0342
              },
              "confidence": 0.7958
            },
            "severity": {
              "type": "score",
              "score": 2.8898,
              "probabilities": {
                "0": 0.0132,
                "1": 0.0152,
                "2": 0.0402,
                "3": 0.9314
              },
              "confidence": 0.826,
              "legend": {
                "0": "Information request, cosmetic documentation issue, or no impact",
                "1": "One person affected or a minor issue with a workaround",
                "2": "An entire company or department affected, not all customers",
                "3": "All customers affected by an outage, lost access, or systemic erroneous charges"
              }
            }
          },
          "usage": {
            "input_tokens": 412,
            "output_tokens": 0,
            "cost": 0.00003708
          }
        }
      }
    }
  ]
}

Download the complete original results