Ticket classification kept coming up as a use case for System 1 models, so I tried it myself. Jev is TypeSafe’s decision model: you send context and typed questions, and it returns structured answers your code can use. For a ticket, that could mean choosing a destination from a supplied list; your code handles the routing.
I chose Laya, an open-weight alternative for the same kind of constrained decision, and ran a small, offline incident-routing experiment. On my initial 24 historical incidents, its zero-shot predictions matched the recorded final destination in four cases.
That result ruled out autonomous routing with this setup. It also gave me a concrete follow-up: test whether better incident context changed the decisions.
Why Laya looked worth testing
“System 1” borrows the language of fast, intuitive decisions. Here, I’m using it for models designed to return constrained decisions rather than generate an extended answer, not as a guarantee of human-like intuition.
Laya’s model card describes a non-autoregressive model: it returns typed answers in a single forward pass rather than generating text token by token. It documents a Jev-compatible serving interface, and its examples include routing requests between departments. TypeSafe’s intent-routing documentation presents a similar application: classify incoming requests and send them to suitable handlers.
If the required output is a destination, a model built to choose one is worth investigating. This experiment tested Laya, not Jev.
Eight destinations and a review option
I ran the experiment with Python 3.12.6 and the Laya 0.3.5 package, using redacted historical incidents labeled by their final destination. The choice set was fixed: eight routing destinations plus human review.
The retrieval-augmented approach uses similar previously routed incidents as examples for a language model. In this zero-shot Laya test, I instead supplied incident text and descriptions of the possible destinations, without retrieving examples. I wanted to see whether that bounded setup showed enough signal to justify further investment.
On the initial 24 incidents:
- Historical-destination matches: 4/24, or 16.7%.
- Destination choices rather than human review: 21/24, or 87.5%.
- Matches among those destination choices: 4/21, or 19.0%.
The model usually selected a destination, but most selections disagreed with the reference label. Those figures do not support letting this setup route incidents autonomously. Any confidence-based policy for bypassing review would also need validation on this task.
What changed with comment context
A short summary can describe a broad workflow while omitting the symptom that distinguishes one destination from another.
Eighteen of the original incidents had an eligible early comment. I compared those same 18 records with and without comment context, keeping the model, labels, candidate choices, and criteria fixed:
| Input | Historical-destination matches |
|---|---|
| Summary fields only | 2/18 (11.1%) |
| Summary fields plus comment context | 5/18 (27.8%) |
Eleven predictions changed, with a net gain of three matches. That is worth investigating, but five matches out of eighteen is still weak.
The six incidents without eligible comments were excluded from both sides. Comparing 4/24 with 5/18 would mix different cohorts. The paired result is 2/18 versus 5/18, and this small, comment-available subset is not representative of all incoming incidents.
Why I’m not calling this a RAG comparison
Separate historical evaluations of the retrieval-augmented router reported roughly mid-50% top-1 accuracy and mid-to-high-60% top-2 hit rates. Those used different cohorts and evaluation protocols. I cannot use them to claim a controlled victory over Laya.
Historical final destinations are also reference labels, not automatically perfect ground truth. A mismatch could reflect an incorrect prediction, missing evidence, ambiguous ownership, or a questionable label. These results do not separate those explanations.
Nor does this pilot establish that System 1 models cannot classify tickets. Laya’s current documentation discusses domain fine-tuning; this experiment tested a zero-shot setup.
The next experiment
One follow-up worth considering is retrieval-assisted selection: retrieve a shortlist of plausible destinations, then ask Laya to choose among them or recommend review.
First, measure how often retrieval includes the reference destination. Then compare that combination and the existing RAG pipeline on the same frozen incidents, with audited labels, identical information cutoffs, and shared scoring rules. Report both destination-match rates and the proportion sent for review.
For now, the result is specific: this zero-shot setup was not adequate for autonomous routing. The next test should examine relevant examples and available incident evidence, rather than assume another round of prompt wording will close the gap.