How we train our Jev-like Decision models

Most automation work is a choice from a fixed list, such as labeling an item, classifying a document or routing a request. When the options are known in advance, we don't need a generative model that writes its answer token by token; we need a decision model, which reads the prompt once and gives every option a probability, so it answers faster and costs less.

In this article we walk you through how we first trained Logits Sigma-1, fine-tuning Gemma 4 31B: how it reads a question, what we trained it on, how we picked the model, and how we serve it on one GPU. Then we show how we built Sigma-1 Duo, a mixture of two models with one of our custom adapters, which shares second place among 300 entries on the Jev Decision Index, and what we learned.

tl;dr

  • Sigma-1, the first model in the family, is Gemma 4 31B with one rank-64 LoRA adapter, trained on a set of 476,186 rows.
  • A decision model like Sigma-1 doesn't write its answer. It reads it from the probabilities it gives each option letter, so it answers in one pass, as fast as the fastest 31B entries on the board.
  • Training scores each answer three ways: whether it's wrong, whether the model has drifted from plain Gemma, and whether its confidence is the right size.
  • We compared 20 training runs of Sigma-1 and picked one on tasks none of them had trained on.
  • Merging the adapter at 0.75 instead of 1.0, so the model keeps a quarter of plain Gemma, made it better on decision tasks from new domains.
  • On the Jev Decision Index, Sigma-1 scores 69.47, and Sigma-1 Duo, a 50/50 mixture of two models, one of them carrying a custom adapter we trained, scores 72.28 and shares second place among 300 entries.

How a Jev decision model works

Answering with a probability for each option, instead of writing text, is an old idea: MMLU has scored multiple-choice answers this way since 2020. What Jev adds is the request format, typed questions about one shared state.

A Jev request has two parts: a state and a list of questions about it. The state is the case itself, as text or JSON, such as a support ticket, an invoice or a log line. Every question is typed. A choice question offers 2 to 255 options, each with a key and a short description, and the model answers it with a probability for every option, summing to 1, plus the key it picks. A yes/no question gets back the probability of yes (Figure 1). One state can carry many questions, up to 64 in one benchmark, and in ranking tasks the order of those probabilities is the ranking.

FigureOne decision requestillustration

The Jev Decision Index ranks open-weight models that answer this format. Each model runs a public suite of 37 benchmarks, but its rank comes from private eval sets that only the maintainers see. 60% of that score comes from fresh items that test the public benchmarks' skills, and 40% from decision tasks in new domains.

The model

Sigma-1, our first model, uses Google's Gemma 4 31B as its base. To specialise it for decisions, we trained one LoRA adapter, rank 64, on the attention and MLP projections of every layer of the language model, and merged it into the weights at a scale of 0.75. The released model is 0.75 × fine-tuned + 0.25 × base.

Sigma-1 gets one prompt per question. The prompt has four parts: an instruction, the list of every question in the request, the state, and last the question itself, with its options labelled A, B, C and so on. The answer is the model's next-token probabilities over those letters (Figure 2).

Every question's prompt starts with the same first three parts. When they are long, the engine computes them once per request and reuses them for each question. When that shared block is 3,000 tokens or fewer, we write it twice and tell the model to "read the questions and the state again". The idea comes from RE2; PelaAI's KnowLine Gen1.5 does the same with the same 3,000-token limit.

FigureHow Sigma-1 reads a questionillustration

Because Sigma-1 models read their answer instead of writing it, serving is a single pass over the prompt, with no tokens to generate. For Sigma-1, with an FP8 KV cache, that fits a 31B model and prompts of up to 131,072 tokens on one 96 GB GPU.

Training data

Sigma-1's training set has 476,186 rows, 480M tokens in all, drawn from six kinds of source; the run covered about 78% of it (1,500 of an epoch's 1,911 steps). Before training, we removed any row that matched a public evaluation item exactly or shared a run of 13 words with one. Figure 3 shows how the sources add up.

  • Public decision data: about 152,000 rows from tasksource's jev-typed-decisions collection, taken from the 411 sources tagged for commercial use.
  • Other public datasets for tool use (glaive, Synth-APIGen, ToolACE and xLAM), knowledge questions (MedMCQA, AQuA-RAT and others) and ranking (WANDS, DBpedia-Entity and others).
  • Public training splits: 38,278 rows (8.0%) from the training sets of public datasets such as BANKING77, CLINC150, ARC and GSM8K, put into the Jev request format.
  • Synthetic tasks we generated, covering aspect sentiment, chords, phishing emails and a smart-home simulator. None of them copies a public evaluation item.
  • Public records from NHTSA complaints, OSHA reports, federal court outcomes, openFDA, CVE, USPTO and public Jira trackers, turned into questions whose answer follows from the record.
  • Teacher rows: 64,862 rows (13.6%) written and labelled by teacher models, larger models whose answers become training labels. In a pipeline on our own inference API, Kimi K3.1 Pro designs the task families, DeepSeek V4.1 Flash writes the states, and both vote on the labels. Each state spells out every rule needed to answer it, so a label can be checked by reading the row.
FigureSigma-1's training datarows

The loss

Our loss adds three terms, each catching a different way to be wrong. Take one row whose right answer is A, where the model gives A 0.72, B 0.20, C 0.06 and D 0.02 (Figure 4):

  • A wrong answer (soft cross-entropy). The cost is minus the log of the probability on the right answer: −ln 0.72 = 0.33. At 0.99 it would cost 0.01; at 0.10, 2.30. When the label spreads its probability over several options, as annotators or teacher models sometimes do, every option is scored this way, weighted by the label.
  • Drifting from the base model (KL to untuned Gemma, weight 1.0). The same row goes through untuned Gemma, which answers 0.55, 0.30, 0.10 and 0.05. The KL divergence measures how far apart the two answers are, here 0.07. It keeps the fine-tune close to the model it started from.
  • Confidence of the wrong size (Brier, weight 0.3). The Brier score adds up the squared gap between each probability and the label, 1 for A and 0 for the rest: 0.28² + 0.20² + 0.06² + 0.02² = 0.12. It rewards probabilities that can be read as real chances, which is what calibration means. At weight 0.3 it adds 0.04.

The loss for this row is 0.33 + 0.07 + 0.04 = 0.44.

FigureScoring one answerworked example

The term weights and the learning rate, 1e-4, follow the model card of another entrant, FlexDM-LoRA 2.1. The run took about ten hours on 8 rented H200 GPUs.

How we ran the experiments

We ran the work in rounds: 20 training runs in all, 13 on rented H200 machines and 7 small QLoRA runs on one RTX 5090, plus several more models we built by merging. Coding agents ran them for us through a small research harness. We kept a queue so no GPU sat idle, and logged every model, run, evaluation and verdict in git, 452 entries by the end. Before spending on an idea, we tested it cheaply. We started with data checks on a CPU, then a QLoRA run on one GPU or a short run of 1 to 4 hours, and only the ideas that held up got a full run on eight H200s.

We compared the models on four kinds of evidence:

  1. A public mini-suite: 14,956 rows sampled from the public benchmarks, 10.7% of the suite's requests. On 19 other entrants, its score had an RMSE of 0.21 points against their full runs, with a largest miss of 0.45.
  2. Tasks outside our training data: gains on the tasks our data covered didn't always carry over to others, so from the second round on we selected on 11 public benchmarks that our data didn't cover.
  3. New-domain sets we built: 40% of the board's score comes from new domains, which the public suite doesn't cover, so we built question families from public data that none of our training touches. Our first set, with weather reports, earthquakes, bank records, air quality and more, turned out too easy; a harder second set did the work.
  4. Held-out teacher families: families from our teacher pipeline that never entered training.

We compared every new model against the best one we had and against stock Gemma, on the same items, and counted a difference only if it held up across 2,000 bootstrap resamples of those items. From the second round on, we wrote down the rule for choosing between two models before we saw the results. Our harness looked at the tasks our data did not cover, our new-domain sets and the held-out teacher families, and used the tasks our data did cover only to catch overfitting. The harness applied that rule itself, and we decided only what to spend and when to change the rule.

Once we had a winner, it went through three last checks before we released it as Sigma-1. Its training rows had to share nothing with the public benchmarks, and audits had to find its labels at least 98% accurate. Its served version also had to match the model we had tested on at least 98.5% of decisions, and it matched on 99.16%.

Sigma-1 Duo

Sigma-1 Duo, the mixture in the family, runs two models and gives each option the average of their two probabilities. One is PelaAI's KnowLine-31B-Gen1, with an adapter from our second round of runs merged in at 0.5. The other is d3-ultra from the vLLM Semantic Router team, with its weights converted to FP8.

What we learned

  1. Test on tasks you did not train for. One of our early recipes beat stock Gemma on the tasks its data covered, but it lost on the others: on the 8 of 17 we could compare, it scored 2.71 points lower.
  2. Teacher data teaches the teacher's habits. When we trained only on rows our teacher models wrote, the model got 4.8 points better on new tasks written by the same teachers. On public tasks outside its data, it got no better.
  3. Keep some of the base model. We merged the adapter at 0.75 instead of 1.0, so the released model keeps a quarter of stock Gemma, and that made it better on decision tasks from new domains.

Try Logits Sigma-1

sign up

To train a decision model for your own task, book a demo.