hüküm benchmarks
hüküm1 and hüküm1nlp are Turkish decision models built on Turkish-Gemma-4b-T1-Scout with LoRA and a Clef decision head. They read a text and a set of typed questions and return calibrated probabilities for every option, without generating text. Below, both are compared with four other open Turkish decision models. Every model answered the same 56,756 questions with its own inference code and calibration.
Six models, same questions
Accuracy in %. Knowledge is the mean of TR-MMLU, TurkishMMLU, Global-MMLU-tr, INCLUDE-tr, EXAMS-tr and MMLU-Pro-TR. The human-gold subset has 260 questions, so differences under about 6 points are within noise.
hukum-bench, task by task
HakemBench, decision sets and sentiment
Same metric as the table above. HakemBench's group mean covers its 7 tracks; the human-gold row is reported separately and is not in the mean.
Published numbers from other models
For context only. Most of these were measured with lm-evaluation-harness log-likelihood or with free generation, where the model never sees the options side by side. A decision model sees every option before choosing, so a higher hüküm number here is not a win. HakemBench human-gold, typed-decisions, XNLI and TrGLUE use the same test sets as ours. On WinoGrande-tr our score is on a 400-question repaired dev subset and on MMLU-Pro-TR on a 1,400-question subset. Accuracy in %; ▲ marks the highest number in each column, ◆ means the set's training split was in that model's training data.
What the NLP stage changed
hüküm1nlp starts from hüküm1 and trains one more epoch on 17 Turkish NLP datasets recast as decisions, with 25% replay of the general mix. These are held-out test rows of those datasets (XQuAD-tr was never trained on).
The general decision sets lost 1–5 points in the same stage: HakemBench human-gold 67.7 → 63.1, typed-decisions 77.4 → 75.4, MMLU-Pro-TR 27.6 → 24.5. Use hüküm1nlp for Turkish text understanding and hüküm1 for general decisions.
How to read these numbers
The format
Each benchmark question becomes a decision: the question is the state, the choices become letter keys with the choice text as description, and yes/no tasks become yes/no questions. Choices keep their source order.
Not comparable with lm-eval
A decision model sees every option before choosing. Log-likelihood scores from lm-evaluation-harness do not show the alternatives, so they are not directly comparable with these.
◆ Trained on that set
The set's own training split was in that model's training data. hüküm1 also saw ~5.3k yes/no questions derived from TrGLUE training splits and 57 XCOPA validation rows, which are not marked.
Leakage screening
Questions sharing 50% or more of their 8-grams with any of our training text were dropped. Large sets are capped at 2,000 questions; XNLI, TR-MMLU and TrClaim are used in full.
Questions a model could not read
Karar-4B reads at most 62 options, so 409 jev-bench-tr and 7 cisimcik questions were skipped; ufakzeka accepts at most 10 score levels (11 cisimcik questions skipped). Those scores use the remaining questions.
Noise
One training run per model. Sets of 100–400 questions carry 3–5 points of sampling noise. ECE is each model's own calibration error, lower is better.