Comparing embedding models for related-article recommendation

chezo.uno is a personal blog built with Hugo, and the related articles at the foot of each post are written into the front matter by prelims-cli. It embeds each article and recommends the nearest neighbours by cosine similarity.

This runs on GitHub Actions, which is what sets the constraints.

• No GPU. Inference is CPU-only, so the model has to run under ONNX Runtime
• Limited memory. A standard runner has 16 GB on a public repository and 7 GB on a private one. Not just the model has to fit — so does peak memory while embedding
• It runs on every new post. Run time is time before the post is live

Under these conditions only small embedding models are available. Today there are two, one per language (ruri-v3-30m for Japanese, granite-small-english for English), switched on a language setting.

This started as curiosity about Bekko Embedding, a small multilingual model (a25m model card). It would be interesting if 8M / 25M active parameters were enough for this job — and from that came two hopes: that it might replace both current models, and that even if it did not, it might extend coverage past Japanese and English.

So both Bekko sizes, plus granite-embedding-97m-multilingual-r2 as another small multilingual model, were put on the same footing as the two current ones.

There is no ground truth for "a good related article". Instead the hand-written tags on each article act as the referee: if two articles sharing a tag get recommended to each other, the topics are taken to match. The tags were not written for recommendation, so as a referee they are at least independent.

What follows defines every number in this report. None of it is an off-the-shelf benchmark — these are proxy metrics written for this investigation — so the arithmetic is recorded. The implementation is scripts/compare_embedding_variants.py. The first metric (tag agreement) turned out to have the design fault described below, so model comparisons use the two after it: fixed-pool precision@k and pairwise AUC.

Tag agreement (any-tag@k)

Let T(a) be the tag set of article a and R_k(a) its top k recommendations. Only pairs tagged on both ends are scored; if either side is untagged, the pair does not count.

T(a) is the union of tags, categories and keywords from the front matter, lowercased. Which keys count is set by --tag-keys. Every headline number here uses tags and categories only.

This definition is faulty. The model chooses P, the set of scored pairs — it decides which articles to recommend, so the denominator differs per model. Recommending an untagged article drops out of the denominator entirely and costs nothing, so two models are not being asked the same question. What that leads to is shown in "the ranking flips with the reading". The two below are the repaired version, and every model ranking in this report is decided with them.

Fixed-pool precision@k (judged pool)

Restrict the candidates to tagged articles first, then take the top k. This is the condensed-list treatment used in information retrieval for incomplete relevance judgments. Sources are restricted to tagged articles too, so the denominator is fixed at "sources × k" and every model is made to pick the same number from the same pool. With S the set of tagged articles:

The theoretical ceiling — the highest hit count any ranking could reach — is reported alongside. An article with fewer than k same-label partners cannot fill all k slots, so for Japanese at k=3 the ceiling is 203 of 285 slots, and for English 53 of 69.

Pairwise AUC

Over all pairs of tagged articles, the probability that a same-label pair ranks as more similar than a different-label pair. 0.5 is chance. It uses the whole similarity ordering rather than a top-k cutoff, so it resolves differences that a change in hit count cannot show, and it does not depend on topk.

The implementation does not enumerate the probability; it computes the Mann-Whitney U statistic. Rank every pair by similarity (ties take the average rank) and take the rank sum over the same-label pairs:

In this form the rank sum is maximal — 1.0 — when every same-label pair sits above every other, and 0.0 when they all sit below. No pairwise comparison loop is needed, so even the 4,465 Japanese pairs come out instantly.

Both metrics report a paired bootstrap 95% interval on the difference. Precision resamples source articles; AUC resamples articles rather than pairs, because pairs sharing an endpoint are correlated and resampling them directly makes the interval too narrow. 1,000 resamples, fixed seed.

Hub concentration

Count how many articles recommend article x (its in-degree), then look at the maximum and at the share of all recommendation slots taken by the top 5 articles. N is the article count.

Lower is better. It catches an article that attaches itself everywhere regardless of topic. It sometimes moves before tag agreement does, which is why it is kept as a separate view.

Probe separation

Embed three sentences translated into three languages — two on one topic, one unrelated — and take the difference in cosine similarity. Implemented in scripts/check_onnx_model.py.

That this metric does not measure ranking quality is shown under "Is the metric valid?".

Membership changes and order-only changes

When comparing two settings, count separately whether the set of top k changed or only the order did. A reordering shows up in the front matter diff, but the reader sees the same articles.

The agreement rates below land between 0.3 and 0.5. Whether that is high or low is not determined by the number alone. To give it a scale, here is the probability that two randomly chosen articles from the same corpus share a tag — the floor you would get by recommending nothing at all.

In Japanese the observed values sit 8–9× above chance, so the metric is picking up signal. In English, chance reaches 21%. Only 23 articles are tagged and the vocabulary concentrates on python / oss / machine learning, so even a random pick often hits.

This is part of why English resolves differences so poorly: a high chance level compresses the range left for models to differ in. It is not only about article count. The int8 build of granite-97m-multilingual reaches 0.294 at topk=10, just 1.4× chance.

Since this is a proxy, what it can and cannot measure has to be settled first. Four things were checked.

The winner changes with where the tags come from

Besides the hand-written tags and categories, the front matter carries keywords extracted automatically by TF-IDF (on 405 of 420 articles, 2,350 distinct terms, the most common being こと, a semantically empty formal noun). Including those in the referee means measuring "topical agreement" as agreement with the very method the embeddings are meant to replace. Swap the referee and the winner does change.

In Japanese, bekko's +7.6pt becomes −2.3pt. The chance level does not move at all (0.047 either way), so this is not the scale stretching — the ordering itself changes. Every number in this report is measured with hand-written tags / categories only.

In English the referee makes no difference (−0.6pt and +0.2pt). The contamination shows up only in Japanese because keywords coverage differs sharply — 405/420 in Japanese against 29/33 in English — and because the hand tags cover only 95 Japanese articles. Swapping the referee inflates the scored set from 271 pairs to 2,090, and every one of those added pairs is TF-IDF derived.

Measured again with the repaired metric, the shape of the contamination is visible. Under the keywords referee, fixed-pool precision@5 puts a8m ahead by a significant +7.6pt [+5.6, +9.6], while under the same keywords referee the AUC gap is only +1.1pt [−0.4, +2.5]. a8m's advantage is concentrated in the top few results rather than spread through the ranking, which fits the explanation that articles sharing TF-IDF terms at the surface rise to the top.

Absolute values cannot be compared across languages

Chance is 0.047 in Japanese against 0.209 in English, a factor of four. 0.40 in Japanese and 0.40 in English do not carry the same meaning. Everything compared across languages here is limited to the ordering and sign of the differences between models.

Probe separation does not measure ranking quality

This metric takes the similarity gap between a same-topic pair and an unrelated sentence, over three-language probes of identical content. bekko-a25m separates less than a8m (0.173 against 0.379 in Japanese) yet recommends better. What it captures is how spread out the embedding space is, not which model ranks better.

The two current models return 0.72–0.85 even for unrelated sentences, compressing every value into a narrow band. ruri's 0.072 in French and granite-english's 0.055 in Japanese show separation all but vanishing outside their range. This metric is only good for asking "is it broken?" and "can a threshold be set?". It does work as evidence that a per-language model cannot be reused for other languages.

39 of the 69 Japanese labels and 19 of the 26 English ones appear on a single article. A label used once cannot form a pair by definition, so more than half the vocabulary contributes nothing to the metric. What is left forms only 212 same-label pairs in Japanese and 53 in English.

Set that against the total and the ceiling appears. Of the 4,465 pairs formed by the 95 tagged Japanese articles, only 212 (4.7%) share a label — that is the entire supply of "correct answers". In the fixed pool at topk=3, no ranking can exceed 203 hits out of 285 slots, and the current model takes 103 (51% of the ceiling). That is not a bad number, but with a supply this size it is hard to make a few points of hit-count difference mean anything.

English had machine_learning and machine-learning as separate labels. Tags were normalised partway through this investigation, and every number in this report was re-measured afterwards.

Resolution — how small a difference can this metric see?

The resolution of tag agreement depends on |P|, the number of recommendation pairs tagged on both ends — and |P| is small. Japanese produces 1,260 recommendations (420 articles × topk=3), yet only 174 of them, 14%, can be scored. The reason is not a low agreement rate. Agreement itself is 49%, ten times chance, so the signal is clearly there. What binds is tag coverage: 95 of 420 articles carry hand tags, and since both ends of a pair need tags, that 23% enters squared.

So does a two-point difference between models mean anything? Measured with paired bootstrap 95% intervals, the three metrics differ sharply in resolution.

Fixed-pool precision tightens as k grows, reaching ±1.3pt at k=10 in Japanese. Where the old metric could only say "any difference is within ±7pt", the repaired one produces cells that can judge differences of 1–2pt. Cells that actually cleared zero appear in the experiment sections.

The other way to raise resolution is to tag more articles. With all 420 tagged, the same-label pair count would grow considerably and the intervals would tighten further. AUC is wider than k=10 precision in English because resampling articles takes the full force of a 23-article sample's variance.

Get the pooling wrong and nothing raises — the recommendations just get worse. ruri and granite-english ship their own sentence vectors inside the ONNX, so the right pooling was settled by measurement, matching ours against theirs. bekko and granite-multilingual ship none, so each was confirmed from its 1_Pooling/config.json.

Models are judged with fixed-pool precision@k and pairwise AUC (how it looked under the old, model-dependent-denominator metric is kept later in this section for the record). Precision is checked for a consistent sign at topk 3 / 5 / 10. AUC does not depend on topk.

At 2,000 characters the only cell clearing zero is granite-97m int8 in English, and in the wrong direction (P@5 −7.8pt [−13.9, −1.7], P@10 −5.7pt [−10.4, −1.3]). That is quantization damage, not the model's ability (see the appendix). The two bekko models overlap the current one in every cell of every language — though the point estimates lean one way: AUC gives a25m +1.7pt in Japanese and a8m +3.8pt / a25m +3.2pt in English, positive for bekko in all four comparisons.

"Equal, possibly slightly better, but this sample cannot say so" — that is the fixed-pool reading of experiment 1. Re-measurement under production conditions (8,000 characters), and the cells that do reach significance, come after "the ranking flips with the reading".

How it looked under the old metric (for the record)

What follows is the reading from before the denominator fault was found. The per-language model is the dashed baseline, with the three multilingual candidates over it. The numbers themselves are correct, but the ranking between models cannot be read from them.

Measured with hand-written tags / categories only.

The same data returns opposite rankings depending on how it is read. On Japanese agreement, the current ruri is best at topk=3 and 5; on recall, bekko-a25m is best at all three. Neither is the correct one — they differ in how the denominator is taken.

The denominator of agreement is "pairs that could be scored", and the model can move it. A model more inclined to recommend tagged articles gets a larger denominator and a lower rate. And indeed a25m always has more scored pairs than ruri (194 against 174, 307 against 271, 564 against 513). The denominator of recall is "correct answers available", which the corpus alone fixes and no model can move.

In English both readings point the same way. On agreement and on recall, the bekko models lead at topk=3 and 5, and the current granite-small-english is never best in any column. But there are only 58–199 scored pairs.

The int8 build of granite-97m-multilingual pushes Japanese in-degree up to 47 (the current ruri is 35). Quantization is covered in the appendix.

All five models so far were compared under one fixed condition — embedding only the first 2,000 characters of the body. Experiment 2 varies what experiment 1 held fixed.

2,000 is the default for max_content_chars, put there to keep memory down. It was not chosen for quality. The first question is how much it throws away.

The counting needs care. prelims strips HTML tags, code fences, math and URLs before embedding, so counting the raw Markdown badly overstates what the model sees. This corpus holds many posts migrated from Medium, heavy with HTML and URLs: on the raw files the Japanese median looked like 1,205 characters with 154 posts over 2,000. The table above is post-filter, and it matches the production log exactly (93 Japanese and 15 English posts re-embedded when the limit went 2,000 → 8,000).

English was embedding less than half of each body. With a median of 1,854 characters, longer posts were being compared on their introductions alone. The distribution makes it plainer.

78% of Japanese posts fit inside 2,000 characters. The largest bucket is under 500 characters, holding 150 of the 420. Raising the limit to 8,000 changes the input for only 93 posts. English has only 55% under 2,000, and 12% of posts run past 8,000.

That difference is the whole explanation for why the same change helped English and did nothing for Japanese. Only 1 Japanese post of 420 extends past 8,000 characters, so for Japanese, 8,000 is effectively "the whole article".

This is the one result in the whole investigation that points the same way under either reading. Hits go +1 / ±0 / +6 in Japanese and +1 / +2 / +2 in English. Five of six cells positive, the sixth flat, and not one negative.

The reason this holds up where the model comparison did not is that both arms are the same model. Whatever tendency a model has toward recommending tagged articles is identical on both sides, so the denominator bias cancels. The metric breaks across models, not across settings.

Read as agreement, English looks like the bigger gain (+3.4 / +5.3 / +0.6 against Japanese +0.2 / −0.5 / +1.0) — but that is the English denominator shrinking (60→58, 103→96). By hit count both languages gain about equally.

All eight cells have a non-negative point estimate and not one comes out negative. No single cell reaches significance (the two at k=10 touch zero at the edge of the interval), but the same direction holding across metrics, across languages and across k is the strongest argument for this change. Unlike the model comparison, both arms are the same model, so the old metric's denominator bias largely cancelled too — this is the one experiment whose conclusion did not change when the metric was repaired.

8,000 it is. Both languages gain correct pairs and neither gets worse. Post-filter the cost is 1.33× the tokens and 2.5× the attention in Japanese; in English the absolute volume is small.

Past that point, unmeasured

While the measurement still ran on raw text it went up to 20,000 characters (effectively unlimited), and neither language moved beyond 8,000. That was not re-measured after filtering. But the longest post-filter article is 8,890 characters in Japanese and 15,581 in English, so what survives past 8,000 in Japanese is the tail of a handful of articles.

The two readings rank them in opposite orders

Take the discrepancy experiment 1 exposed and pin it down on the dedicated model against bekko-a25m. The same recommendations, scored two ways, with the difference (bekko − dedicated model) overlaid. The zero line is the dedicated model.

Repairing the metric did not settle the ranking. What is actually still open is a question one step earlier — how much of a reader's sense of "related" does this proxy metric capture at all? That cannot be answered from inside the metric, so I judged the recommendations themselves by hand and compared.

The subjects are the 389 articles whose recommendations changed. I wrote a dedicated judging tool that puts the two recommendation lists side by side and lets you tick the ones you consider genuinely related. Three things mattered in its design.

• Blind by default. Which column is which model is shuffled per article and hidden, and tags and tag-hit marks are hidden too. The tags are the very metric being used to pick a model, so judging with them visible would mean validating the metric with the metric itself
• Each recommendation is judged once. Where both models recommend the same article, ticking one side applies to the other. No pair can carry contradictory labels
• Strata are not pooled. The 162 judged articles centre on a seed-fixed random 40 and the total-swap group (zero shared recommendations — the 64 articles where the models disagree most). All 19 changed English articles were judged. The total-swap group was selected to maximise the difference, so it cannot be used for an overall estimate. The unbiased estimate is the random 40 alone

162 articles / 827 pairs judged (143 Japanese, 19 English), 458 recommendations marked related, 19 articles marked "none of these are related". One judge, the author.

Result 1 — the model difference still cannot be detected

a25m is non-negative in all four strata and the current model leads in none of them. Per article it is 46 wins for a25m, 36 for the current model, 80 ties (sign test p = 0.32). The direction is consistent, yet every stratum's interval spans zero. The +1.7pt from the unbiased random 40 is, in counts, 2 more hits out of 120 slots.

Result 2 — the proxy metric saw only 9% of relatedness

This is the real point. Take the 827 judged pairs and evaluate tag agreement as a predictor of the human labels, and what the metric was measuring comes out as a number.

Precision 0.755, recall 0.087. If the tags agree, a human almost always calls the pair related — that direction is trustworthy. But of the 458 recommendations a human called related, the tags found 40.

The reason is simple: 86% of judged pairs (714/827) have no tag on at least one end and were invisible to the metric from the start. And in that invisible territory, the human marked 392 pairs related.

Split by language, the same metric turns out to break in two different ways.

Japanese is "right when it fires, almost blind otherwise" (precision 0.86, recall 0.06). English is "sees a fair amount, but firing does not mean right" (recall 0.29, precision 0.63). The tagged share differs — 23% (ja) against 70% (en) — and where tags are dense, tag agreement turns loose enough to pick up 9 pairs a human called unrelated. Tilt it either way and the proxy still does not stand in for a human.

This asymmetry decides how the experiments should be read.

• "A recommendation whose tags agree is sound" — this direction is broadly safe (precision 0.755)
• "A recommendation whose tags disagree is unsound" — this direction is flatly wrong (recall 0.087)
• And the model comparison rode on the second inference. Almost all of the difference between the two models happens in untagged territory, and the metric was not looking there at all

So the numbers in experiments 1 and 2 were not "broken": they measured a narrow slice of relatedness, accurately, as that slice. The slice was simply too narrow to speak about which model is better.

Differences the metric cannot see — some examples

What happens outside the tags shows up in the individual cases I read while judging. a25m's winning pattern is detecting the same entity, which is the kind of match that essentially never appears in a tag.

Neither habit is broken. Which one you prefer is a design choice, not something measurement decides — which is also how it felt after reading 162 articles.

Result 3 — so should the choice be made per article?

If the habits differ, the next question follows naturally: maybe "this article wants this model" is a real thing. If it is, a mechanism that picks the better model per article would beat both.

But a win-loss table cannot answer that. Even two evenly matched models will trade wins and losses when they recommend different articles, so "there are wins and losses per article" is not evidence of structure. The question is whether those wins are predictable. Three tests.

The oracle row is the clearest. Taking the per-article winner raises 0.603 to 0.698 (+9.5pt). But "pick the winner" is an operation that, on two perfectly equal models, still earns +7.9pt [+5.7, +10.0] just by skimming noise. The observed +9.5pt merely sits at the edge of that distribution — any structure there is about the same size as the noise.

The trend by year — a25m doing better on newer technical articles — lines up neatly in the point estimates, but 2020 onwards is n=15 with an interval of [−6.7, +26.7]. The correlation with year itself is r = +0.086, p = 0.301, so the neatness of the ordering is explained by how thin the samples are.

In the end it comes down to which one you prefer. Neither measurement nor inspection produces a ranking, and switching per article is not supported. What is left is a preference about which habit you want on your own blog.

Consolidating onto one model means choosing between the two multilingual candidates. And what mostly matters is what happens when an article is written in a language other than Japanese or English — exactly what was not measured. Beyond the two languages measured here, that means leaning on published coverage information.

Both were trained on a wide set of languages. bekko was trained on data covering more than 100 languages so that it can search across languages, retrieving English documents from a Japanese query without a translation step. granite's base encoder is pre-trained on more than 200 languages. The difference is not the training range but how far the per-language numbers are published.

The 14 languages bekko evaluates individually

Not the limit of the training range but the range with published measured scores. Measured on English NanoBEIR plus community-published translations into 13 languages.

English · Spanish · Italian · Portuguese · German · Vietnamese · Japanese · French · Swedish · Norwegian · Korean, and the bottom three, Arabic · Serbian · Thai.

The 52 languages granite-97m-multilingual lists as enhanced-support

The base encoder is pre-trained on more than 200 languages, so a general-purpose embedding is available for any of them. On top of that, quality is claimed for the 52 languages given explicit retrieval-pair and cross-lingual training data.

Quoted as written on the model card.

sq  Albanian     ar  Arabic       az  Azerbaijani  bn  Bengali
bg  Bulgarian    ca  Catalan      zh  Chinese      hr  Croatian
cs  Czech        da  Danish       nl  Dutch        en  English
et  Estonian     fi  Finnish      fr  French       ka  Georgian
de  German       el  Greek        he  Hebrew       hi  Hindi
hu  Hungarian    is  Icelandic    id  Indonesian   it  Italian
ja  Japanese     kk  Kazakh       km  Khmer        ko  Korean
lv  Latvian      lt  Lithuanian   ms  Malay        mr  Marathi
no  Norwegian    fa  Persian      pl  Polish       pt  Portuguese
ro  Romanian     ru  Russian      sr  Serbian      sk  Slovak
sl  Slovenian    es  Spanish      sw  Swahili      sv  Swedish
tl  Tagalog      te  Telugu       th  Thai         tr  Turkish
uk  Ukrainian    ur  Urdu         uz  Uzbek        vi  Vietnamese

It is also trained on Python · Go · Java · JavaScript · PHP · Ruby · SQL · C · C++ code, and claimed to support cross-lingual code search.

The declarations are at different granularities, so breadth of coverage cannot rank them. granite's 52 is a list of languages given explicit retrieval-pair and cross-lingual data, not per-language scores. bekko's 100+ is the set of languages included in training, also not per-language scores. Neither publishes anything in the shape of "this language will be fine".

There is exactly one real difference: whether per-language measurements are published. bekko has numbers for 14 languages. granite publishes no per-language figures. As material for choosing what to consolidate onto, those 14 languages are the only thing resembling evidence.

Position on public benchmarks

The two models are compared directly in bekko's announcement post. Not per language, but it is the only third-party measurement available, so here it is.

bekko-a25m leads on five of the six entries; granite leads only on MMTEB's retrieval task (60.3 against 57.5). a25m also has fewer active parameters (24.9M against 28.3M).

Related-article recommendation deals in symmetric similarity between articles, so MMTEB (ALL) — the average over all 131 tasks — is closer to it than the query → document retrieval task. There the gap is 58.3 against 51.9, 6.4pt. But this is not a per-language number, so it does not answer "which is better in a specific language that has no dedicated model".

On the granite side, IBM itself writes that "pruned models show more pronounced degradation on low-resource languages". The fallback is used mainly for low-resource languages, so that lands squarely on the use case. Losing to a25m by 4–5pt even in Japanese — a high-resource language — looks consistent with that note.

Consolidate onto one multilingual model, and change the input length too. The verdict flipped twice during the investigation. Before the metric was repaired: "no quality improvement large enough to justify replacing". After the repair: "quality is equal or marginally better, but the difference does not cover the added cost". Then, after human evaluation and measuring the cost for real: "quality is equal. The ranking cannot be settled in principle, and switching per article is not supported. What is left is preference and cost — and the cost turned out smaller than expected" — that is the final form.

ja   → bekko-embedding-v1-a25m    199 MB  changed (was ruri-v3 + "トピック: ")
en   → bekko-embedding-v1-a25m    199 MB  changed (was granite-small-english)
rest → runs on the same model

max_content_chars   2,000 → 8,000         changed

• Quality dropped out of the decision. In the fixed pool, Japanese P@3 is tied, a25m is +1.4pt at P@10, AUC +1.8pt (n.s.). Under human judgment the random 40 gives +1.7pt [−9.2, +12.5], and per article 46 wins to 36 with 80 ties (p = 0.32). All 19 changed English articles were judged and came out an exact tie (40/57 against 40/57). a25m is non-negative in every stratum and the current model leads in none, yet not one result reaches significance. And judging all 389 would still not reach it (see "Human evaluation"). What cannot be measured cannot ground the decision
• The deciding factor is a preference between habits. a25m picks up the same entity — same artist, same author, sister event — while the current model keeps topic purity. The case for switching per article was rejected by the tests (Result 3), so it comes down to choosing one habit. For a personal blog, I judged that entity connections — which appear in neither the tags nor the metric — matter more to a reader
• The cost was measured and it was small. On the real GitHub Actions run, 3m29s on the first run (re-embedding all 452 articles, model download included). That breaks down as 3m8s embedding and 17s download, so 199 MB matters less than the number suggests — with 25M active parameters in a MoE, an article takes about 0.4s. Later runs only touch new articles, so the steady- state cost is unchanged. I also confirmed that running the same weights twice produces byte-identical recommendations
• A side benefit. Switching by detected language disappears and the configuration gets simpler. The same model runs for languages other than Japanese and English — the multilingual coverage that motivated this in the first place, obtained without giving up quality
• max_content_chars from 2,000 to 8,000. This is the most solid result in the investigation. All eight fixed-pool and AUC cells have a non-negative point estimate; none is negative. Both arms are the same model, so the old metric's flaw does not touch it. It goes in on "a small gain, and no loss if it misses"
• The corpus length distribution should have been checked before touching the input length. The input actually changed for 93 of 420 Japanese articles and 15 of 33 English ones. For the rest, raising the cap changes not one character. What a setting can reach is decided by the corpus, not the model
• Quality outside Japanese and English is adopted still unverified. What was measured is Japanese and English, both languages that have a dedicated model. How a25m behaves elsewhere is unknown — but this is still better than before. For a language with no dedicated model the only option used to be reusing the English model, and on the probes the separation margin outside its range all but vanished: 0.139 for French, 0.055 for Japanese. An unverified multilingual model is the better bet against a dedicated model known to be out of range
• The rival, granite-97m-multilingual, was dropped. Only fp32 is usable (the int8 build collapses in English, AUC 0.747 → 0.680), and even at fp32 its English AUC is below a25m's, with a25m ahead on five of six public-benchmark entries

On proxy metrics

The first metric could answer a coarse question. Asked "should this be replaced", it returned "no improvement large enough to justify replacing" — enough, if all you want is an upper bound. It broke one step finer down, at the moment "which model is better" was to be decided on a few points of difference: the design that lets the model choose what gets scored surfaced, and the ranking inverted depending on how you read it.

The repair took two operations. Take the denominator away from the model (fix candidates to tagged articles), and drop the cutoff and use the whole ranking (pairwise AUC). Both are standard responses to incomplete relevance judgments in information retrieval, not new inventions. After the repair the intervals shrank from ±5pt to the 1–4pt range and, for the first time, a cell came out clear of zero. Fixing a metric's design did as much for resolution as adding data would have.

And the flaw that remained was invisible from inside the metric. The repaired metric had a fair denominator but still saw only 9% of relatedness (see "Human evaluation"). Precision is a relatively high 0.755, so "a tag hit means it is sound" holds — yet the difference between the models happens almost entirely in untagged territory, outside what was being measured. A correctly designed metric was correctly measuring a narrow slice — that is the final diagnosis.

And not one of the errors found in this investigation was found by staring at the metric.

• The referee included TF-IDF-derived terms — found by swapping the referee
• Measurement ran on different text than production — found by checking the numbers against the production run log
• The ratio was being dragged by the size of the denominator — found by reading individual articles one at a time
• That denominator effect was manufacturing the apparent conclusion about input length — found by decomposing the rate into hits and denominator
• A model-movable denominator was a flaw in the design itself — confirmed when the same data under two readings inverted the ranking
• Even after the repair the metric saw only a fraction of relatedness — only revealed by judging the recommendations themselves by hand

Every one came from outside the metric, or from setting metrics against each other. A proxy metric cannot be validated on its own. A proxy breaks not only on "what counts as correct" but on "who decides the denominator", and once fixed its limit is set by "how much of the field it sees". Even after writing a note on the third one saying "this has a side effect", it took a while to notice that my own conclusion was built out of that side effect. A note is not absolution — the error sat in the conclusion until the design was fixed.

If you are building a proxy metric, take a sample of human judgments first, small is fine, and evaluate the proxy as a predictor of it — cheap insurance. Doing that last is why I carried the misreading "low metric = bad recommendations" for so long. In fact 76% of the recommendations were sound.

The biggest finding here did not come from the metric either. The reason input length matters in English and not in Japanese lies in the corpus length distribution — 327 of 420 articles do not reach 2,000 characters. The metric only confirmed it; the explanation was on the data side.

What this investigation measured is two languages, Japanese and English, both high-resource, both with a dedicated model available. A fallback is needed precisely for languages that meet neither condition. That asymmetry is why the conclusion calls a25m the "first choice" and stops short of declaring it verified.

To go further, the inference would have to be assembled like this.

1. Public benchmarks give a relative ordering across several languages (third-party measurement)
2. This corpus confirms, at two points only, that the ordering also holds for related-article recommendation
3. Extrapolate: for an unmeasured language, if its score in (1) is comparable to Japanese, (2) should be comparable too

(3) is the leap. The distance being crossed is not just "a different language" — it crosses the resource-volume axis. Low-resource languages are where models lean on cross-lingual transfer, and where every model degrades most. IBM's note about granite, "pruned models show more pronounced degradation on low-resource languages", is about exactly this territory.

There is only one way to find out. Take a few dozen articles in the target language and run the same procedure. Nothing in this toolchain is language-specific, so the same metrics apply as soon as a corpus exists. With parallel translations you could also see whether the recommendation set holds across languages. chezo.uno has only Japanese and English, so that is not possible today.

As for other limits: with hand-written tags as the referee, the writer's tagging habits go straight into the metric. Depending on topk there are only 58–536 scorable pairs. And this is two languages on one blog — every difference reported here is a difference within that scope.

Human evaluation has limits of the same kind. There is one judge, the author, and no second judge to measure agreement against. Blinding (model names, tags and tag-hit marks all hidden, left and right shuffled per article) cut off bias towards a model, but the standard for "related" still rests on the author's sense of it. 162 of the 389 changed articles were judged: a sample in Japanese, the complete set in English.

ruri and granite were measured on int8 model_quantized.onnx, bekko on fp32 model.onnx (bekko's author does not recommend the quantised build). To check whether that favoured bekko, all three models were run at both int8 and fp32.

Quantisation does not break the two production models. The AUC differences are ruri +0.6pt [−0.2, +1.5] and granite −0.4pt [−3.2, +3.2], both spanning zero, and the fixed-pool hit difference stays within ±2. int8 is fine as it is — no reason emerged to take ruri to 120 MB or granite to fp32.

The only thing that broke was the candidate, granite-97m-multilingual, and the damage lands differently per language. In English the AUC collapses 6.7pt, 0.747 → 0.680, and in the fixed pool it produced the only cell in any model that falls significantly short of the current model (experiment 1). In Japanese the AUC barely moves, yet hub concentration is clearly worse at int8 (maximum times recommended 47 against 39). The same quantisation shows up as ranking quality in English and as hub concentration in Japanese.

This model is a pruned build of the 311m (22 layers → 12), and the natural reading is that with the redundancy already cut there is nothing left to spend on quantisation. Quantisation tolerance is a model-specific property; measuring it on one model does not generalise to another. In this very investigation three models came through untouched and one broke.

So granite-97m-multilingual is compared using its fp32 numbers. Listing its int8 figures as "this model's performance" would mistake quantisation damage for model quality.

As a by-product, it also became visible that the tag referee is remarkably insensitive to recommendations changing. Between ruri int8 and fp32 the recommendations change for 319 of 420 Japanese articles, yet fixed-pool hits move by 2 and AUC by 0.6pt. A largely different set of recommendations can be at the same level as far as tags can measure.

All of it lives in prelims-cli's scripts/.

• check_onnx_model.py — the ONNX inputs and outputs, whether the required inputs are satisfied, the output rank, pooling pinned down against the model's own sentence vectors, and the probe separation margin
• compare_embedding_variants.py — runs the corpus under two settings and reports the diff and the metrics. Alongside tag agreement and hub concentration, it reports fixed-pool precision@k and pairwise AUC with paired-bootstrap 95% intervals (--bootstrap, default 1,000 resamples, fixed seed). --vary config compares any two models, and --cache-dir reuses embeddings so a topk sweep costs one run
• rerun_all.sh — re-runs every comparison in this report (models × 2 languages × 3 topk levels, input length, quantisation, the referee swap, per-article diffs) in one command and leaves the output in a file. Every number in this report comes from that output
• judgments/ — the human evaluation labels (162 articles, 827 pairs) and the scoring script. The stratified precision@3, the bootstrap intervals and the confusion matrix against the tag metric all reproduce from it

bash scripts/rerun_all.sh          # re-run every comparison (minutes, with a warm cache)

# to run a single comparison:
python scripts/compare_embedding_variants.py ../chezo.uno/content/post \
    --language ja --permalink-base /post --topk 5 --vary config \
    --tag-keys tags categories --cache-dir ~/.cache/prelims-compare \
    --a "language=ja,prefix=トピック: " \
    --b "model_name=hotchpotch/bekko-embedding-v1-a25m,\
model_file=onnx/model.onnx,pooling=mean"

The embedding cache is keyed on article body text alone, so fixing front-matter tags leaves the cache warm. That property is why re-measuring after fixing the referee (the tags) takes minutes.