Jev is NOT a state-of-the-art reranker (but it's close)

We run millions of retrieval queries a month at Kapa. All of them go through a reranker in some way, so picking the right one matters a lot to us.

Erik Kaunismäki from turbopuffer recently showed TypeSafe AI's Jev beating dedicated rerankers on their benchmark. Not everyone has found the same: DevX Labs found it the fastest and cheapest, but only twelfth on accuracy. So we wanted to see what held in our domain, which is complex real-world company knowledge.

TL;DR: Jev is a good reranker, and it beat every Cohere and Qwen model we tried. The new Jev 1.14 came within 0.003 of the top. But Voyage AI's rerank-3 still came out on top, and it was also the cheapest and the fastest.

Benchmarking on real-world company knowledge

Company Knowledge Bench has 1,000 test cases built from real production queries. Each one comes with its own knowledge base and a labelled criterion that says which chunks a complete answer needs.

These knowledge bases are what our customers actually index: product docs, Slack threads, support tickets, wikis and code. That means stale pages, contradictory facts and many ways to answer the same query. We built our own benchmark because academic benchmarks looked nothing like our production traffic. You can read more about how it works in the benchmark post.

The retrieval score of our benchmark measures if the top 10 reranked chunks satisfy the criterion, with partial credit when only some of the required chunks are found, averaged across all 1,000 cases.

How we set up the test

This is a test setup, not what we run in production. We picked a standard retrieval pipeline because it makes everything easier to compare. For each query, an LLM (GPT-6 Luna) decomposes it into three distinct search queries. We retrieve the top 30 chunks for each query via hybrid search, merge them with reciprocal rank fusion, and send the top 50 to each reranker. Then we score the top 10 chunks it returns.

We tested Voyage rerank-2.5 and rerank-3, Cohere rerank-v3.5, v4.0-fast and v4.0-pro, the open-source Qwen3 reranker 8B via OpenRouter, and Jev 1.13 and 1.14 in two modes. We capped the candidate set at 50 because Jev needs one call per chunk, and 50 is a common candidate set size for reranking anyway.

Two ways to turn Jev into a reranker

Jev isn't trained as a reranker. It's a general-purpose decision model: given some context (here, the user query and one chunk), a question and a set of labels, it returns a probability distribution over those labels. To turn that into a ranking, one has to pick a question and a way to collapse the distribution into one normalized number per chunk. There are two standard ways to do it, and we tried both.

Noul (binary). A simple yes or no question: is this chunk relevant to the query? Jev returns a Bernoulli distribution over the labels yes and no. We sort the 50 chunks by the probability of yes. It's a pointwise classifier very similar to how most specialized rerankers work, but for Jev it requires one call per query and chunk pair.

Score (graded). We ask how relevant the chunk is on a scale of relevance levels, from unrelated (1) to a perfect match (10), each mapped to a number. Jev returns a probability for each level, and we sort by the expected score: each level's value times its probability, summed. This is the method turbopuffer used. This allows Jev to make a more nuanced decision.

A perfectly calibrated model should produce more or less the same ranking for both approaches (up to ambiguity of the context, choices and question).

Results: Voyage wins on score, cost and latency

Voyage rerank-3 scored 0.535, at $1.91 per 1,000 questions and 0.18s median latency. No other reranker beat it on any of the three.

Jev came very close. We first ran Jev 1.13 and then reran everything with Jev 1.14. At 0.532, Jev 1.14 in Noul mode landed just 0.003 behind Voyage rerank-3 and ahead of Voyage rerank-2.5 (0.526). It also beat Cohere's newest model, rerank-v4.0-pro, on score (0.506), cost ($2.49 vs $3.42) and latency (0.28s vs 0.77s). Jev 1.13 in Noul mode scored 0.519.

Noul came out slightly ahead of Score in both versions: 0.532 vs 0.525 for Jev 1.14, and 0.519 vs 0.516 for Jev 1.13, with Score costing a bit more ($2.82 per 1,000 questions). On our data, the extra detail of the graded scale didn't improve the ranking.

The open model was the most expensive. Qwen3 reranker 8B cost 4.3x more than Voyage, took 14x as long, and ranked second to last.

Every model saw the same 50 chunks, and only the order changed. Even so, the best score was 18% higher than the worst (0.535 vs 0.454).

What we took away

First, results depend on the data and the score. turbopuffer's benchmark and ours point in different directions, and both can be right. If you're choosing a reranker, test it on your own data and evaluate it in a way that makes sense for your use case.

Second, Jev is worth a look if you want a reranker you can steer with a rubric, and with 1.14 it got remarkably close. For our domain, though, Voyage was still the best, so we're sticking with it. Good job to Tengyu Ma and the rest of the Voyage team at MongoDB.

Cohere costs come from billed units. For Qwen3 reranker 8B we used what was available on OpenRouter. Self-hosting it would most likely make it cheaper and faster.