We evaluate paper retrieval on 23 complex queries submitted to Undermind, specified as natural language research goals. We sample randomly from non-sensitive anonymized queries, stratifying for diverse research fields. We filter out overly broad review queries.

For each goal, we construct a gold standard by scoring every paper any retrieval system retrieved, using GPT-5.6 Sol (xhigh) as a judge. The judge assesses relevance from each paper's abstract. Thus, no retrieval system is disadvantaged: every retrieved paper gets a chance to enter the gold standard. A paper enters the gold standard when its score reaches the rubric's threshold for precise relevance. Additionally, to ensure a high degree of completeness of the gold standard, we run several of the retrieval systems multiple times with high budgets to aggregate a comprehensive set of candidates.

We compare Undermind against frontier models running as search agents: GPT-6.1 Sol and GPT-6 Astra in the Codex CLI harness, and Claude Opus 5.5 in Claude Code. Each agent has web search and shell access, so it can reach public web APIs. Undermind runs the production deep search with an extended budget to trace the full ten-minute curve. A production search stops on its own after about 2.5 minutes on average. Resubmitting a similar search continues from existing results, so users can reach the rest of the curve via multiple search submissions.

We ask each retrieval system to score its findings with the same relevance rubric the gold standard judge uses. Those scores order each retrieval system's returned list, and precision is measured over that order. Search time counts the time each system spends searching: deep search's research loop, and the web agents' turns. Deep search reports its list after each round, and the web agents record each paper as they find it. We interpolate recall and precision linearly between reports.