Before a biomedical search benchmark: an evaluation checklist

Status: evaluation plan, not a result. No queries were executed, no relevance judgements were collected, and no performance scores are reported for this article. The checklist below is a proposed protocol, not evidence that a product has passed it.

Define the decision before the metric

Start with the work the search is meant to support. Finding a known accession, surveying a mechanism, and assembling evidence for a review are different tasks. A single average over all three can conceal a failure that matters to a particular reader.

Write down the intended users, information needs, source scope, and unacceptable failures before examining results. Keep three questions separate: is this hit relevant, does the source support the claim, and is the evidence suitable for the intended use? Search relevance alone does not answer the latter two.

For RBAi Scholar, the public query guide is the documented starting point for retrieval behavior. This note proposes how to evaluate that behavior; it does not independently verify it.

Build a query set with distinct tasks

Create a small development set for learning the interface and a separate, held-out evaluation set. Do not repeatedly tune a query on the evaluation set and then present the improved ranking as an unbiased outcome. Preserve the original information need alongside the exact submitted text.

These are examples for designing a set, not executed queries or known outcomes. Verify a supposed no-match control against the evaluation corpus before labelling it absent. Stratify by the biology that matters to your intended users, including organism, terminology, ambiguity, and rare identifiers. Do not let one familiar gene dominate the set.

The current guide states that Boolean operators are interpreted in keyword mode, not semantic mode; multiple quoted requirements in semantic mode are combined with AND. It also documents unsupported grouping and refusal conditions. Include those cases deliberately, but do not score an intentionally invalid query as ordinary relevance failure.

Freeze the run record and evidence trail

Before running a comparison, fix the query set, requested result depth, relevance rules, and handling of duplicates. Use the same information needs across systems, while preserving any mode-specific query translation. A translation is part of the method and must be disclosed.

Capture at least:

• The exact query, retrieval mode, filters, parameters, and requested result count.
• UTC retrieval time, endpoint or interface, and available service, model, and index versions. Mark unavailable versions as unavailable.
• The raw ranked output, original record identifiers, canonical source links, and the fields actually used to judge relevance. Keep the original rank even if a later analysis deduplicates records.
• Source type and version information where exposed: journal article versus preprint, record update date, and any correction or retraction notice checked at the source. Do not imply that a search service supplies all these fields.
• Any query-interpretation notices, rejected requests, timeouts, or missing fields. Preserve errors without storing API keys or sensitive inputs in shared records.

Here is a proposed local record format, not an API response or a completed run. null marks information that still needs to be captured:

{
  "status": "planned",
  "query_id": "identifier-01",
  "information_need": "Find literature concerning accession GSE135222",
  "query": "GSE135222",
  "mode": "keyword",
  "cutoffs": [5, 10],
  "retrieved_at_utc": null,
  "service_version": null,
  "index_version": null,
  "raw_results": null,
  "relevance_judgements": null,
  "errors_and_notices": null
}

If the live index cannot be frozen, call the exercise a dated live-service evaluation rather than a reproducible comparison over an identical corpus. Saving outputs helps audit that run; it does not make a changing index identical on a later date.

Decide how relevance will be judged

Write a rubric before labelling results. Include the target entity, organism, mechanism, and evidence type where they matter. Decide whether abstracts are sufficient to judge the information need or whether the full source must be read. Do not silently promote an abstract-level match into support for a scientific conclusion.

Use explicit labels such as relevant, not relevant, and not yet judged, with a short rationale and the evidence inspected. Where feasible, have two assessors judge independently without seeing which system supplied a hit, then record disagreements and their resolution. That is a proposed safeguard, not a claim that these opening notes have received such review.

Treat unavailable papers, missing abstracts, conflicting metadata, and uncertain relevance as visible gaps. Do not change the rubric after seeing a preferred system’s results without reporting the change.

Report precision at a fixed depth carefully

For an agreed cutoff k, precision@k is the number of relevant results in the first k ranks divided by k:

precision@k = relevant results at ranks 1 through k / k

The Introduction to Information Retrieval discussion of ranked evaluation explains why this is useful for first-page relevance and why it is sensitive to the query set. It does not require an estimate of every relevant document in the corpus, and it should not be mistaken for recall.

For this proposed protocol:

1. Predeclare cutoffs such as k = 5 and k = 10; these are evaluation choices, not reported scores.
2. Judge all returned ranks being scored. Do not publish a complete precision@k when some of those ranks remain unjudged.
3. If a valid search returns fewer than k results, keep the fixed denominator k, record the returned count, and treat the unfilled ranks as non-relevant for this fixed-depth measure. If you additionally report precision over returned results, label it as a different measure.
4. Keep request errors and refusals in a separate failure analysis. Do not silently substitute an empty successful result for a failed request. Disclose the count and handling of every excluded query.
5. Set a duplicate policy in advance. Preserve the raw ranking; do not remove duplicate hits after the fact simply to improve a score.
6. Show per-query and per-family values before any average. State which queries enter an equally weighted mean and how judgement gaps are handled.

Raw semantic similarity or keyword-ranking scores are not a common probability scale and should not be compared as if they were. Precision@k says nothing on its own about how much relevant literature was missed, the completeness of the index, clinical suitability, or the correctness of a generated answer.

Review failures as part of the method

Record the original request and the specific problem rather than describing every disappointing search as a relevance failure. Useful categories include:

• An operator sent to a mode that does not parse it.
• A quoted requirement that narrows the intended topic incorrectly.
• A documented broad-query refusal, distinct from an unavailable service or timeout.
• A correct literal match that concerns the wrong organism, mechanism, or study type.
• A near semantic match that does not satisfy the information need.
• Duplicate records, stale metadata, missing source links, or a preprint mistaken for a journal article.

The public guide’s refusal and limitation sections describe behavior to inspect. Keep the distinction between a documented guard and an observed failure: the former is documentation until the relevant request has actually been made and recorded.

Before calling it a benchmark

Do not publish a benchmark claim until the query set, executed run records, relevance rubric, judgement coverage, failure handling, and limitations can be inspected. Choose any acceptance criteria before looking at evaluation scores and explain why those criteria fit the task. Do not invent a universal passing threshold.

State who actually ran the evaluation and who actually judged relevance. Release only material you are permitted to share, with credentials and sensitive information removed. An executed search test would still evaluate retrieval under a particular setup; it would not validate an entire biological-research workflow.

This note stops at the plan. Its examples and cutoffs are proposals, and there is no pass/fail finding, latency measurement, accuracy claim, or product ranking to infer from it.