How I Reproduced BM25, Dense Retrieval, and SPLADE on a 16GB MacBook

I did, on a MacBook with 16GB of memory, using a public set of exercises called the Foundations of Retrieval onboarding path, maintained by the Castorini information retrieval research group. The path uses two open-source toolkits from Castorini: Anserini, which is Java-based, and Pyserini, its Python counterpart. It tests them on two real document collections and gives exact expected scores for each exercise. Every number matched. Getting there took an out-of-memory crash, a native library crash, and a gated model access request that a valid application programming interface (API) key alone did not satisfy. None of that shows up in the official documentation, so this article covers it as carefully as the numbers themselves. By the end, you will know how to run the three methods locally, what can break on normal hardware, and how to read the scores. The practical application is straightforward: if you are building a RAG system, a document search product, or an internal knowledge base, retrieval quality is the first thing to verify. Before changing prompts, swapping large language models, or adding agents, you need to know whether the right documents are being found at all. Reproducing these baselines gives you a reference point: what BM25 can do, what dense retrieval adds, what learned sparse retrieval costs, and how to check whether your own scoring pipeline is behaving correctly. What BM25, dense retrieval, and SPLADE actually do All three methods solve the same problem: given a search query and a large collection of documents, return the documents most likely to answer it, ranked best first. They differ only in how they turn text into something a computer can compare and rank. BM25 reads a document and a query, counts which words they share, and weighs each shared word by how rare and how frequent it is. It produces a sparse vector, a long list of numbers where almost every entry is zero, and the nonzero entries are word weights. It needs no training data. The weights are determined by a fixed formula, tuned by two constants (k1 and b), rather than learned from examples. Dense retrieval replaces that formula with a trained neural network. A document encoder reads the document and outputs a fixed-length list of numbers, an embedding, meant to capture meaning rather than exact wording. A query encoder does the same for the query. The two embeddings are compared with a dot product. That means multiplying matching positions and adding the results. The highest-scoring documents win. This requires a model trained on large amounts of labeled query-document pairs. Learned sparse retrieval, the category SPLADE belongs to, is a middle path. Like BM25, it produces a sparse vector where most entries are zero. Unlike BM25, a trained model determines which words matter and by how much, rather than relying on a fixed formula. SPLADE version 3 is the specific model used in this article. All three follow the same broad pattern: encode the query, encode the document, then compare the two. This is called a bi-encoder architecture. Jimmy Lin’s paper on a representational framework for information retrieval lays this out formally. Section four of this article verifies it by hand. Two collections, one full scale and one small on purpose The exercises use two test collections. MS MARCO passage ranking is a Microsoft benchmark of 8,841,823 passages with 6,980 development queries, released for non-commercial research. It is full-scale: the same core operations a production search system relies on- indexing, searching, and evaluating ranked results- run once here instead of continuously. The BM25 result here matches the official MS MARCO leaderboard entry for “BM25 (Lucene8, tuned)”, published on June 26, 2019. It is a publicly recorded benchmark result, not a number picked to make the article look good. NFCorpus is a smaller medical retrieval dataset with 3,633 documents and 3,237 queries. Because it is small, all three methods can run on a laptop CPU in minutes or hours instead of requiring a GPU. It is used here for a direct, same collection comparison of BM25, dense retrieval, and SPLADE side by side. Treat the NFCorpus numbers as a real result on this specific dataset, not a general ranking of which method wins everywhere. Its small size is what makes the comparison practical on a laptop, not a shortcut around real evaluation. All commands below assume you are working from the root of a cloned Anserini or Pyserini checkout, matching the official onboarding guide’s own convention. Setup friction the official docs do not mention Three problems showed up before a single retrieval command ran. Cloning both repositories with full git history timed out repeatedly. Anserini and Pyserini have years of commit history; a shallow clone fixed it immediately: git clone --depth 1 https://github.com/castorini/anserini.git A required submodule (anserini-tools , shared between both projects) failed mid clone with a connection reset. Raising git’s HTTP buffer size resolved it: git config http.postBuffer 524288000 git submodule update --init --recursive --depth 1 Both toolkits require Java 21 specifically. The default installed JDK was Java 11 (via Homebrew), which caused confusing failures until JAVA_HOME was pointed explicitly at the right version, in every terminal session: export JAVA_HOME=$(/usr/libexec/java_home -v 21) export PATH="$JAVA_HOME/bin:$PATH" None of these three problems is exotic. Anyone reproducing this on a machine that is not already configured exactly like the maintainers’ will likely hit at least one of them. BM25 on the full MS MARCO passage collection, and the memory crash The first real exercise builds a Lucene inverted index over all 8,841,823 MS MARCO passages, then searches it with BM25. An inverted index maps each word to the documents that contain it. It is the same basic structure used by full-text search engines. The indexing command is straightforward: bin/run.sh io.anserini.index.IndexCollection \ -collection JsonCollection \ -input collections/msmarco-passage/collection_jsonl \ -index indexes/msmarco-passage/lucene-index-msmarco \ -generator DefaultLuceneDocumentGenerator \ -threads 9 -storePositions -storeDocvectors -storeRaw On a 16GB machine, this command fails. bin/run.sh hardcodes a 192GB Java heap ceiling: java -cp `ls target/*-fatjar.jar` -Xms512M -Xmx192G ... The operating system’s out-of-memory killer ends the process partway through: 2026-07-12 10:15:49 - Starting to index... 2026-07-12 10:16:49 - 0.00% of files completed, 3,400,000 documents indexed bin/run.sh: line 3: 95895 Killed: 9 java -cp `ls target/*-fatjar.jar` -Xms512M -Xmx192G ... This captured terminal output shows macOS killing the indexing process after 3.4 million of 8.8 million documents, with no error message beyond Killed: 9 . This is what a silent out-of-memory kill looks like in the terminal: no Java exception, only the process disappearing. Lowering the heap ceiling to 10GB and the thread count from 9 to 4 fixed it. The second run indexed all 8,841,823 documents in two minutes and sixteen seconds with zero errors, matching Anserini’s own documentation of roughly four gigabytes of index size on disk. Retrieval and evaluation follow: bin/run.sh io.anserini.search.SearchCollection \ -index indexes/msmarco-passage/lucene-index-msmarco \ -topics collections/msmarco-passage/queries.dev.small.tsv \ -topicReader TsvInt \ -output runs/run.msmarco-passage.dev.bm25.tsv \ -format msmarco -parallelism 4 \ -bm25 -bm25.k1 0.82 -bm25.b 0.68 -hits 1000 python tools/scripts/msmarco/msmarco_passage_eval.py \ collections/msmarco-passage/qrels.dev.small.tsv \ runs/run.msmarco-passage.dev.bm25.tsv The result was an MRR at 10 of 0.18741227770955546, matching the official expected value from the Castorini guide to every digit. MRR at 10, or mean reciprocal rank, is the official MS MARCO metric. It measures where the first correct passage appears in the ranked list for each query, so higher is better. Proving the score is literally a dot product This is the section that matters most, because it turns “BM25 gave a score of 17.9” from a fact you accept into a fact you can check. Take document 7187158 (an answer to the query “what is paula deen’s brother”) and reconstruct its BM25 weight vector directly from the index: from pyserini.index.lucene import LuceneIndexReader index_reader = LuceneIndexReader("indexes/lucene-index-msmarco-passage") term_frequencies = index_reader.get_document_vector("7187158") bm25_weights = { term: index_reader.compute_bm25_term_weight("7187158", term, analyzer=None) for term in term_frequencies.keys() } Build the query as a simple vector: one for every query word, zero everywhere else. from pyserini.analysis import Analyzer, get_lucene_analyzer analyzer = Analyzer(get_lucene_analyzer()) query_tokens = analyzer.analyze("what is paula deen's brother") query_weights = {token: 1 for token in query_tokens} Compute the dot product by hand, summing the weight of every word that appears in both: score = sum( bm25_weights[term] for term in bm25_weights.keys() & query_weights.keys() ) The hand computed result is 17.949487686157227. Lucene’s own search returns 17.94950 for the same query and document, the same number to the precision Lucene itself reports. BM25 search is, by construction, comparing a query vector against a document vector and ranking by the highest dot product, so the two numbers matching is neither a coincidence nor an approximation. Once you reproduce that score with a short script, it stops feeling mysterious. It becomes a number you can audit. The same check works for dense retrieval and SPLADE. You can reconstruct the document vector, encode the query, compute the dot product, and compare it with the engine’s reported score. The runnable scripts for all three are in the companion repository. Dense retrieval, and a crash between two libraries that do not know about each other Dense retrieval swaps BM25’s word counting formula for BGE-base-en-v1.5, a published embedding model from the Beijing Academy of Artificial Intelligence (BAAI), released under the MIT License, which explicitly permits commercial use. Running it against the same MS MARCO queries uses a prebuilt HNSW index (Hierarchical Navigable Small World, a graph structure built for fast approximate search over dense vectors, downloaded rather than built locally): bin/run.sh io.anserini.search.SearchHnswDenseVectors \ -index msmarco-v1-passage.bge-base-en-v1.5.hnsw \ -topics collections/msmarco-passage/queries.dev.small.tsv \ -topicReader TsvInt \ -output runs/run.msmarco-passage.dev.bge.txt \ -encoder BgeBaseEn15 -hits 1000 -threads 4 Running this with more than one search thread crashed the process outright, not with a Java exception but a segmentation fault inside a native threading library: # A fatal error has been detected by the Java Runtime Environment: # SIGSEGV (0xb) at pc=0x0000000133938734, pid=5863, tid=259 # Problematic frame: # C [libomp.dylib+0x58734] void __kmp_suspend_64(...) The Java virtual machine and PyTorch (used internally by the encoder) each load their own copy of the OpenMP threading runtime, and on macOS the two copies collide. The fix is one environment variable plus running search on a single thread: export KMP_DUPLICATE_LIB_OK=TRUE With that set, the same command completed cleanly. MRR at 10 came out to 0.3521, an 87.9 percent improvement over BM25’s 0.1874 (higher is better). BM25, dense retrieval, and SPLADE side by side on NFCorpus NFCorpus is small enough to compare all three methods on the same laptop CPU. I used nDCG at 10, a metric that rewards methods for placing relevant documents near the top of the ranked list. BM25 with default parameters scored 0.3218. BGE-base dense retrieval scored 0.3808, an 18.3 percent improvement. SPLADE version 3, a learned sparse retrieval model from Naver released under a non commercial Creative Commons license (more on that below), scored 0.3624, landing between the two: 12.6 percent above BM25 and about 4.8 percent below dense retrieval. SPLADE introduced its own friction. naver/splade-v3 is a gated model on Hugging Face: a valid API token alone does not grant access. A Hugging Face token is not enough for this model. The account must also request access on the model page before the token works, and the error returned before access is granted (GatedRepoError ) does not explain that distinction. After access was granted, SPLADE took roughly 68 minutes to encode NFCorpus on CPU. BGE-base took about 4 minutes on the same collection. That is a real cost difference to plan around if you do not have a GPU. What this actually takes, in practice Reproducing BM25, dense retrieval, and SPLADE correctly, at real scale, on a normal laptop, is genuinely doable. It takes explicit memory tuning on the indexing step, one environment variable to work around a native library conflict, and clearing a gated model access request the documentation does not fully explain. None of these problems are exotic. They are common enough that anyone attempting this is likely to hit at least one of them, which is exactly why they are worth writing down instead of quietly working around and never mentioning. If you already run a retrieval or RAG system, try the dot-product check on your own index. Use your own scoring function and compare one query-document score by hand. It takes a few minutes, and it can catch scoring bugs before they reach production. That is the part worth carrying into any real retrieval project. Do not start by asking whether your RAG system feels good. Start with three checks: can your retriever beat a known baseline, does your metric match the product behavior you care about, and can you manually verify one query-document score? If you cannot explain one retrieved result, you cannot trust a thousand of them. There is a real open question underneath these numbers. BM25 needed no training data and still landed within roughly half the score of a model trained on hundreds of thousands of labeled examples. Does a hand designed, untrained scoring function still deserve a place in a modern retrieval stack, or is it still around mostly because it is cheap and familiar? The gap between 0.1874 and 0.3521 on MS MARCO suggests one answer. The gap between 0.3218 and 0.3808 on a ten times smaller, out of domain collection suggests a smaller one. Data and model licenses This article reports benchmark results from public research datasets and models. It does not redistribute any of the underlying data or model weights, and does not build or sell a commercial product using them. MS MARCO (Microsoft) is released for non commercial research purposes only, provided as is without warranty. The results here are reported only as a reproduction of a public research benchmark; this article does not redistribute the dataset or use it to build a commercial product or service. Readers should review the MS MARCO terms before using the dataset or any research outputs in products or services. The numbers reported here come from running published, standard research benchmarks on that dataset, the same practice followed by essentially every retrieval paper and blog post that cites an MS MARCO leaderboard result. NFCorpus is released under CC BY-SA 4.0, distributed via the BEIR benchmark, which permits reuse and adaptation with attribution and share alike terms. BGE-base-en-v1.5 (BAAI) is released under the MIT License, which explicitly permits commercial use. SPLADE version 3 (Naver) is released under CC BY-NC-SA 4.0, a non commercial license. It is used here only to compute and report a benchmark score, not redistributed, modified, or used as part of a commercial product or service. Glossary BM25. A keyword based scoring formula that ranks documents by matching query words, weighted by how rare and how frequent each word is. Needs no training data. MRR (mean reciprocal rank). A ranking metric that scores 1 divided by the position of the first correct result, averaged across queries. Higher is better; 1.0 is a perfect score. nDCG (normalized discounted cumulative gain). A ranking metric that rewards relevant results appearing near the top of the list more than further down. Higher is better. Dense retrieval. Search using embeddings, lists of numbers produced by a trained model, compared by dot product instead of matching exact words. Sparse retrieval. Search using vectors where most entries are zero and the nonzero entries correspond to specific words or word like units. BM25 is unsupervised sparse retrieval; SPLADE is learned sparse retrieval. Learned sparse retrieval. Sparse retrieval where a trained model decides term weights, instead of a fixed formula like BM25’s. Bi encoder. The shared architecture behind all three methods: one encoder for documents, one for queries, and a comparison function that scores the pair. HNSW (Hierarchical Navigable Small World). A graph based data structure used to search dense vector collections quickly without comparing a query to every single document. Embedding. The output of an encoder: a fixed length list of numbers meant to represent the meaning of a piece of text. Inverted index. A data structure mapping each word to the list of documents that contain it, the core structure behind keyword search engines like Lucene. Selected sources [1] J. Lin, “A Proposed Conceptual Framework for a Representational Approach to Information Retrieval” (2021), arXiv:2110.01529 [2] R. Nogueira and J. Lin, “Document Expansion by Query Prediction” (2019), arXiv:1904.08375 [3] N. Thakur, N. Reimers, A. Rückle, A. Srivastava, and I. Gurevych, “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models” (2021), arXiv:2104.08663 [4] C. Lassance, H. Déjean, T. Formal, and S. Clinchant, “SPLADE-v3: New baselines for SPLADE” (2024), arXiv:2403.06789 [5] S. Xiao, Z. Liu, P. Zhang, and N. Muennighoff, “C-Pack: Packaged Resources To Advance General Chinese Embedding” (2023), arXiv:2309.07597 [6] Castorini research group, “Foundations of Retrieval onboarding path,” github.com/castorini/onboarding [7] Castorini research group, Anserini “start here” documentation, github.com/castorini/anserini [8] Microsoft, “MS MARCO Passage Ranking Leaderboard,” microsoft.github.io/MSMARCO-Passage-Ranking-Submissions All commands, verification scripts, and the exact reproduced numbers referenced in this article are available in a companion repository.