Stop Wrestling with Elasticsearch! SearchArray Brings BM25 to Pandas

Stop Wrestling with Elasticsearch! SearchArray Brings BM25 to Pandas

What if I told you that 80% of developers over-engineer their text search? You spin up Elasticsearch clusters, wrestle with complex query DSLs, and burn through infrastructure budgets—only to discover your "big data" problem is actually just 100,000 movie titles sitting in a CSV. Sound familiar?

Here's the painful truth: the Python^{↗ Bright Coding Blog} data stack has a massive gap when it comes to lexical search. You've got Pandas for manipulation, NumPy for computation, scikit-learn for ML... but the moment you need BM25 scoring or phrase matching, you're suddenly duct-taping together external systems that feel alien to your workflow. Your beautiful DataFrame pipeline? Shattered. Your mental model? Fractured across REST APIs and JSON query languages.

What if full-text search felt like... just another array operation?

Enter SearchArray—the open-source library that's making waves by embedding search engine capabilities directly into Pandas and NumPy. No clusters. No DSLs. Just .score() on your DataFrame column like it's .mean() or .sum(). This is the secret weapon top ML engineers are quietly adopting for hybrid search, reranking pipelines, and rapid prototyping—and once you see how it works, you'll wonder why we ever accepted the complexity of traditional search infrastructure.

What is SearchArray?

SearchArray is an open-source Python library created by Doug Turnbull (softwaredoug) that indexes arrays of strings into term indices for efficient BM25 and TF-IDF scoring. Its tagline says it all: "Full text search that feels like a numpy array."

Born from a frustration with how disconnected lexical search has become from modern data science workflows, SearchArray reimagines search indexing as a first-class citizen of the Python data stack. Instead of treating search as an external service you query via HTTP, it makes search an in-memory array operation you perform directly on Pandas Series, NumPy arrays, or any array-like structure.

The library is trending now because it arrives at a critical inflection point: the explosion of RAG (Retrieval-Augmented Generation) and hybrid search architectures. Data scientists building these systems increasingly need to combine vector similarity with lexical BM25 scoring—but they're forced to bolt on massive search engines for the latter. SearchArray says: what if you didn't have to?

With SearchArray, you prototype your ranking logic in pure Pandas, validate it against labeled data, and only then—if scale demands it—translate your proven approach to production search infrastructure. It's experimentation-first engineering, and it's changing how teams think about search relevance.

Key Features That Set SearchArray Apart

SearchArray isn't a toy project—it's a sophisticated search indexing library with capabilities that rival production search engines, all wrapped in a Pandas-native API:

• Term and Phrase Search: Pass a string for term search, or a list[str] for exact phrase matching with BM25 scoring
• Slop-based Phrase Matching: Pass slop=N for phrase queries with edit distance—directly mirroring Lucene's PhraseQuery behavior
• Raw Statistics Access: Peek under the hood with termfreqs and docfreqs methods for custom scoring algorithms
• Pluggable Tokenization: Bring any tokenizer matching def tokenize(value: str) -> List[str]—stemmers, custom preprocessors, whatever Python ecosystem offers
• Memory Mapping: Handle larger-than-RAM datasets by passing data_dir for memory-mapped index storage
• Custom Similarity Functions: Pass any Python function to compute similarity scores—see the built-in similarity implementations
• DataFrame-native Scoring: Score entire DataFrames, enabling seamless combination with recency, popularity, or other ranking signals
• Solr edismax Compatibility: Includes a Solr edismax query parser implementation for familiar query syntax

The killer insight? SearchArray scores everything, always. This means you can multiply BM25 scores by temporal decay factors, blend with neural reranker outputs, or build boolean query combinations—all within standard Pandas operations. No context switching. No serialization overhead.

Real-World Use Cases Where SearchArray Dominates

1. Rapid Search Relevance Experimentation

You're tuning a search system and need to test whether phrase matching outperforms term disjunction for movie title queries. With SearchArray, you iterate in a Jupyter notebook, compute NDCG@10 against your labeled judgments, and commit the winning configuration—all before lunch. No deployment pipelines, no index rebuilds in staging clusters.

2. RAG Pipeline Hybrid Retrieval

Your RAG system retrieves candidates via vector similarity, but you know BM25 catches exact entity matches that semantic search misses. SearchArray lets you load your corpus into Pandas, compute both vector and lexical scores in the same DataFrame, and learn the optimal fusion weights—experimentation that would require three separate systems traditionally.

3. Top-N Candidate Reranking

Your Elasticsearch cluster returns 1000 candidates efficiently. Now you need to rerank with a complex model combining BM25, click-through rate, and freshness. SearchArray slurps those results into a DataFrame and lets you express the reranking logic in pure Python—no Painless scripting, no function query arcana.

4. Offline Evaluation and Dataset Construction

Building training data for a learned ranker? SearchArray lets you compute feature vectors (BM25 scores for multiple fields, phrase match flags, term proximity statistics) directly on your annotation dataset. The termfreqs and docfreqs arrays become features in your XGBoost or LightGBM model.

Step-by-Step Installation & Setup Guide

Getting started with SearchArray is deliberately trivial—it's designed to eliminate infrastructure friction, not add to it.

Basic Installation

# Standard installation via pip
pip install searcharray

# For development or bleeding-edge features
pip install git+https://github.com/softwaredoug/searcharray.git

SearchArray has minimal dependencies: it builds on NumPy and Pandas, with optional support for memory-mapped storage via standard library modules.

Environment Setup

# Core imports you'll use in every SearchArray project
import numpy as np
import pandas as pd
from searcharray import SearchArray

# Verify installation
print(SearchArray)  # Should show module path, no errors

Creating Your First Index

# Load sample data—any array-like of strings works
df = pd.DataFrame({
    'title': [
        'The Shawshank Redemption',
        'The Godfather',
        'The Dark Knight',
        'Pulp Fiction',
        'Fight Club'
    ]
})

# Index the column—this is where the magic happens
df['title_indexed'] = SearchArray.index(df['title'])

# Inspect what was created
print(df['title_indexed'].iloc[0])
# Output: Terms({'The': 1, 'Shawshank': 1, 'Redemption': 1}, ...)

Notice the default whitespace tokenization is intentionally simple—you bring your own tokenizer for production use. This design choice keeps the core library lean while leveraging Python's excellent NLP ecosystem.

Custom Tokenizer Configuration

# Example: case-insensitive tokenization with stemming
from nltk.stem import PorterStemmer

stemmer = PorterStemmer()

def smart_tokenizer(text):
    """Lowercase, split, and stem tokens."""
    return [stemmer.stem(token) for token in text.lower().split()]

# Apply custom tokenizer during indexing
df['title_stemmed'] = SearchArray.index(
    df['title'], 
    tokenizer=smart_tokenizer
)

Memory-Mapped Indexing for Larger Datasets

# For datasets approaching RAM limits, persist index to disk
import tempfile

with tempfile.TemporaryDirectory() as tmpdir:
    df['title_mmap'] = SearchArray.index(
        df['title'],
        data_dir=tmpdir  # Memory-mapped backing store
    )
    # Search operations remain identical API
    scores = df['title_mmap'].array.score('fight')

REAL Code Examples from SearchArray

Let's dive into actual code from the repository, with detailed explanations of what's happening under the hood.

Example 1: Basic Phrase Search with BM25 Scoring

This is the canonical SearchArray example—indexing a Pandas column and searching with phrase matching:

from searcharray import SearchArray
import pandas as pd
import numpy as np

# Assume df['title'] exists with string data
df['title_indexed'] = SearchArray.index(df['title'])

# Search with a phrase (list of tokens)
# This computes BM25 scores for the exact phrase "cat in the hat"
scores = df['title_indexed'].array.score(['cat', 'in', 'the', 'hat'])

# Sort scores to find top matches
sorted_scores = np.sort(scores)
print(sorted_scores)
# BM25 scores:
# array([ 0.        ,  0.        ,  0.        , ..., 
#        15.84568033, 15.84568033, 15.84568033])

What's happening here? The .array.score() method operates on the underlying SearchArray extension array. Passing a list[str] triggers phrase search mode—it looks for documents where these terms appear in this exact order with proximity scoring. The zeros are documents with no match; the ~15.85 peaks are strong phrase matches. This is pure BM25 scoring with term frequency, inverse document frequency, and field length normalization—all computed vectorized via NumPy.

Example 2: Term Search and Top-N Retrieval

# Single string = term search (OR of tokens by default)
raw_scores = df['title_indexed'].array.score('Cat')

# Get ranking indices (ascending, so we flip for top matches)
ranked_indices = df['title_indexed'].score('Cat').argsort()

# Retrieve top 10 matches using standard Pandas iloc
top_n_cat = ranked_indices[-10:]
top_matches = df.iloc[top_n_cat]

print(top_matches[['title', 'title_indexed']])

Critical insight: Notice how df['title_indexed'].score() returns a Pandas Series aligned with the DataFrame index, while .array.score() returns the raw NumPy array. Both work—you choose based on whether you need Pandas alignment or raw speed. The argsort() gives you document positions by BM25 relevance, letting you slice the original DataFrame with zero friction.

Example 3: Combining Search with Business Logic

This example showcases SearchArray's core philosophy—search is just another column operation:

from datetime import datetime
import pandas as pd

# Assume df has timestamp and title_indexed columns
now = datetime.now()

# Calculate recency feature in hours
df['hrs_into_past'] = (now - df['timestamp']).dt.total_seconds() / 3600

# Combine BM25 with recency for a hybrid score
# No special syntax—just Pandas arithmetic!
df['score'] = df['title_indexed'].score('Cat') * df['hrs_into_past']

# Sort by combined score
df_sorted = df.sort_values('score', ascending=False)

Why this matters: In Elasticsearch, you'd write a function score query with decay functions. In Solr, you'd craft function queries. In SearchArray? You write Python. Your data scientists already know this. Your relevance engineers can iterate without reindexing. The mental model is unified with the rest of your ML pipeline.

Example 4: Custom Tokenizer Implementation

# Default whitespace tokenizer (shown for reference)
def ws_tokenizer(string):
    """Dumb whitespace split—minimal, fast, often insufficient."""
    return string.split()

# Enhanced lowercase tokenizer
def ws_lowercase_tokenizer(string):
    """Case-normalized tokens for case-insensitive matching."""
    return string.lower().split()

# Apply to indexing
df['title_indexed'] = SearchArray.index(
    df['title'], 
    tokenizer=ws_lowercase_tokenizer
)

# Now 'CAT', 'Cat', and 'cat' all match the same postings

Design philosophy exposed: SearchArray intentionally avoids baking in tokenization. Python's snowballstem and NLTK ecosystems exceed Lucene's capabilities. By accepting any str -> List[str] function, SearchArray lets you simulate—and surpass—production search engine tokenization without leaving Python.

Advanced Usage & Best Practices

Pro Tip 1: Precompute Multiple Field Indexes

For multi-field search, index each field separately and combine scores with learned weights:

df['title_idx'] = SearchArray.index(df['title'])
df['overview_idx'] = SearchArray.index(df['overview'])

# Learn these weights via coordinate ascent or gradient descent
title_weight, overview_weight = 2.0, 0.5

df['combined_score'] = (
    title_weight * df['title_idx'].score(query) +
    overview_weight * df['overview_idx'].score(query)
)

Pro Tip 2: Use Memory Mapping for Service Reranking

When reranking in production APIs, initialize once with data_dir and reuse:

# At startup: load or build memory-mapped index
search_idx = SearchArray.index(corpus, data_dir='/app/search_index/')

# Per request: fast memory-mapped access, minimal RAM
scores = search_idx.score(user_query)

Pro Tip 3: Extract Features for Learned Ranking

The raw statistics methods enable feature engineering:

# Get term frequency matrix for custom feature computation
tf = df['title_idx'].array.termfreqs('search_term')
df = df['title_idx'].array.docfreqs('search_term')

# Use as features in XGBoost/LightGBM ranker

Optimization Strategy: Batch Queries

For offline evaluation, score multiple queries against the same index using Pandas operations rather than Python loops—vectorization is your friend.

SearchArray vs. Alternatives: Why Make the Switch?

The verdict: Choose SearchArray when your problem is experimentation, prototyping, or reranking within the Python data stack. Stick with Elasticsearch for billion-document scale or when you need distributed querying. Use rank-bm25 only if you need the absolute simplest possible BM25 without phrase support or Pandas integration.

Frequently Asked Questions

Is SearchArray production-ready for high-traffic APIs?

SearchArray targets reranking and experimentation, not primary retrieval at massive scale. For 1000s of documents in API reranking, it's excellent. For millions of QPS, use it offline to validate approaches before implementing in Elasticsearch/Solr.

Can I use SearchArray with scikit-learn or PyTorch?

Absolutely! The NumPy array output from .score() drops directly into sklearn feature matrices or PyTorch tensors. It's designed for ML pipeline integration.

How does SearchArray handle tokenization for non-English text?

Better than most alternatives—you pass any tokenizer function. Use jieba for Chinese, MeCab for Japanese, or any custom preprocessor. You're not limited to built-in language support.

What's the memory overhead of SearchArray indexes?

Significantly leaner than loading text into sparse matrices. The memory-mapped data_dir option lets you exceed RAM. For precise numbers, benchmark with your tokenizer and document distribution.

Can SearchArray replace my Elasticsearch cluster?

Not directly—it's a complement, not replacement. Prototype in SearchArray, validate with labeled data, then implement the proven approach in your production search cluster. Some teams use it permanently for offline evaluation pipelines.

Does SearchArray support fuzzy matching or synonyms?

Phrase slop (slop=N) provides edit-distance phrase matching. Synonym tokenizers with overlapping positions are on the TODO list. For now, preprocess synonyms into your tokenizer.

How do I get help or contribute?

Join the #searcharray channel on Relevance Slack for community support. The project welcomes contributions, especially on efficiency improvements and query parser extensions.

Conclusion: The Future of Lexical Search is Array-Native

SearchArray represents a fundamental shift in how we approach text search in Python. By embedding BM25 indexing directly into Pandas and NumPy, it eliminates the artificial boundary between "search infrastructure" and "data science workflow." No more context switching. No more query DSLs that feel like programming in a foreign language. Just .score() on your DataFrame and move on.

For ML engineers building RAG systems, relevance engineers prototyping ranking features, and data scientists who've been told they "need Elasticsearch" for problems that fit in RAM—SearchArray is your permission slip to simplify.

The library is actively developed, well-documented with Colab notebooks, and backed by a welcoming community on Relevance Slack. The codebase is clean, the goals are focused, and the philosophy—small data expressiveness over big data scalability—is refreshingly honest about the problems most of us actually face.

Ready to stop over-engineering your search? Head to github.com/softwaredoug/searcharray, pip install searcharray, and index your first column. Your future self—prototyping ranking models in a single Jupyter notebook—will thank you.

Have you tried SearchArray? What's your experience with hybrid search in Python? Drop a comment or join the discussion on Relevance Slack—let's build better search together.