What are the best real estate vector search benchmarks in 2026, and how do you actually evaluate semantic property search?

The Direct Answer: There Is No Off-the-Shelf Real Estate Vector Benchmark

If you came here looking for a single, canonical "real estate vector search benchmark" you can download and run, the honest answer is that none exists as of September 2026. Unlike general-purpose retrieval, where datasets such as MS MARCO, BEIR, and GLUE (introduced in Levy and Bowman's 2018 multi-task benchmark paper, arXiv:1804.07461) dominate evaluation, real estate semantic search has no equivalent standardized corpus. The closest development in the benchmarking world is Qdrant's release of Qdrant-FineWeb-10B in 2025 — a 10-billion-record dataset for vector retrieval benchmarking that uprated the typical benchmark dataset size by roughly 1,000x. That dataset matters to real estate teams because it finally lets you test vector databases at a scale closer to national property inventories, historical MLS archives, and document stores, rather than toy datasets of a few million rows.

Also worth reading: How Accurate Is AI Home Search, and How Should Buyers Evaluate It in 2026? · How Do AI Home Search Checklists Actually Work in 2026? · How Is AI Transforming Real Estate Matching Platforms Like RealTelligence?

What this means in practice is that anyone building AI-driven property discovery has to construct their own evaluation harness: a domain-specific query set, a labeled relevance corpus, and a set of retrieval metrics. That is more work, but it is also more honest. Generic benchmarks reward generic embeddings; real estate queries are vague, geographic, financial, and emotional all at once — "3 bed under $600k near good schools with a short commute" — and no public dataset captures that distribution well.

Why Real Estate Semantic Search Is Harder Than Generic Retrieval

The core difficulty is that property search queries are long-form, vague, and human in a way that keyword search was never designed for. Builders of semantic search for long-form content have documented this failure mode repeatedly: users describe outcomes and lifestyles, not attributes. A query like "somewhere quiet with space for a home office and a decent garden, about 40 minutes from the city" contains no tokens a traditional MLS filter can match. Vector search solves the matching problem but introduces an evaluation problem: how do you know the embedding model ranked the right homes first?

There is also a structural data problem. Real estate records — deeds, mortgages, liens, leases — are semi-structured documents that can be processed as JSON objects, but scanned lease documents and county recorder files require OCR and entity extraction before they can be embedded at all. If your pipeline garbles the square footage or the sale date, no amount of embedding quality will save the search experience. Evaluation therefore has to cover two layers: document extraction accuracy and retrieval relevance, and conflating the two is the single most common way teams misdiagnose poor search results.

Geography adds a third layer. Real estate sits inside the geographic information systems domain alongside urban planning, transportation, and public health, and pure vector similarity ignores spatial constraints. A home 0.3 miles outside a school-district boundary is semantically "near good schools" but practically wrong. Any benchmark you build must therefore combine vector similarity scores with hard geographic and attribute filters — a hybrid retrieval pattern, not a pure k-NN lookup.

What the 2025–2026 Benchmark Landscape Actually Offers

Three developments define the current state of play. First, Qdrant-FineWeb-10B: a 10-billion-record vector retrieval benchmark dataset released by Qdrant, covered by Business Wire and Blocks & Files, which uprated benchmark dataset size by 1,000x. Its value to real estate teams is infrastructure stress-testing — recall at scale, query latency under load, and memory behavior with billions of vectors. It tells you nothing about property-domain relevance, but it tells you whether your vector database will survive a national inventory.

Second, the continued relevance of NLU benchmarks like GLUE and its descendants for evaluating the language-understanding component of query parsing. When a user types a vague query, something must convert it into structured filters plus a query embedding; GLUE-style evaluation of that parser is a legitimate proxy test even though GLUE itself contains no property data.

Third, domain-specific AI systems have started publishing their own benchmark claims. In 2025, Leni — an AI system in the real estate data analytics space — reported topping four major AI benchmarks, outperforming systems from OpenAI, Anthropic, Google, and Perplexity in its evaluation category (per PR Newswire). Treat vendor-published benchmarks with skepticism: they are self-selected, the evaluation sets are rarely public, and "outperforming GPT-class models" on a curated real estate task says little about your specific query distribution. The right posture is to note the claim, then replicate the spirit of the test on your own data.

Building Your Own Benchmark: A Practical Step-by-Step

Start with query collection. Pull 500 to 2,000 real queries from your own search logs, or if you are pre-launch, synthesize them across the archetypes that dominate property discovery: budget-constrained, lifestyle-driven, commute-driven, investment-yield, and renovation-tolerance queries. Weight the set toward vague queries, because those are where semantic search earns its keep — the lessons published by teams building search for vague, human queries consistently show that vague queries are both the highest-value and the hardest to evaluate.

Next, label relevance. For each query, have domain raters mark properties as relevant, partially relevant, or irrelevant. Two raters per query with a disagreement rate tracked explicitly is the minimum bar; if raters disagree more than about 15–20% of the time, your relevance guidelines are too vague and the benchmark will be noisy no matter how good your models are.

Then define metrics. Use recall@10 and recall@100 for retrieval quality, nDCG@10 for ranking quality, and p95 query latency as your performance gate. A common threshold pattern: recall@100 above 0.90 before you even bother tuning ranking, because if the right answer is not in the candidate pool, no reranker can save you. Set a latency budget — for consumer property search, p95 under 300–500 ms for the vector stage is a reasonable target on 1–10 million property embeddings.

Finally, run ablations. Test at least three embedding models, with and without hybrid keyword fusion, with and without geographic filtering, and with chunk-level versus document-level embeddings for listing descriptions. Log every configuration. Teams that skip ablation end up cargo-culting embedding model choices from blog posts written for e-commerce catalogs, which have a very different query distribution.

Comparing Your Options: Vector Databases and Evaluation Approaches

The infrastructure choice matters less than vendors claim, but it is not irrelevant. Here is how the main paths compare for a real estate search workload in the 1–100 million vector range:

For most real estate teams, the pragmatic answer is Postgres with pgvector until recall or latency numbers force a move, because property records are already relational and PostGIS gives you the geographic filtering that pure vector stores handle awkwardly. Dedicated vector databases earn their keep past roughly 10–20 million listings or when you add document-level embeddings for leases, deeds, and market reports on top of the listing inventory. Note also the contrarian position published on HackerNoon arguing that many RAG systems do not need embeddings at all — for structured MLS data with clean numeric fields, a well-built filter engine plus a small semantic layer can outperform a full vector pipeline, and it is cheaper to run. Test that hypothesis before committing to a vector-first architecture.

Common Mistakes That Invalidate Real Estate Search Evaluations

The first mistake is evaluating on synthetic queries only. LLM-generated test queries cluster around the phrasing patterns of the model that made them, which inflates scores for embedding models trained on similar web text. Always mix in real logged queries, even if you can only label a few hundred of them.

The second is ignoring the extraction layer. If scanned deed and lien documents are OCR'd at, say, 92% character accuracy, and your benchmark only measures retrieval on clean JSON listings, you have measured a pipeline you do not actually run. Benchmark the full pipeline, including OCR error injection on the document path.

The third is single-metric optimization. Teams that optimize only nDCG end up with search results that are technically relevant but feel wrong — the top hit matches "quiet" but ignores the commute constraint entirely. Report at least two relevance metrics plus one constraint-satisfaction metric (what fraction of results actually meet the hard filters the user stated).

The fourth is benchmarking at the wrong scale. Running recall tests on 100,000 vectors and then deploying to 8 million listings produces latency and recall cliffs you never saw coming. The Qdrant 10-billion-record release exists precisely because the industry kept discovering this the hard way; scale your test set at least 10x beyond your expected production corpus.

The fifth is treating vendor benchmark claims as transferable. Leni's reported wins over OpenAI, Anthropic, Google, and Perplexity systems in 2025 are interesting signal about domain-tuned models, but a benchmark you did not run on data you do not own is marketing, not evidence.

When to Act, and What It Costs

If you are shipping a property discovery product in late 2026, the timing argument is straightforward: embedding models improved materially through 2025–2026, vector database scale limits are no longer the binding constraint for any realistic property corpus, and the differentiator has shifted to evaluation discipline. Teams with a labeled internal benchmark iterate on embeddings weekly; teams without one are stuck guessing. Budget two to six engineer-weeks to build the evaluation harness and label the first 500–1,000 queries — that investment is small relative to the engineering cost of the search system itself.

On cost: pgvector on an existing Postgres instance is effectively free at moderate scale. A self-hosted dedicated vector database for 10 million property embeddings with 768- to 1536-dimensional vectors runs roughly 30–60 GB of RAM at full in-memory settings, translating to a few hundred dollars per month in cloud costs. Managed vector services typically run $0.05–$0.30 per million queries plus storage, which at consumer property-search traffic volumes (say 1–5 million queries per month) means $100–$1,500 per month depending on the provider. Embedding API costs for a 1-million-listing corpus at roughly $0.02–$0.10 per million tokens run between about $20 and $200 for a full re-embed, cheap enough to re-embed whenever you switch models — so model lock-in is not a real excuse for staying on a weak embedding.

The honest caveat: none of this guarantees a better product. Semantic search in real estate can produce confident-looking but wrong results — surfacing a home "near transit" that is near a freight line, or matching "renovated" to listings whose descriptions merely contain the word. A benchmark built with real queries, honest labels, and constraint-satisfaction metrics is the only defense, and as of September 2026, building it yourself is not optional. It is the work.

Where This Field Goes Next

Two trends will likely produce a true public real estate retrieval benchmark within the next 12–24 months. The first is the scaling of public benchmark corpora — Qdrant's 10-billion-record FineWeb dataset shows that large-scale benchmark curation is now routine, and a domain-specific derivative for property listings is an obvious follow-on. The second is the growth of AI-driven property platforms (OLX Group, for example, reported in 2025 that real estate was among its strongest-performing verticals alongside motors and jobs), which creates both the incentive and the data volume for shared evaluation sets. Until then, the definitive benchmark for real estate vector search is the one you build against your own query logs — and the teams that treat evaluation as a first-class engineering artifact, not an afterthought, will be the ones whose search actually feels like it understands what a buyer means.