Document Intake and Embedding for Semantic Search -- Part 2

tl;dr:
I implemented semantic search over a collection of documents I had handy. In short, it works too well! My benchmark is saturated; every model does nearly perfectly. Now I want to find out where they do differ in performance.

Why?

I’m interested in knowledgebase systems.

What is a knowledgebase?

If you ask me, a knowledgebase is simply a collection of documents and some sort of index mechanism which allows them to be selectively retrieved by their relevance to a given objective. Critically, “a collection” implies curation.¹ 

By this definition, a library might count as a knowledgebase,²  although the index is somewhat coarse-grained. Let’s consider how you might use a library to further develop an idea about what a theoretically ideal knowledgebase might look like.

Efficiently Retrieving Information from a Knowledgebase

At this point, if you’re familiar with LLMs but don’t happen to know about embeddings, you might think of scanning the full content of every book with instructed chat sessions. You could just chunk the book into sections, run a chat thread over each chunk, and have them all return essentially a classification: “relevant or not-relevant”.

This would work! Supposing there were 20 books of 50k words each, that’d be approximately 1 million tokens. Running this scan would cost you something like $1, at 2025 prices.⁴  It might take a while, depending on how bursty your inference provider is willing to let you be.⁵ 

A more efficient strategy for semantic search might be to use text embeddings. You could think of this as (very approximately) like a similar LLM “reading” all those same chunks of text and then placing them into an index based on what they’re about. Instead of an alphabetically-sorted keyword index like you might find in the back of a reference book, these indexes are more like a multi-dimensional “map”.⁶

In this case, the index only needs to be created once, and then it can be used for multiple, different queries. Creating the index involves an LLM, and is computationally intensive, though it’s more efficient than dumping documents into chat threads. In practical terms, you might imagine that all those Dewey Decimal areas above contain 5,000 books, or 250 million words. This amount of material could be indexed for ~$25 in AI inference costs. Then, if your typical query was ~100 words, you could run 100,000 queries for another $1 in AI inference costs. Amortized, this is like $0.0003 per query.⁷ 

Running queries like this is also fast - you can pretty easily have results in less than a second!

What I’m Working On

I’ve continued working on my knowledgebase experiments. Now that I have a collection of documents with which I have at least a passing familiarity, I’m well positioned to run queries on them and assess the vibes.

The flow looks like this:

• I email myself a URL, possibly with some metadata.
• The URLs are fetched, PDFs are rendered for reading, and the text content is stored.
• The documents are broken up into chunks.
• Embedding vectors are computed for the chunks.

The highlights here are that it’s very easy to experiment with different chunking strategies and with different embedding models. They’re basically pluggable.

If I wanted to, for example, experiment with context-enriched chunking⁸, I could just write the new chunker and specify a config that uses it. The new table will be automatically created in Postgres and it’ll automatically be populated with chunks on the next batch run. Adding a new embedding model works similarly. Once a new embedding configuration is registered, the schema will be automatically updated and all the embeddings will be computed on the next run. The benchmark script will also be aware of the newly registered config.

Benchmark Saturation

Okay, so my current benchmark is just some queries that I noted down while I was reading some of the articles I mailed myself. I tried to write my queries such that they did not include any of the same high-entropy words as the passages/chunks I hoped they would retrieve, in order to stress the semantic representations.

In short, all the models, even the smallest, retrieve exactly what my rubric expects. So, if I want to learn about the relative capabilities of these models, I’ll have to think of something else to try.

Why is my benchmark saturated?

I think I chose decently tricky queries that should only be answerable when there’s some degree of concept modeling present (rather than queries that you could just answer by doing text search, or a fancy text-search with a thesaurus). I expected some to fail, but none did. Why? Are these embedding models just far better than I’d expected?

One possibility is that there simply aren’t enough documents embedded in my database and so there’s very little possibility for confoundment. That is to say, maybe these queries are actually casting a very wide net through semantic space, but there’s such a sparse distribution of passages in that space that even a wide net will still manage to catch the right passage (and even rank it as the best one), because there is nothing else to get swept up in the net.

For example, I have one article in my collection that describes a text-based web browser that uses AI to rewrite pages. When I search for “an app that lets me look at websites with a TUI”, well, there’s only a handful of articles in my collection that even mention browsers heavily in the first place. So, even if the recall was relatively imprecise, it might still get the correct answer! It could be that my collection of documents is sufficiently small that something like this happens for all of them.

Scaling Up the Collection

It might be interesting to try indexing some substantially larger collections of documents. However, this creates difficulties for a vibes-based assessment: ideally I have actually read all of these documents so that I have a good sense for what is present in the collection and what degree of confoundment might be expected.

I can go and get a large corpus of material, but if I don’t read all of it, I cannot know if a given query returns all of the results that I would reasonably have expected it to return. Certainly I can evaluate the results that are returned and judge their relevance to the query, but I have no idea how many relevant results are not being returned.

Anyway, more documents seems like the best next move. Even if I can’t be as rigorous as I’d like, I can still learn something about how recall performance is affected by the amount of material in the collection.

Some Other Ideas: Knowledge-graph Traversal

I include a document ID and document sequence number in my chunks tables because the sequence number turns the chunks into a very simple graph. That is to say, you could follow the sequence number up or down to get additional context around any given passage. Or an LLM could make a tool call to request this if you were doing RAG and the passage retrieved was relevant but insufficient.

I’ve been thinking about also including references to other documents in the metadata if, for example, reading one somehow otherwise prompts me to discover the other. That is probably too labor-intensive to track, though. Another possibility which leverages my own judgements less (but which is accordingly easier), would be to just parse links in documents. These could theoretically be crawled automatically. Of course that eventually covers the whole Internet, so there’d need to be some rules to limit the depth, and I don’t know how much bandwidth I’d want to spend on that anyway.

Wikipedia has plenty of internal links and is available as a bulk download, so using their data could sidestep these issues and might be a quicker way to create a toy knowledgebase for exploring graph features.

Code

Source on GitLab.