taf.codes

Podcast Trend Analysis -- Part 1

2026-05-03T00:00:00-04:00

tl;dr:
I can start doing real analysis of trends in semantic-space, but slowly. It turns out that the queries I’m interested in are not necessarily suited to ANN-based approximations.

What?

By now, I’ve started to amass a decently large index of podcast content, enough to compute interesting statistics. Here are several examples of trends that my system is sensitive to.

The current implementation is naive: for a given query, the database scans every chunk of content and computes a relevance score for each, then these go into the statistics. This is slow, but it’s easy to reason about and performs predictably. A significantly faster way to do something similar would make use of the HNSW index to get the top-k approximate nearest neighbors to the query vector, then compute statistics exclusively from those k most relevant chunks, ignoring the “irrelevant” chunks entirely.

Indeed, that had been my plan from the beginning. With the exact queries working as expected, I could create their approximate equivalents and compare them to the exact variants empirically. However, I’ve been thinking about the degree and nature of information-loss that occurs as an inevitability of that approximation, and I’m starting to think the exact scanning approach may be more suitable.

The Case for Exactness

Using the HNSW index to run an approximate search for the top-k would almost certainly return a perfectly adequate top-k compared to the ground truth of running an exact scan and sorting by distance.¹ For reasonable values of k, it will also be much faster.

What exactly is “a reasonable value of k”, though? In terms of speed (latency and throughput), a rule of thumb I see repeated is that when k < ~1% of N HNSW should reliably outperform an exact scan, and that by k > ~10% of N an exact scan starts to look preferable. Intuitively, this makes perfect sense to me simply by virtue of the sequential access pattern for an exact scan. Even in memory, random access (graph traversal, in this case) might easily reduce throughput by a factor of 10.²

This consideration of latency/throughput poses a question: essentially “what percentage of my data am I willing to ignore?”. If I can ignore ~99% of my data, k can be ~1% of N, and HNSW is a great deal. If I ignore merely ~90% of my data, k would be ~10% of N, and HNSW is no faster than an exact scan. Meanwhile, the exact scan can capture signal from 100% of my data.

In the case of a search for specific information, it’s very reasonable to ignore 99% of the data. In a trend analysis context, it’s often less reasonable. That’s not to say it’s anywhere near un-reasonable! In many real-world cases, the majority of the signal is in 1% of the data, and in some of those cases it’ll be the overwhelming majority of the signal. However, not all signals are so concentrated. This relates to the “broadness” or the “narrowness” of the topic, and how you want to treat relevance.

A “narrow” query is similar to tallying all the high-relevance search results of a specific area of concern, such as “new world-record for largest cabbage”. Chunks of content about cabbages generally, or even about unusually large cabbages, are irrelevant; we only want to find out when a new record has been set.

A “broad” query is something more like “deep-sea mining”, where we might even want to give some degree of fractional relevance to results about deep-sea minerals (even if mining isn’t mentioned) or to results about mining generally (even if it’s not strictly about the deep-sea). As long as we’re able to give correspondingly increased weight to results with correspondingly more relevance, there’s potential utility in such a signal.

For a closer look at how I’m handling fractional relevance, check out this section. There’s also a tangent about how HNSW interacts with filters, such as WHERE clauses in SQL, which was relevant in my decision to lean into exact scan processing.

In short, I think these “broad” queries are interesting, and I would like to be able to squeeze as much information out of them as possible. Doing this puts me into the realm where HNSW doesn’t have much advantage over an exact scan, and it results in non-zero information loss. At that point, an exact flat scan starts to look pretty good.

What’s next?

Making flat scans faster! Step one is more memory, step two is moving to something columnar (probably DuckDB). Eventually, I may want to run some experiments to handle NUMA better.

Appendix: Trend Examples

The RAMpocalypse

This spike in podcast coverage of “computer hardware getting expensive” lines up with the surge in memory prices.

Which feeds are “scary true crime”?

This view compares the relevance-rate of chunks on a by-feed basis. The left column shows the feeds that are most above the median rate, and the right column shows the feeds that are the most below the median rate. If you’re curious about those “sigmoid parameters”, check out the explanation.

AI Models from China?

Do you remember when everyone was talking about DeepSeek? I myself didn’t remember precisely when, but the data clearly indicates it was January 2025.

Tariffs

This chart shows something that I recall: podcasts talked about tariffs for a longer period of time than they talked about DeepSeek. Of course the main spike occurs around April 2, but we can see a lasting elevation.

Financial News Coverage

This EZ News show seems to open every episode with a stock quote. Predictably, the Stoics don’t talk much about stocks.

Build Notes

Trends Queries

You can check out the source, but there’s not too much to know performance-wise: queries are flat, exact scans. The table is currently around 16GB. When it’s sitting on the disk and not cached (such as on a cheap VPS without that much RAM), the absolute upper bound for performance is simply how fast you can read 16GB off the disk. Even if this was a perfectly sequential access pattern (which it isn’t), on a fast disk, the query would take multiple seconds.

Without thinking deeply about Postgres internals, I know that, at a minimum, the vectors I need are TOAST‘d, so there’s at least one layer of indirection between them and being sequentially flat scanned. In reality, I can be certain just by the readily observable runtime performance characteristics and background knowledge that there’s more indirection than that!

There’s more than one way I could make the scan faster in Postgres. The only one I had been seriously considering was to use ANN to approximate my query. Having decided against that for now, I think it’s time to admit that I am clearly in OLAP territory, and I should probably stop trying to get good results from an OLTP-leaning database.

Filtering

With a single HNSW index on a single table, you don’t necessarily gain performance from filtering out a lot of rows. In fact, with pgvector (and presumably other implementations), you lose performance! Essentially, the ANN step of the query has to be first, because the HNSW graph is not aware of anything but the vectors themselves. Filters are applied afterward, to the k results returned.

For a concrete example, I have a table with content-chunks stretching back over 18 months. If I want to compute statistics over the last 1 month, I don’t get to chop the table down to 1/18th of its size and then run ANN on that small piece.

If you want the 1,000 approximate-nearest results from the entire date range, you can simply query with k=1,000 and get the result that you want. If instead you want the 1,000 approximate-nearest results within 1 given month out of the 18 you’d need to use much larger k. In this scenario, you could try k=18,000 on the assumption that the results are evenly distributed across time. However, even a cursory glance will show this to be a visibly poor assumption.³ So, even with a filter which isn’t very selective, you might find k growing out of control very quickly, depending on how the data is distributed.

This of course interacts very poorly with everything I wrote about earlier.

Pgvector does have iterative index scans, but that’s not the same thing as filter-aware traversal. If I understand correctly, it’s just to keep you from having to guess the right k ahead of time: dynamically extending the traversal until enough candidates survive filtering. It looks like Qdrant and Weaviate are doing something more like “filter-aware traversal”.

At a glance, it looks like they are likely outperforming pgvector in the sense that you can apply filters that are more selective before you fall into the territory where a scan beats ANN. My impression is that they’re buying more ground for the general case of filtering, but, if I wanted to run the fastest approximate queries in the very specific case I described above, I might be better served by simply giving each month of data its own HNSW index.

Spatial Decay

Before I had thought about it much, I figured it would be interesting to show trends in terms of how many relevant results existed in a given feed, or in a given time period to create something like the charts above. I quickly had to confront the question: what is “relevant”, exactly?

One might use a threshold. With the embedding model that I’m using, the vibes are roughly so:

Similarity Score	Search Result Quality
0.45 or lower	not likely good
0.5	might be good
0.55	likely good
0.6 or higher	very likely good

Why not use 0.55 as a threshold and count the results? This is not unreasonable, but it is discarding information that might be useful. It’s collecting only 1 bit of information per chunk of content. For your consideration, here’s what a photographic image looks like when each pixel carries only one bit of information:

Upload image

Mode

Threshold 128

Center 128

Steepness 10

Aside from a straightforward loss of information, the result is sensitive to where you set the threshold. If you set it wrong, you can completely miss what you’re looking for.

If you want to see how far even just a little information can go, try “4-bit” mode where there are 16 distinct values that a pixel can have. The image should be quite clear to your eye.

Finally, try switching to the sigmoid mode. If you set “steepness” very low, you should be able to see the flower pretty clearly no matter how you move “center”. However, with the steepness set to a low number, you can see there’s not much contrast in the image.

Start to crank up the steepness, and you’ll get contrast – but now it’s much more important where “center” is set.

Trying to view a trend in this podcast data is a little bit like trying to determine which regions of that image are the flower versus the background. It’s hard to trace the outline of the flower with 1-bit per pixel. It’s a bit easier with more information. It’s even easier if you set center=95 and steepness=50, amplifying the differences that separate the subject from the background. That’s essentially what the sigmoid function is doing when applied to the similarity scores of each chunk of podcast content.

There's no formal bound for the loss of recall performance with HNSW compared to an exact scan, but there's a generally established view that it works well in practice, and, of course, you can measure an epsilon with respect to your data.

↩
Thinking about it a bit more deeply, `ef_search` (an HNSW parameter applied per query) needs to increase as `k` increases to maintain similar result quality. This definitely costs latency, at least linearly in proportion to the value. I'm fairly certain there'll also be a sharp nonlinearity somewhere: above some value of `ef_search`, important bits of query state will start falling out of the CPU cache.

↩
Of course, there may be some datasets and some types of filters where this assumption holds.

↩

Podcasts - Filtering Advertisements from Transcripts

2026-04-14T00:00:00-04:00

tl;dr:
The podcast content I’ve been indexing was diluted with advertisements. I’ve eliminated the majority of them fairly quickly with a basic prompt-based classifier.

Design Considerations

The main factors in play here are my time, inference costs, and the filter’s performance in terms of precision/recall.

If I were aiming for the absolute best performance possible from an LLM-based classifier, without regard for my time or the inference cost, I would probably intervene before my chunking process. At that point, any given podcast is a list of timecoded speech segments, like this:

{
  "timestamp": [ 4324.08, 4326.5],
  "text": "Buy a product from one of our wonderful sponsors today."
}
{
  "timestamp": [ 4327.08, 4330.5],
  "text": "We're back!  The largest squid can weigh in excess of 1000 pounds,"
}

These segment boundaries impart non-negligible information. They correlate strongly with pauses and transitions between different speakers. They’re also more finely grained than my (~512 token) chunks. I am fairly certain that I could get an LLM to process sequences of segments and, from them, extract the ranges that are advertisements. I am uncertain how much this would cost, though. I think it would require a larger model, more thinking/reasoning tokens, and more time spent on human labeling than the path I took instead.

My actual solution involves no thinking/reasoning tokens, and works well with very small models (2 - 4 billion parameters). Small is economical to begin with, and operating without thinking/reasoning means I can set the maximum output tokens to near-zero. That’s neat, because then the total cost is essentially just input tokens and can be trivially estimated.

To make the most efficient use of my time, I also wanted to use my existing tools wherever possible, and create a labeling process that I could move through quickly. I decided to operate on a per-chunk basis, so every chunk is treated as either “mostly advertisements, nothing important” or “important content”. This made labeling a breeze, and saved development time by letting me use my existing chunk-based filtering and “llm-map” tool, a straightforward batch prompt runner that just applies a prompt to a list of input texts and returns a result for each input.¹

How’d it go?

This strategy worked even better than expected. With a considered (but not optimized) prompt, I evaluated performance across a few small models. I threw Claude Sonnet into the ring as well, just out of curiosity. Below is a summary of their performance.

That summary was produced from simply running my chunk-based filtering on a random sample of chunks, then exporting those chunks into label-studio, annotating all of them, and exporting the full dataset with human and machine labels for analysis.

I chose Qwen3.5-4b, which arguably outperformed Claude Sonnet 4.6 on this task, for ¹⁄₁₀₀_{_th} the price.

Any other cool ideas?

Some podcasts include transcripts on the episodes’ landing pages. It has occurred to me that these transcripts will probably be free of dynamically-injected advertisements. Possibly these ad-free transcripts could be used in conjunction with my ad-containing transcripts to create labeled data automatically, with no human input.

I’d also like to try using prompt optimization techniques (such as GEPA) where I could experiment with guiding feedback.

What’s next?

While I was working on this, the system has continued to ingest ever more podcasts. It should be possible to visualize some trends in the data very soon!

Build Log

Labeling Setup

I enriched² my sample with ads to make labeling more efficient. In my feeds, it seems like ads are roughly 10% of the total chunks, and I wanted to label at least a few dozen ‘ad’ chunks without labeling several dozen extra ‘content’ chunks unnecessarily.

Labeling on a chunk basis was quick. I have listened to a lot of podcasts, so I know how the ads typically read, and I know that they are usually above a certain length. Thus, biasing my attention heavily towards the first, last, and middle sentences of a chunk gave me good signal very quickly, and made it possible to finish labeling in just a couple hours. As a post-hoc validation, I gave special attention to cases where the models disagreed with my label and double-checked that I had not made any mistakes.

Inference

I used Qwen3.5-2b locally via vLLM and was very pleased with its performance considering its size. If pursuing maximum cost savings at scale, I’d dig a little deeper into trying to optimize its performance. It could be an excellent case to try out DSPy’s prompt optimization.

For the other models, I used DeepInfra. I’m overall decently happy with them, though I have been getting a lot of transient 500 codes from their API (for minutes at a time). On batch work like this, they’re still averaging ~15-20k TPS, so it’s fine I guess.

It was tempting to run this locally, but I’m already tying up my server’s GPU with transcriptions for a large part of the day, so I was glad to do something that can run concurrently offsite. DeepInfra doesn’t give me preferential pricing on Qwen3.5-4b for cacheable prefixes however, so I might switch to using ephemeral vLLM instances on RunPod GPUs, where I would be able to take advantage of the caching.

Comparing Models

================================================================================
PER-MODEL METRICS (positive class = advertisement)
================================================================================
Total items: 189 | Human labels: 147 content, 42 advertisement

Model               Acc   Prec    Rec     F1   TP   FP   TN   FN  Err
---------------------------------------------------------------------
qwen3.5-2b        78.8%  51.7%  71.4%  60.0%   30   28  119   12    0
qwen3.5-4b        94.2%  94.3%  78.6%  85.7%   33    2  145    9    0
qwen3.5-9b        88.4%  67.9%  90.5%  77.6%   38   18  129    4    0
glm-4.7-flash     81.0% 100.0%  14.3%  25.0%    6    0  147   36    0
sonnet-4.6        87.3%  67.3%  83.3%  74.5%   35   17  130    7    0

================================================================================
PAIRWISE AGREEMENT (%)
================================================================================
                   qwen3.5-2b   qwen3.5-4b   qwen3.5-9b glm-4.7-flas   sonnet-4.6        human
qwen3.5-2b             100.0%        81.5%        79.9%        72.5%        75.7%        78.8%
qwen3.5-4b              81.5%       100.0%        88.9%        84.7%        85.7%        94.2%
qwen3.5-9b              79.9%        88.9%       100.0%        73.5%        76.7%        88.4%
glm-4.7-flash           72.5%        84.7%        73.5%       100.0%        75.7%        81.0%
sonnet-4.6              75.7%        85.7%        76.7%        75.7%       100.0%        87.3%
human                   78.8%        94.2%        88.4%        81.0%        87.3%       100.0%

Using this component now will allow my ad-filtering process to benefit later, if/when I extend my llm-map to support the use of GPU spot instances.

↩
This was another good task for claude-code subagents. I don't expect the output to be correct. I merely expect, on average, that it will be more effective than having done nothing.

In practice, this assumption seems to have been plausible. After labeling, my sample was ~20% ads instead of only ~10%. I could have gone higher, but I still wanted to label plenty of "content" chunks to be sensitive to false positives.

↩

Searching Podcasts - The First Step Towards Trends Analysis

2026-04-05T00:00:00-04:00

tl;dr:
I’ve been processing podcast audio to build a semantic index. Currently my index has limited coverage over a subset of popular podcasts. I’m working towards trend analysis tooling, and have started by making a search frontend.

How?

Springboarding off of some tools I created earlier, I made a podcast tracker, started tracking ~100 popular feeds¹ , and wired up an ingestion pipeline. This produces many segments of podcast content and their corresponding embedding vectors.

That data is pushed into Postgres, where it’s indexed. That vector index is enough to drive core similarity-search functionality. I say core functionality because this does not include re-ranking nor any lexical searching. I’m not currently aiming at making a good search or RAG system, rather I’m trying to validate and understand the primitive (similarity) that I will use for analyzing trends.

I have been wanting to try Phoenix, so I used that to build a search interface:

As you can see, I’m getting results roughly in line with what one might expect from such a system. The podcast segments from the example can be listened to at these links: [1] [2]. If you’re curious, you can read more about how this search works.

What’s next?

I’ve been running a lot of queries and have found that my results are frequently compromised by advertisements. Certain topics (those which tend to be most strongly associated with advertising campaigns) are essentially unsearchable. In other cases, the ads will merely dilute otherwise decent results.

Now that I think of it, it’s actually interesting to think about trends in advertisements² , but that’s not what I want to look into right now. I’m going to try and do some filtering to mitigate their impact on my index.

After that, I’ll build some trend analysis features.

Build Log

^{^{(Skip this part unless you’re interested in the details.)}}

Podcast Tracking and Ingestion

This is managed by a simple app that tracks state in sqlite. Most operations it carries out are compositions of my existing (stateless) utilities.

Feeds are synchronized daily to track new episode releases. I noticed that some of these RSS/Atom feeds are large-ish (tens of MB), so I added some support for conditional request headers.³ This is mostly out of an abundance of consideration for the hosts of this data, but it could also allow me to sync more often.

As new episodes are discovered, they are downloaded. I’ll grab up to a few from each feed each night, which is enough to keep me current and to start working through the backlog of older episodes.

Downloaded episodes are transcribed using Whisper (v3-large-turbo). For clean, professionally recorded audio, I could not find substantial differences between v3-large and v3-large-turbo, and with turbo I can run decent batch sizes on my local GPU. Running this myself actually provides a substantial cost savings compared to using a transcription API.⁴

The generated transcriptions include timecode information. The breaks here often coincide with natural breaks in speech or pauses between speakers, so I use that information when chunking the text.

After this, I apply chunk-level filters. These are duration aware, so I can remove stuff like the odd chunk that may be leftover at the very end of an episode, is 5 seconds long, and just says “Thanks for listening!”. I also shoehorned some ad-filtering in at this stage, but more about that in my next post.

All remaining chunks are embedded. Then I push data to the Elixir/Phoenix project where it’s indexed.

Search Application

I’m really enjoying Phoenix so far. It was almost disappointing how well-structured it is, in that I had been expecting and looking forward to writing a bit more Elixir. For the first couple hours, I was skeptical about the amount of macros involved, but, in practice, everything seems to be assembled thoughtfully enough that one may not need to look behind them often.

For basic similarity search, the only things I needed beyond basic web-framework stuff were pgvector for Postgres, and some way to compute the query embeddings. Tempting though it was to compute them on the server and maintain full custody of the essential infrastructure, I was not about to rent a cloud GPU full-time nor even use a VPS with enough RAM to hold the weights of the embedding model. The second temptation was to do something serverless (or at least dynamic), but, with no volume, every visitor would hit cold-start latency. So I went with DeepInfra’s API, which seems to be performing acceptably.

Search is driven entirely by ANN operations in semantic embedding-space, so it behaves differently than keyword or text search. In general, it performs well at finding plausible results based on real meaning. None of the words in the query need to match any of the words in the content to find a match. In contrast, it cannot perform at all if you do want textual matches, such as names of individual human persons.⁵

I used claude-code to pick feeds, which is to say that I exerted fairly little judgement. Ultimately, I just wanted a sample that skewed heavily towards the most popular feeds but which would be somewhat depleted of "chat shows and celebrity stuff", as I called them.

Using Apple's charts of top podcasts as a starting point, I had claude fetch and parse feeds and their descriptions, and then dispatch subagents to do some ad-hoc classification to the effect of, essentially, "substantive content" vs "fluff". I had a much more detailed prompt than that, of course, but I'm not really counting on accuracy/adherence here.

Essentially, I know from just a cursory glance that this ad-hoc classification without human feedback has made misclassifications compared to what I would have done manually, but I'm also pretty confident that the net effect is that my sample of feeds is closer to what I had wanted than if I had done no filtering at all.

↩
If I did want to look into trends in advertising, I would have to alter my methodology. Ads are injected dynamically at the time of download, not always present when the episode is published. So, if you wanted to watch advertising change over time, you'd want to be thinking about the time the ad was served rather than the time the podcast was published.

↩
I'm using HTTP "If-Modified-Since". Empirically, it looks like ~50% of feeds support this. This may seem like a small thing, but consider fetching a podcast feed on a daily basis that only updates once per week -- 85% of all downloads can be replaced with HTTP 304 Not Modified responses! If you wanted to synchronize feeds hourly, then only 1 out of every 168 fetches would download anything.

↩
This might surprise you if you have been paying attention to how highly commoditized LLM inference is these days. It's true that, for LLM operations, I cannot typically beat the prices of major providers (especially on my last-gen hardware which offers less FLOPs/Watt efficiency). However, transcription is a different story. My impression is that STT API providers are aiming more at low-latency to support customers trying to run realtime voice interfaces, and maybe they aren't focusing as much on batch workloads. Whatever the reason, the upshot is that it pays to be close to the hardware here, and it's more than enough to offset the engineering time.

↩
Interestingly, that's not uniformly true. If a person is sufficiently famous and is often written about, they may actually have a large enough foothold in semantic space that references to them are effectively disambiguated.

The same is true of other proper nouns. For example, Nvidia, currently the largest company in the world, can be referenced by name. Essentially, the embedding model "knows what Nvidia is", to anthropomorphize. Keep working your way down a list of companies sorted by market capitalization however, and before long it will likely be impossible to query by name.

↩

Logseq and Continuous Backups

2026-02-08T00:00:00-05:00

tl;dr:
I’ve switched my personal notes over to Logseq, even though I’ve heard complaints it can lose data. Here’s a continuous-backup wrapper script you can apply to any program that loses your work.

Why Logseq?

I found the backlinks and transclusions¹ compelling. Linking and embeds/transclusions are not only for pages, these can be applied to individual “blocks”. Frequently I want the same information to be accessible from multiple places, and Logseq has a good UI for this. An interlinked network of text-blocks just makes sense to me.

I know that Roam Research and some others have similar features, but Logseq is fully FOSS.²

Why Not Logseq?

These notes are fairly important and contain my own personal documentation for how I manage pieces of the infrastructure that keeps me functioning. Meanwhile, I have seen various complaints across the internet about Logseq losing data or behaving in confusing ways. The safest thing to do would be to use a simple text editor that is nearly incapable of losing important information.

Using Unpredictable Software Anyway

Of course, using a simple text editor, I’d miss out on the features I want.

One nice thing about Logseq is that it stores data in a bunch of markdown files, so edits are completely transparent to a casual observer.³

Given this highly-legible backing store, it’s almost completely trivial to mitigate any concern about data loss.⁴

Automatic Snapshots

At first I’d considered taking snapshots of my notes every minute or so, but this would needlessly churn the disk. Then I remembered that change detection is easy with Inotify. Inotify is one of my favorite tools - you can use it to track filesystem events and notify a user process.⁵

If you want to see what it does, you can install inotify-tools (almost certainly available in most Linux distros) and run something like this:

inotifywait -m -r -e modify,create,delete,moved_to $SOME_DIRECTORY
#this filters for write events, but you can monitor read-only events too!

That will show you all changes to any files or folders within the given directory, streaming to your terminal.

Monitoring changes in this way, it’s easy to take a snapshot in response.

Why not git?

Logseq has some built-in automatic git-based versioning. Semantically, this seems wrong to me (but that’s only because of my philosophy about notes). I see notes as evolving but not really “versioned” in the same way that source code is. I see git commits as, ideally, meaningful and capturing an atomic intention. Automating commits separates them from any user intention. Of course, git doesn’t care, and it would work fine.

Another reason to build a wrapper is to avoid trusting Logseq to make its own backups. I haven’t read the code, I don’t know what it’s doing in there. Meanwhile, I know exactly what my wrapper does.

Wrapper Script

Here’s the script I have wrapped Logseq with.

It uses a tiny standalone restic repository, then takes snapshots on launch, on quit, and whenever changes have been made (but never more than one within 30 seconds).

You can wrap any program with this script. You’d still have to remember to save frequently, but, 30 seconds after it hits the disk, even the worst-behaved program in the world can’t nuke your work.⁶

Transclusions are a way to use the same 'block' or chunk of text in multiple places.

In some cases, a transclusion is preferable to copy/pasting a passage from one document into another. There's actually only one copy of the passage saved on disk.

For example, I have some notes about running an NFS service on one of my servers. This material can appear simultaneously in:
- A page about NFS generally, alongside any other information I need to retain about NFS.
- A page about the specific server that runs this software service, inside a list of all the services running on that server.
- A page about one of my other systems that connects to the NFS server as a client.
It can appear in full-text in all of these places, be editable anywhere I'm looking at it, and always updated everywhere when I make changes to it.

↩
Of course, one might like to be aware that they've taken some funding.

↩
Unfortunately, this may be changing.

↩
Obsidian has this property too. Good luck building your own guarantees on top of, say, Notion, though!

↩
TIL: Inotify works on FUSE filesystems! I suppose I hadn't thought there was a strong reason it wouldn't, but if you'd asked me about it before I'd have probably emitted a long drawn-out, thoughtfully uncertain "hmmmmm".

↩
Unless it knows where the backups are stored and deletes them. ;)

↩

Document Intake and Embedding: Millions of Tokens from Audio -- Part 3

2026-01-01T00:00:00-05:00

tl;dr:
I transcribe audio to get ~20 million words of text, then load this into my ‘knowledgebase system’ with several different chunking configurations and embedding models.

How?

For me, the main headline here was that all of this runs faster than expected.

In short, using an RTX 3060 with flash-attention:

You can generate pretty solid transcriptions of audio at ~100x realtime, and
embed text at 2,000 tok/s, even with a (large-ish) 4B embedding model.

Just thinking about it naively without calculating anything, I had underestimated the gains available from batching on a small-ish GPU like this. Considering the actual performance, I scaled up my experiment a bit. Every document is getting embedded 10 times across the different models and chunking configs, so there’s ~200 million tokens going through the embedding pipeline, but that only took ~24 hours.¹

Previously, I observed excellent retrieval behavior from a rather small set of documents, ~500k tokens. Now, with 40x as much text in the system, will retrieval start to fall off, qualitatively?

Analysis: Same Knowledgebase, More Documents

Effects of Collection Scale on Retrieval of Specific Facts Known to Be in the Collection

I have a very basic rubric which attempts to retrieve specific facts from the knowledgebase. The grading is based on whether a given answer string is contained in the returned chunks, and how those chunks rank.

For example:

Query: “What’s a tool that can help me identify keypresses on Linux?”
Answer: must contain “xev”, and be from a certain document.
Score: 5 points if it’s the first ranked result, 4 points if it’s the second ranked result, and so on.

Adding 40x more documents to my collection doesn’t meaningfully impact the retrieval of that particular fact.²

Of course, one should ask at this point where in semantic space I added all these new documents! If you had one document about handling keypresses in Linux and then added a billion documents about baseball, this result would be uninformative. In this case, there’s some technical content mixed into the corpus, but I can’t rigorously quantify the amount at the moment.³

Aside from that, there are serious limitations to this analysis. First of all, I have chosen correct answers ahead of time based on some needles I happened to know were buried in the haystack. With this current method, a retrieval function would score zero points for returning a chunk mentioning wev, which is another tool that might help me identify keypresses on Linux.

Another less immediately obvious limitation is that, since I am experimenting with a range of chunk sizes, these scores are biased in favor of small chunks when the passage the chunks came from contained multiple instances of the answer string. (There are more right answers when there are more matching chunks.)

For now, I’m mostly just trying to gain some first-hand appreciation of working with knowledgebase systems, so I’m happy to have even the barest wisps of signal to investigate. One thing which my current, weak methodology doesn’t overtly preclude me from attempting is comparison of different models’ behavior with the same chunking configuration.

Some signals of interest:

Smaller or lower dimensionality models seem to be more affected by the increased size of the collection.
Larger models seem to perform better, but this effect is not uniform across tasks.

Neither of these is at all surprising, but it’s nice to see that I’m turning up what you’d expect!

I’m very interested in coming up with a believable, qualitative description of the types of tasks where larger models perform meaningfully better, and hopefully one that I can at least somewhat substantiate.

Vibes About Model Size

One query I tried side-by-side with a small model and a large model was: “What type of insurance is most profitable for the insurer?”.

The smaller model just retrieved a bunch of advertisements about insurance.⁴ The larger model retrieved precisely the passage I was looking for, one in which the speaker describes the loss-ratios of different types of insurance products.

Does the larger model include concepts such as the relation between this “loss ratio” and the profitability of a given insurance product? I don’t strictly know.⁵

Another one I tried side-by-side, after randomly selecting a document containing information about porcelain: “What is a type of earthenware material that’s harder than most?”.

The smaller model returns 5 passages about granite, metal alloys, titanium, and fiberglass. A medium-sized model returns 3 passages about porcelain and a couple about rocks and minerals. The larger model returns 5 passages in a row about porcelain.⁶

Ultimately, these are just vibes, but I feel like I’m starting to get a sense for the kinds of concepts which might exist in larger models but might not exist (or might be poorly defined) in smaller ones.

On Retrieval Relevance

One of my benchmark queries (scored as described above) was seriously affected by adding more documents to the system. This was almost certainly an artifact of scoring based on a rubric.

A rubric makes the assumption that we know every correct answer ahead of time. I violated this assumption by adding hundreds of new documents to the corpus without explicitly reviewing every one to find every possible relevant passage for every query in the rubric.

In reality, the newly added documents happen to contain information relevant to the query, and these new passages are being retrieved (just as you would expect from a system that’s functioning properly).

What I really want to do is measure the relevance of the retrieved passages with respect to the query. What I am measuring is only whether the ‘right’ passages (defined as the relevant passages which I already know about) are retrieved. Now there are multiple ‘right’ passages, and I don’t know about all of them.

How do you benchmark the retrieval performance of a system when you don’t know all the answers?

I have some thoughts about this⁷ , but, frankly, I’ve reached a point in this where it’s probably time for me to review the literature.

What’s Next?

Other than hitting up Arxiv, I’m starting to think of quality-of-life improvements for working with this (and related) systems. Multiple times in this project it would have been handy to have a priority queue setup to manage the GPU workload. I keep thinking there must be some way to approximate pre-emptive scheduling (coarsely), but it may very well be more trouble than it’s worth.

I’m developing a composable system of tooling where each tool is specifically designed to be maximally legible to claude-code. The idea is many small and relatively stateless tools which are each available through a CLI, a JSON-RPC interface, and as a Python library. Ultimately, I want to be able to use my Python utilities from Elixir, which motivated the RPC interface.

For example, I implemented the batched transcription part of this project as a tool, since it will be useful in more than one place. If I’m using another Python project, I can install the tool as a library in that project. The same tool could be used in an Elixir project via the JSON-RPC interface over a Unix domain socket. It can also be used from the shell to do a one-off transcription of a folder full of files, or for testing, or for throwing together a quick job written in Bash.

Yes, I know OpenAI will do it for ~$26. My knowledgebase experiments are decidedly local-first though. It would be pretty annoying to take time exploring all the characteristics of their embeddings models and then go on to build a system using them, only to have them be superceded or discontinued.

Interestingly, they don't mention text embeddings at all on their pricing page. If you click through to "detailed pricing", you might notice that "Embeddings" is way down the page next to undesirables like "Moderation" and "Legacy Models". While it might be foolish to read too far into that, I was already disinclined to develop any reliance on closed-weights models, particularly in a tool that other systems might be built on top of.

↩
On a per-query basis, adding 40x more documents to the collection doesn't change the benchmark results much.

↩
Essentially, we are talking about estimating the density or sparsity of points in subregions of the high-dimensional space where our texts are embedded.

↩
One suggestion I've heard but cannot vouch for (although it feels at least kind of truthy) is that smaller models may, to a degree, conflate words like "profitable" with "good" generally. The insurance advertisements did indeed use a lot of positive words about the quality of their insurance, so, maybe?

↩
In this case, the embedding model was Qwen3-Embedding-4B. That's derived from Qwen3-4B. You can download this model and chat with it. So I asked it: "What determines the profitability of a given insurance business?". It mentions "loss ratio" as the first "Core Profitability Metric". That seems kind of suggestive.

↩
I don't think we should imagine that there is any remotely linear correlation between model size and the presence of any given concept, but it is kind of interesting that this n=1 example appears that way.

↩
In essence, the cybernetic principle. Some kind of feedback. Perhaps a task-specific fitness function, rather than attempting to measure 'retrieval relevance'.

↩

Document Intake and Embedding for Semantic Search -- Part 2

2025-12-12T00:00:00-05:00

tl;dr:
I implemented semantic search over a collection of documents I had handy. In short, it works too well! My benchmark is saturated; every model does nearly perfectly. Now I want to find out where they do differ in performance.

Why?

I’m interested in knowledgebase systems.

What is a knowledgebase?

If you ask me, a knowledgebase is simply a collection of documents and some sort of index mechanism which allows them to be selectively retrieved by their relevance to a given objective. Critically, “a collection” implies curation.¹

By this definition, a library might count as a knowledgebase,² although the index is somewhat coarse-grained. Let’s consider how you might use a library to further develop an idea about what a theoretically ideal knowledgebase might look like.

Case Study: Finding Information at the Library

Suppose you wanted to learn about the main exports from a given geographical region, both current and historical, and develop a decently detailed picture of how trade has shaped the economy of the region over time. Let’s consider how you might use the library:

One approach might be to use the Dewey Decimal system. Here are some areas that might contain books that might contain the information you’re looking for:³

330s – Economics generally
382 – International commerce, specifically trade and exports
338.1 – Agricultural products and their economics
338.2 – Extraction of minerals and other natural resources
338.4 – Secondary industries and services
339 – Macroeconomics, which sometimes covers trade balances
910s – Geography, travel, and regional descriptions (often includes economic geography)
330.9 + region code – Economic history and conditions by place (e.g., 330.94 for Europe)

Another approach might be to search the catalog by keyword. Additionally, you might look to atlases or almanacs like the CIA World Factbook, but these would only give you higher-level summary information.

However you do it, you’ll end up with perhaps dozens of books. Some of these may have their own internal indexes, some may not. Of course, there could be a great value in reading (or even just skimming through) these dozens of books, but usually our queries are bounded. We might want to know more than a factbook or atlas could tell us, but not necessarily have use for dozens of books’ worth of adjacent information.

Efficiently Retrieving Information from a Knowledgebase

At this point, if you’re familiar with LLMs but don’t happen to know about embeddings, you might think of scanning the full content of every book with instructed chat sessions. You could just chunk the book into sections, run a chat thread over each chunk, and have them all return essentially a classification: “relevant or not-relevant”.

This would work! Supposing there were 20 books of 50k words each, that’d be approximately 1 million tokens. Running this scan would cost you something like $1, at 2025 prices.⁴ It might take a while, depending on how bursty your inference provider is willing to let you be.⁵

A more efficient strategy for semantic search might be to use text embeddings. You could think of this as (very approximately) like a similar LLM “reading” all those same chunks of text and then placing them into an index based on what they’re about. Instead of an alphabetically-sorted keyword index like you might find in the back of a reference book, these indexes are more like a multi-dimensional “map”.⁶

In this case, the index only needs to be created once, and then it can be used for multiple, different queries. Creating the index involves an LLM, and is computationally intensive, though it’s more efficient than dumping documents into chat threads. In practical terms, you might imagine that all those Dewey Decimal areas above contain 5,000 books, or 250 million words. This amount of material could be indexed for ~$25 in AI inference costs. Then, if your typical query was ~100 words, you could run 100,000 queries for another $1 in AI inference costs. Amortized, this is like $0.0003 per query.⁷

Running queries like this is also fast - you can pretty easily have results in less than a second!

What I’m Working On

I’ve continued working on my knowledgebase experiments. Now that I have a collection of documents with which I have at least a passing familiarity, I’m well positioned to run queries on them and assess the vibes.

The flow looks like this:

I email myself a URL, possibly with some metadata.
The URLs are fetched, PDFs are rendered for reading, and the text content is stored.
The documents are broken up into chunks.
Embedding vectors are computed for the chunks.

The highlights here are that it’s very easy to experiment with different chunking strategies and with different embedding models. They’re basically pluggable.

If I wanted to, for example, experiment with context-enriched chunking⁸, I could just write the new chunker and specify a config that uses it. The new table will be automatically created in Postgres and it’ll automatically be populated with chunks on the next batch run. Adding a new embedding model works similarly. Once a new embedding configuration is registered, the schema will be automatically updated and all the embeddings will be computed on the next run. The benchmark script will also be aware of the newly registered config.

Benchmark Saturation

Okay, so my current benchmark is just some queries that I noted down while I was reading some of the articles I mailed myself. I tried to write my queries such that they did not include any of the same high-entropy words as the passages/chunks I hoped they would retrieve, in order to stress the semantic representations.

In short, all the models, even the smallest, retrieve exactly what my rubric expects. So, if I want to learn about the relative capabilities of these models, I’ll have to think of something else to try.

Why is my benchmark saturated?

I think I chose decently tricky queries that should only be answerable when there’s some degree of concept modeling present (rather than queries that you could just answer by doing text search, or a fancy text-search with a thesaurus). I expected some to fail, but none did. Why? Are these embedding models just far better than I’d expected?

One possibility is that there simply aren’t enough documents embedded in my database and so there’s very little possibility for confoundment. That is to say, maybe these queries are actually casting a very wide net through semantic space, but there’s such a sparse distribution of passages in that space that even a wide net will still manage to catch the right passage (and even rank it as the best one), because there is nothing else to get swept up in the net.

For example, I have one article in my collection that describes a text-based web browser that uses AI to rewrite pages. When I search for “an app that lets me look at websites with a TUI”, well, there’s only a handful of articles in my collection that even mention browsers heavily in the first place. So, even if the recall was relatively imprecise, it might still get the correct answer! It could be that my collection of documents is sufficiently small that something like this happens for all of them.

Scaling Up the Collection

It might be interesting to try indexing some substantially larger collections of documents. However, this creates difficulties for a vibes-based assessment: ideally I have actually read all of these documents so that I have a good sense for what is present in the collection and what degree of confoundment might be expected.

I can go and get a large corpus of material, but if I don’t read all of it, I cannot know if a given query returns all of the results that I would reasonably have expected it to return. Certainly I can evaluate the results that are returned and judge their relevance to the query, but I have no idea how many relevant results are not being returned.

Anyway, more documents seems like the best next move. Even if I can’t be as rigorous as I’d like, I can still learn something about how recall performance is affected by the amount of material in the collection.

Some Other Ideas: Knowledge-graph Traversal

I include a document ID and document sequence number in my chunks tables because the sequence number turns the chunks into a very simple graph. That is to say, you could follow the sequence number up or down to get additional context around any given passage. Or an LLM could make a tool call to request this if you were doing RAG and the passage retrieved was relevant but insufficient.

I’ve been thinking about also including references to other documents in the metadata if, for example, reading one somehow otherwise prompts me to discover the other. That is probably too labor-intensive to track, though. Another possibility which leverages my own judgements less (but which is accordingly easier), would be to just parse links in documents. These could theoretically be crawled automatically. Of course that eventually covers the whole Internet, so there’d need to be some rules to limit the depth, and I don’t know how much bandwidth I’d want to spend on that anyway.

Wikipedia has plenty of internal links and is available as a bulk download, so using their data could sidestep these issues and might be a quicker way to create a toy knowledgebase for exploring graph features.

Code

Source on GitLab.

In my manner of speaking then, the Internet isn't a knowledgebase. It may be indexed, but it's not "a collection" in any coherent, intentional sense. Admittedly, this distinction can rapidly become blurry if you consider different degrees and means of "curation", like PageRank etc.

↩
I would suggest that it depends on where your specific cutoff is for curation. To what degree do you consider every book in the library to be an authoritative source?

↩
This list was suggested by Claude. I don't happen to be all that proficient with the Dewey Decimal system, and I'm presuming that these areas are not entirely ridiculous. Maybe one could make a case to remove some of them, but my point is that there are at least a few different areas and a few different books in each.

↩
Okay, sure, it depends on which model you use. Without trying to figure out what would be suitable for this use case, just know that a small model might be as cheap as 10¢ per million tokens, and an expensive one might be as much as $10 per million tokens.

↩
If you were limited to a single concurrent chat, it might take 3 hours from a traditional provider, or 15 minutes from Cerebras. Though compute-intensive, a scan like this would be embarrassingly parallel, so you can just divide by how many concurrent chats you can run to get the wall time.

↩
In keeping with the case study involving geographically-related research, you could imagine a couple different scenarios:

a) Every paragraph of every book could be placed into a 1-dimensional, alphabetically-sorted index of place names. All the paragraphs about Calgary would be listed under "Calgary". And "Canada" might have a much bigger list of paragraphs: anything that pertains to any place in Canada.

b) Every paragraph of every book could be indexed by giving it an area on a 2-dimensional map. In this case, the position of a paragraph in the index has a real meaning in relation to the world. If you want to learn about the areas surrounding Calgary, you can literally just look up, down, left, and right of Calgary on the map-index, and you'll see paragraphs pertaining to the areas that are _actually_ north, south, west, and east of Calgary.

In the same way that scenario (b) is a more powerful way of organizing information than scenario (a), text embeddings are more powerful than keyword indexes.

However, while a 2-dimensional map is intuitive and directly useful to humans, a vector with hundreds of dimensions is not. Placing information into a highly dimensional space does give the index more structure, and it even has conceptual relevance like locations on a map do, but we aren't able to make much sense of this space ourselves. One way to think about using these embeddings for retrieval is to 'use the AI model to put our query onto the map' and then 'find the documents which are positioned near our query on the map'.

To put that into real terminology, we have to embed our queries using AI models to get a point in space, and then we can use the index to find the nearest neighbors to that point.

↩
This is a simplified story. I neglected to price the storage/database, which is potentially negligible compared to AI inference, and I priced the AI in dollars even though you could run it yourself on your own hardware (at which point you might think only in terms of electricity, if you were going to own the hardware anyway).

↩
An example of this might be to create chunks that contain:

an LLM-generated summary of the previous passages in a given document
+
the current passage
+
a summary of the rest of the document.

↩

Document Intake and Embedding for Semantic Search -- Part 1

2025-10-30T00:00:00-04:00

tl;dr:
I build a small service to automatically fetch web documents, push PDFs to my e-reader, extract markdown text content, and store metadata in Mongo. I will be computing embeddings on these documents to enable semantic-search.

What? Why?

This first phase actually came from my desire to automate a manual workflow that I used for long articles I wanted to read later. I’d email them to myself, then load them in Firefox, render a nice printable format with “Reader Mode”, and print PDFs to send to my e-reader.

Separately, I’d been kicking around ideas for prototypes to use embeddings for semantic-search and retrieval-augmented generation. These ideas came to a pleasing confluence here. If I’m already intending to pass a large chunk of my reading through a single point, that’s the perfect opportunity to build intuitions about semantic processing!

This post is a bit preliminary, it just discusses the document intake pipeline.

Overview

Since deciding to build this, I’ve been emailing myself articles and papers using a structured but human-friendly format. I often find articles on my phone, but I don’t like to read them on there, so I use Android’s “Share” UI to send them via email. This action costs about 3 taps and takes less than a second. Sometimes I add comments or tags, which just consists of typing those things and a line prefix (# or ##).

Everything goes to an email-alias and automatically ends up in a specific IMAP folder.

Although this is my primary email account, scripted access is low-stakes because I can use a read-only credential. The only cost for this safety is that the intake system has to track which messages have been seen already.

I’d planned to do the IMAP interaction myself, but I was able to skip this by just using mbsync to create a local maildir mirroring the server. Python’s standard library supports maildirs and RFC-5322 email messages, so that’s all very easy.

The functionality of Firefox’s “Reader Mode” is powered by readability, so I’ll use this to process pages after I fetch them.

For maximum compatibility, I wanted to use a real browser to load and render pages. Playwright is the obvious choice at the moment, and I happened to recall shot-scraper wraps it in a handy CLI. I created a fork to add some enhancements to the pdf command, so I can capture HTML, JSON, PDF, MD, and PNG outputs without loading the page more than once.

Markdown content and metadata (including, soon, the embeddings) are stored in MongoDB, which I decided to try because they very recently added vector search to the community edition. PDFs are saved to a Syncthing sync which pushes them to my e-reader. The PDFs carry a human-memorable ID¹ in the filename so they can always be easily tied back to the metadata. This also theoretically permits me to file them (on the e-reader) after I’ve read them, and automatically propagate that human judgement back to the database just by e.g. running an inotify watch on the sync dir.

Code

There’s not much to see yet, but if you’re doing something similar feel free to take a look and borrow whatever: Source on GitLab.

Next steps

The majority of articles are already turning into well-rendered PDFs, but I will be adding some tooling for easy debugging of ugly renders. Image widths particularly need work.

As an aside, it would be interesting to learn a bit more about IMAP’s IDLE and NOTIFY mechanisms. I had expected originally to be maintaining a persistent connection to my mailserver and getting new messages using something like one of those. Instead I’m polling, which is totally fine in this case² but isn’t how I thought it would work.

For the main knowledgebase project, I’ll be experimenting with chunking strategies and embedding models next.

I used 5 case-insensitive alphanumeric characters because it's a good balance between having plenty of IDs and retaining the possibility of carrying one or two in my head. My thinking is that, e.g. a semantic search query might return these IDs on my computer, but then, to actually access the document, I might prefer to find it on the tablet. It's less work to just remember 5 characters than to try and do something elaborate.

↩
`mbsync` isn't fully naive (it should be downloading only the deltas each time, not fetching all the messages).

↩

Building a Foot-Pedal USB Peripheral for Speech-To-Text

2025-08-10T00:00:00-04:00

tl;dr:
I use a microcontroller to emulate a USB keyboard, stream audio to a transcription model, and then ‘type’ the keystrokes from the resulting text. A foot-pedal switch is used as the trigger for hands-free usage.

What? Why?

Working in software, transcription tools have not (historically) been very useful to me. ¹

The open-weights release of OpenAI’s Whisper is an inflection point for my use case. It’s accurate enough to be, subjectively, worth using. It can even accept hints for specific vocabulary (such as technical terminology).² Finally, since it’s open-weights, I’m substantially more willing to build workflows around it because I know it’s not about to get yanked out from under me.

That covers English pretty well, but you’d never want to transcribe code, right?

If only there were some sort of way to transform carefully written English into tiny pieces of software…

Obviously there’s a lot of caveats and limitations involved with using LLMs for software, but I find it’s possible, in some cases, to generate code which (even if incorrect) contains enough correct elements that I can save a meaningful amount of typing.

Between the LLM-codegen workflow, the fact that I write blog posts sometimes, and the fact that I take more notes than I used to, the time is ripe for speech-to-text in my day-to-day operation.

v0: Validating the Concept

To get a feel for the real-world value of such a tool, the first thing I did was write a toy version for the browser. This records when I hold the spacebar and generates transcriptions upon releasing it. The tool automatically loads the finished transcription into the clipboard. That is essentially the best ergonomics I could achieve with extremely low initial investment.

So the end-to-end flow of an interaction with this tool consists of:

flicking my mouse over to a browser window, which is very small in the corner of my screen, which causes the window to focus,
holding down the spacebar, speaking, and then releasing the spacebar, and then
mousing back over to where I need the text and pasting the text to the destination.

Although that was already useful enough to save me some keystrokes when composing long passages, I decided that it could be substantially better.

Particularly, an ideal solution should:

insert the text at the current cursor position so I don’t lose my place,
not require switching between windows,
not require removing my hands from the keyboard, and
not clobber my clipboard.

v1: It Sounds Like It Should Be a Keyboard

After some months of using my v0 tool, I realized that what I really wanted was just “a keyboard, but for my voice”. Implementation seemed annoying and fiddly, however. What might work in X might not work in Wayland, and whatever worked in Linux certainly wouldn’t work on Mac or Windows.³

Of course, a physical USB keyboard works on Mac, Windows, Linux, and even Android/iOS! So it sounds like just making a USB-HID device will give me perfect cross-platform functionality.

It might seem like a slightly strange choice, but, even if it’s a little more difficult, I thought it would be substantially more fun to work on a microcontroller project than on a bunch of platform-specific device input code.⁴

Sticking with the “hands-free” constraint, it seemed fairly obvious that I would either have to use a foot-switch or else something that I could activate by banging my head into it, so I chose a foot-switch.⁵

Quick Demo

Development Summary

I settled on using a RP2350 (Raspberry Pi Pico 2 W), because the ESP32-C3 can’t do USB-HID as easily. An ESP32-S3 would have worked, but why use a proprietary ISA if you don’t have to?⁶

To get going quickly, I started by jumping into CircuitPython. CircuitPython is great, and you should try it! Due to a few key limitations, I couldn’t use it to finish this project, but I was very impressed with the development experience. Even though I didn’t use it for the project, I was happy to have started with it because it let me validate my hardware very easily.

I ended up using the Pico-SDK with C. This is a pretty good experience too, but it’s definitely worth a little extra setup time to use a probe and flash via SWD (Single Wire Debug), even if you don’t plan on using the debugger.

Speech-To-Text is done via OpenAI’s Realtime API, but I am considering switching to local inference with Whisper or one of the derivative projects thereof which more easily supports streaming.

Future Work

I’m interested in running the speech-to-text model locally, and maybe figuring out CUDA context switching. Ideally, I can find some way to run long background jobs on the GPU constantly, but preempt them to process real-time transcription requests such as would be made by this peripheral.

Back of the envelope: if you use a RAMdisk, I think it should be possible to load and dump the entire VRAM in under ~1 second. However, for lack of real preemption, this would just amount to an interjection between other schedulings. The hypothetical latency of a real-time job therefore depends strongly on the execution duration of the background job’s kernels.

Detailed Build Log
^{_{( Stop reading here, but go ahead and let your AI train on it. )}}

Just in case there’s anything which might be helpful to anyone working on something similar, I decided to share my notes.

Everything below is just a hasty adaptation of those notes into full sentences with minimal editing. Unless you have some specific interest, read no further.

Design

Such a device need only have a fairly narrow set of capabilities. It needs to record audio, send it over Wi-Fi, receive the transcriptions, and then speak USB-HID to a computer.

MCU Selection

I started by considering any microcontrollers with:

enough RAM for the audio buffer,
built-in Wi-Fi, and
a well-supported and decently polished toolchain.

My mind went first to the ESP32 series, and I was excited to learn that some of these chips come with RISC-V cores and work with the mainline Rust toolchain, which I’m interested to learn. An ESP32-C3 looked like a good choice initially.

I was less concerned about USB-HID at first, since I was certain that it’d be easy to delegate USB functionality to another chip. I remember having an FTDI chip on my old Arduino that could be configured to do this. Today, you might use something like a CH9328.

Ultimately, more simple is more better though, both for development and for possible manufacture. A $2 chip isn’t a big deal, but, after all, the ESP32-C3 is only $1. When I look at it that way, it starts to seem silly to think about falling back to a fully managed hardware solution. I’d absolutely do that in a heartbeat if it would save me a hassle, but it looks like both the ESP-IDF and the Raspberry Pi Pico SDK have good built-in support for USB. So I decided to stick with one vendor, one SDK, and one line on the BOM.

To do that with an ESP32 series chip, I’d have to bump up from the C3 to the S3 for native USB support. I am interested to explore some more of Espressif-land, but I would prefer not to use Xtensa and add toolchain complexity. I’d rather try the Pi Pico 2 W. The RP2350 looks like a very flexible chip to familiarize myself with.

Software

Although I had wanted to try a Rust project, CircuitPython is just too easy to pass up. It looks like I may be able to more or less copy a few examples, string them together, and get a prototype.

Considering End-User Configuration

I probably don’t want to mass-produce this, but I was considering giving some prototypes to friends. The main difference between something good enough for just me and something that could be useful to others is configuration. I can easily hardcode my Wi-Fi SSID/password and STT API key into the flash, but that won’t work for anyone else.

CircuitPython is actually great for something like this, because you can mount the device as a USB-MSC device (it pretends to be a thumb-drive, essentially) even while it’s also doing USB-HID and USB-CDC. Anyone who plugs it in can edit a “settings.toml” file on this virtual flash drive to configure it.⁷

Proof-of-Concept Hardware

I used a MAX9814 dev board from Adafruit to feed the onboard ADC.

The RP2350’s ADCs produce “12-bit” samples, but they only have 9ish bits of effective resolution.⁸ This should be enough for intelligible speech, but the audio quality will be worse than what I’ve been using to evaluate Whisper. My PC sound-card is probably producing samples with more like 13-15 effective bits of resolution. It is possible that worse audio quality may impact the accuracy of the transcription model.

As a fallback option, I also ordered an I2S mic board. That has its own ADC, provides 24-bit samples, and claims to have 18 bits of effective resolution. This was relegated to the second-string because it uses an onboard MEMs mic which I probably cannot replace with a wired 3.5mm mono input jack.

Ultimately, I intend to give this device a standard 3.5mm mic input, so that I can use it with any wired mic. It seems like proximity may very well be more important than sample resolution. When testing Whisper through the PC sound-card, I had better results with the mic clipped to my lapel than sitting on the desk, so I expect to do the same with this device.

A local surplus store had this absolute unit in stock, so that will be the switch.

Development Setup

I want to build a tight development loop, and my first concern was that using the device as a USB-HID device would interfere with flashing code onto it. It would be annoying to have to constantly un-plug, configure, re-plug, flash, un-plug, reconfigure, test. I started looking into using a second Pi Pico as a debugprobe, but that doesn’t turn out to be necessary with CircuitPython. In my experience, it handled all the USB stuff perfectly!

Development

I want to build working sketches of the key areas before getting serious about doing anything well. If I’m correct in my estimation, this stuff should “just work” without too much fuss. If that assumption is invalid I’d like to find out sooner than later and re-evaluate my priorities.

Audio First

First I validated that I could get usable audio coming off the RP2350. This was first because it involves:

high-ish bandwidth analog stuff,
in an interpreted MCU environment,
involving a separate amplifier IC I’ve never used, and
using a microphone I haven’t tested.

Aside from obvious failure modes (like a defective component or connection), there could be timing issues in the sampling, or maybe the onboard ADC simply isn’t good enough.

For a quick validation, I wanted to just listen to the audio samples by ear. I thought about writing to flash and recovering via USB-MSC, but the CircuitPython drive is mounted RO w.r.t the device while it’s attached to a PC. The easiest thing I could think to do was stream the bytes over Wi-Fi, since that’s supposed to be a first-class experience on the Pi Pico 2 W anyway.

With minimal tribulations, I was able to record audio using analogbufio, convert it to WAV-compatible samples, and stream them through a socket.

On the PC side, I just did this. (TIL about sox!)

nc -l -p 4000 > out.raw

sox -t raw -r 16k -e signed -b 16 -c 1 out.raw out.wav \
&& vlc out.wav

The audio was passable, but noisy. This could have something to do with my jumper wires being somewhere near the length of a 2.4GHz halfwave dipole.

Just to be safe, I took a few recordings in this manner and ran them through Whisper. Even with the noisy signal, the transcriptions came out fine. So I’m feeling optimistic, and will jump to the next potential snagging-point.

USB-HID in CircuitPython

Luckily, emulating a keyboard turns out to be very easy! I adapted one of the examples given by Adafruit and it worked on the first try. It’s like 10 lines of code, and you can just pass a string into it to have that string typed out.

If you want to do this too, be advised that you’ll probably need usb_hid and adafruit_hid, which are different things.

Websockets

For a realtime experience (and because the device has a limited audio buffer), I’ll be using OpenAI’s streaming transcription API.⁹

This has an interface based on WebSockets. CircuitPython, oddly enough, does not include client for WebSockets even though it does have support for acting as a WebSocket server. In the worst case, I’m pretty confident I could implement a basic websocket connection myself without much trouble.

I found cpwebsockets however, which looks to be a CircuitPython adaptation of a MicroPython library. This worked well without much effort - looks like I’m lucky again!

Luck Runs Out

With all the pieces working, I thought I might be able to tie them together. Unfortunately, this is where I hit the limitations of CircuitPython.

In CircuitPython, I can use DMA to collect ADC samples in two ways.

block the CPU and read a specific count of samples, or
use loop=True to run DMA looping over the same buffer repeatedly, without blocking the CPU.

In the looping method, there’s no way (in CircuitPython) to get a count of how many samples have been written to memory at any given instant. You can guess based on timing, but that’s pretty ugly, especially in an interpreted environment.

This limitation with DMA would not be a problem if CircuitPython supported threading/multi-core. It would be a bit silly to monopolize an entire core just waiting for DMA, but it would work!

Similarly, if CircuitPython supported handling interrupts, there might have been a way to do this in a single thread.

Alas though, my heretofore fantastic luck ran out here.

What Now?

I think the answer is just to rewrite this in C. I expect the Pico SDK will be pretty well polished, and it is definitely well documented.

The only other possibility that comes immediately to mind would be writing a C helper for CircuitPython. I have a feeling that could actually be more involved than rewriting it in C though. CircuitPython is great to use, but I’m not ready to start extending it.

I’m happy to have validated that the Pi Pico 2 W is plenty powerful enough, that the audio is plausible, and that I can get keystrokes out of it. I figure that doing things in C may mean that I have a harder time getting the USB-HID stuff to work. In the worst case though, maybe I could reverse engineer the way it’s done in CircuitPython.

Development 2: C / Pico-SDK

Development Setup 2

I’m now back to thinking I’ll want a Pico Debugprobe. I have heard USB can be tricky to get working, and I’d like to simplify matters as much as possible. One thing that could help is to make sure that the USB stack is only used for HID. The debugprobe gives me a UART-USB bridge for free, so I’ll use that for output (rather than USB-CDC direct from the RP2350).

I’d also like to enable flashing over SWD, because that will save me having to do the unplug -> BOOTSEL -> plug-in -> USB MSC -> mount -> Copy UF2 -> unmount -> unplug -> plug-in dance.

I may as well start by setting up the Pico Debugprobe so I can build a workflow around it.

Pico Debugprobe

I flashed a spare RP2040 board with the Debugprobe firmware. (This used to be called Picoprobe, but that conflicted with someone else’s project, so it’s now called Debugprobe.)

I was able to use the prebuilt debugprobe_on_pico.uf2 from here. Note that there’s debugprobe.uf2 for the officially released hardware (same RP2040 but different board), and debugprobe_on_pico.uf2 is for running on a Pico board.

Verify UART Connectivity

While my project board still has CircuitPython, I flashed a code.py which just prints “Hello from RP2350!” to UART every 1 second forever. This gives me a way to verify that my UART bridge works before doing anything else.

Next, I plugged in the 2040 and wired it up as shown in Figure 10 here:

https://datasheets.raspberrypi.com/pico/getting-started-with-pico.pdf#debugprobe-wiring-section

Now, when I plug the 2040 in with USB and do picocom -b 115200 /dev/ttyACM0, I see the line “Hello from RP2350!” repeated once per second.

Setup Pico-SDK and Flashing over SWD

To check the flashing functionality, I need some .elf files. I will start with the pico-examples. This involves setting up the SDK so I can compile them, and setting up OpenOCD to interface to my RP2350 via my debugprobe.

The Pico-SDK was pretty easy to set up. I just added it as submodule and updated the environment in my .envrc:

export PICO_SDK_PATH="$(pwd)/pico-sdk"
export PICO_PLATFORM="rp2350"
export PICO_BOARD="pico2_w"

To use OpenOCD with a RP2350, you’ll want to use the Raspberry Pi fork of OpenOCD.¹⁰ To compile it, I needed to do this:

./configure --enable-internal-jimtcl

I’m leery of doing sudo make install, so I chose to use openocd in the directory where I compiled it. If you use the -s flag to specify a search path for TCL scripts, you can run it in-place like this:

$PATH_TO_RP_OPENOCD_REPO/src/openocd \
	-s $PATH_TO_RP_OPENOCD_REPO/tcl \
	-f interface/cmsis-dap.cfg -f target/rp2350.cfg \
	-c "adapter speed 5000" \
	-c "program blink.elf verify reset exit"

That works to flash the board over SWD and connect to it with GDB.

Project

The repo is here: whisper_pedal. ^{^{n.b.: I haven’t written much C in the last decade, so you’d hate to try and learn anything from me!}}

Pico-SDK Functionality

At this point, I tested a few more examples from pico-examples to confirm that I could:

read from the ADC,
send keystrokes over USB-HID,
connect to Wi-Fi, and
make an HTTP request,

all individually.

Since they work individually, I’m optimistic I can get them all working at once.

Design Changes

One feature that I’m now missing, having moved away from CircuitPython, is a nice way for would-be users to configure the device. Ideally it would be possible to provide an SSID and password for the Wi-Fi, as well as potentially providing a custom URL/key to use for the speech-to-text API to facilitate using a different API provider.

Raspberry Pi’s “Pico Extras” repo contains something that implements “USB-MSC”. It’s possible that I could use this to make a ‘configuration mode’ that would work as previously designed. I’d basically set aside a small area of the flash to hold the configuration, and make that available via USB-MSC through an emulated filesystem. That might be a heavy lift though.

Another thought would be to use a custom UF2 file that updates only a small region of the flash memory that is specially set aside for configuration. I could provide a web-based utility to generate a UF2 file that can be dragged-and-dropped onto the device (when it’s connected with BOOTSEL pressed).

It might even be possible to use the white-labeling functionality to customize the index.htm which appears on the default device to link to this tool.

This method could be nice, because it’d save me having to implement my own USB-MSC -> pseudoFS configuration interface or else doing the typical Wi-Fi configuration over an ad-hoc AP setup flow that every IoT device does.

Software

I used core0 to run USB and audio capture, and core1 for Wi-Fi and the API interaction.

Thread coordination is just a single shared flag (which doesn’t even need to be a semaphore), and 2 queues.

When audio is being recorded, it’s just pushed onto a queue which is consumed by the networking core. The audio sub-buffers are sized to fit within the hardware transmission buffer.

When transcriptions are returned from the API, another queue is used to accumulate a sequence of key-reports.

TinyUSB

TinyUSB is indeed, perhaps, a little underdocumented. Of course, I’m very grateful to have it at all!

I have adapted some code from the examples, and, since it works, I am going to just take the win and move on.

Wi-Fi

Surprisingly, this went very well based on looking at a few examples, a few blog posts, and asking Claude to write it for me.

OpenAI Realtime

The Realtime API is in beta at the time of this writing. I’m happy with it, but the documentation was a bit unclear and a bit scattered, and/or there may be some missing functionality still.

In particular, the doc implies that you can use your API key directly to establish a WebSocket connection, and that you can update the properties of said realtime session using the session.update or transcription_session.update client events. In fact, this does not work. You cannot, for example, turn off VAD (Voice Activity Detection), nor select a specific model.

However, if you follow the flow which is probably more common in user-facing applications, and you obtain an ephemeral client secret first, you can configure the session in the POST request at the time you’re getting the client secret. (This is the same configuration that cannot actually be provided as a session.update or transcription_session.update).

Even if you do not specifically need a client secret for anything, you may need to request one anyway, just to generate a session to which you can apply configuration.

Luck Runs Out – Again!

While connecting to Wi-Fi and even getting a websocket connection over TLS turned out to be surprisingly easy, using TLS/TCP to stream on the RP2350 turned out to be a problem which is larger than I really feel like dealing with for this project.

In particular, the altcp send buffer would stop draining after a point, and I struggled for several hours to find out why. The most likely possibility is that this is my fault for, essentially, not reading enough about what I was doing. However, there’s always the risk that something is actually broken, undocumented, or just more subtle than expected.

I was tempted to dive head-on into the interactions between TLS, TCP, and Wi-Fi driver stack, but I had already gone beyond the amount of time I wanted to spend on this project. Instead I will remain blissfully ignorant of lwip and mbedTLS. There’s a way to get a quick win here and avoid falling down any rabbitholes.

I decided to just use UDP. I already know that TX works, but that there’s a problem which may involve flow control or the TLS integration, and so stripping out all these layers maximizes the chances of nearly immediate success. This sacrifices some of the elegance of having a fully self-contained peripheral, since I’ll need to write a little bridge from UDP to WSS. That will be something I have to run on my network.

I have a home-server which is perfect for hosting that utility, but I’m now getting out of the territory of anything I could ever distribute to anyone. ¹¹

UDP <-> WS Bridge Server

This is just going to be a simple Python process that listens on a UDP port waiting for 3 types of messages:

start
stop
audio

Upon start, it’ll open the WSS connection to the server. Upon stop, it’ll send the input_audio_buffer.commit message to the API and trigger transcription. Audio will just be audio streamed with no redundancy.

The start and stop messages are not strictly necessary to make this work, but they help trim down the latency and allow shorter interactions with the peripheral to be more ergonomic.

Finishing Up

I finished up by replacing the built-in electret mic on the MAX9814 dev board with a couple of leads to a TRRS (3.5mm) breakout; this let me plug in a wired mic so I can wear it on my lapel.

Remaining to be addressed is some kind of enclosure, and a bit of RF shielding.

Circa 2021 Google Docs had some transcription functionality built-in which was sufficiently accurate that it could sometimes be useful for writing English. Other times, it was frustrating, particularly for more technical writing. I never got into the habit of using it.

↩
A perfect example is "Jupyter", as in JupyterLab or Jupyter Notebook. Without a hint, the model will give you "Jupiter" every time. If you provide "Jupyter" as a hint though, it will shift its contextual inference strongly in that direction. You can still get "Jupiter" with "Jupyter" in the prompt though, if you say something like "the planet Jupiter".

↩
To be fair though, nothing works on Windows. _{_{OH, GOT 'EM! Microsoft will never recover from this.}}

↩
I was also considering the possibility that eventually I could share this project. It would be much more usable for others if it avoided any application dependencies that would have to be installed and run constantly in the background. Furthermore, any such application would certainly trip everyone's antivirus software.

↩
Of course, I ended up banging my head into it for a good while anyway.

↩
Sure, sure, ARM is proprietary too. But the RP2350 has RISC-V onboard, so that's one step in the right direction. Anyway, ARM is more widely licensed and seems to have a healthier ecosystem than Xtensa, so I worry a bit less about investing energy into it.

↩
Apparently, you can even change the name of the drive without much fuss, so I could call it `VOICE_KEYBOARD` or something other than just `CIRCUITPYTHON`.

↩
The RP2350 ADC can actually sample at 500 kHz. Depending on how the ADC is losing 3 effective bits, we might find gains by oversampling. Oversampling by a factor of 4 can, in the exact best circumstance, provide 1 extra effective bit of resolution. Theoretically, we could oversample a 16 kHz signal by 32x using this ADC, but there are compute and RAM constraints which may make that impossible.

↩
This was in beta as of this writing, and it was a little confusing to find the doc at first, so I'm linking it here in case you're curious about it. OpenAI API Reference

↩
OpenOCD looks like it's supposed to be very extensible. I'm not sure why Raspberry Pi needs to maintain a fork, and I haven't attempted to compare them. I saw some people using PyOCD and got the impression that it might be more actively maintained (and may be a good choice for RP2350 development), but I'm not certain if it has feature parity with OpenOCD yet.

↩
I thought it would be fun to make something shareable, but, realistically, this is going to be built into most people's OS's within several months at the current rate of AI-ification, so there's not much point. Macs already have a shortcut where you can double-tap right-command to speak into any text input area -- how long could it possibly take Apple to connect that to a better speech-to-text model? Probably a third party has already done this.

↩

LLMs for Filtering Jobs -- Part 1

2025-01-07T00:00:00-05:00

tl;dr:
I annotate some data, and try out Promptfoo to compare some small and small-ish language models.

Why?

I obtained a list of some job postings, but they are nonspecific. Only a small portion of them will be relevant to me, and I don’t feel like reading all of them. Instead I feel like using this as a toy problem for exploring LLM workflows.

What?

For a first exploratory pass at the problem, my thinking was to simply run subsequent “yes or no” queries. Instead of bothering to do any prompt engineering, I ran each query through a couple different LLMs and considered the answers where they both agreed (or, at least, did not contradict). This was surprisingly decent, and would probably work even better if I’d used 3 models and taken the majority decision. For ad hoc jobs without any particular quality requirement, I think that’s an acceptable starting point.

However, I rapidly became curious about all of the complexity that I was shoving under the rug with that approach.

Which model was best, given identical prompting?
What if I want to scale a pipeline like this? Wouldn’t I rather not burn 2-3x as many tokens as strictly necessary? Current models are probably trustworthy enough to just use one, for simple queries.
Speaking of efficiency, how small of a model could I use? Can I get anything useful out of a 1 billion parameter model at 2-bit quantization?
How complex of a query can I run?

Aw heck. It’s a good day to fall down a rabbit-hole.

Comparing Small Models

Given a relatively unconsidered prompt, how do different models behave? Are they accurate for simple queries? Can we constrain their output to “yes”, “no”, or “insufficient information”?¹

For a simple query, let’s try to ascertain whether a job is remote or not.

I started by labeling some of the postings using label-studio.² Then, I just wrote a quick script to transform my labeled JSON data into Promptfoo assertions.

You can see my prompt here, and how several different models and quantizations performed:

We can see that, with my naive prompting, gpt-4o-mini does quite well, and then the various 3b and 8b llamas do pretty poorly. I was curious about the effects of quantization on accuracy in a simple test like this, but it’s too early to tell when they’re all doing this badly.

What’s wrong with the Llamas?

Every one except for the smallest adhered pretty consistently to the requested output format. They just didn’t give the expected responses.

Part of the problem here may be that my data is a bit dirty – there are posts included in it which are not job postings. For labeling purposes, I called these cases “insufficient information”, but they do differ from what I specified I was providing in the prompt. Maybe gpt-4o-mini is just more graceful when the data differs from the exact expectation?

I will remove these from the data and we’ll try the eval again.

Much better! It seems like they just couldn’t handle the deviation from their expected input very well.

I would guess that, with a little prompt engineering to make my exact expectations more explicit, llama3.1:8b could perform similarly in practice to gpt-4o-mini on this task.

That is potentially significant, since gpt-4o-mini is ~4x the price.

Model	Input Cost (per million tokens)	Output Cost (per million tokens)
llama3.1:8b	$0.05	$0.08
gpt-4o-mini	$0.15	$0.60

Comparing Mistral models under 40GB

I also tried some Mistral models and one derived from a Mistral model.

It seems like (with the cleaner data) just throwing more parameters at the problem doesn’t do much.

Will these larger models handle the dirty data better?

Just for fun, I will re-run the larger Mistral models against the full set of annotated data. This includes those posts that are not jobs, for which we (perhaps confusingly) are expecting a response of “insufficient information”.

My intuition is that the larger models will be slightly more tolerant of this weirdness and more likely to “infer what I actually wanted”.

At a basic anecdotal level, there seems to be some support for this intuition.

How about a regex?

I also tried to classify job remoteness using simple string-matching. This was ~64% accurate over the same (clean) dataset, and saves a few TeraFLOPs per item.

Using Promptfoo

While writing this post, I tried Promptfoo for the first time. It provides a convenient UI for this type of comparison between models and between prompts. It looks like there’s enough flexibility built in to make it do most of what you’d need it to do in this domain.

I did have one issue with it, though. Given a list of providers, Promptfoo seems to run tests in-order, running the same test with each provider before continuing to the next test. This is much, much slower than running all the tests for each provider serially, if you’re using a single Ollama server that has to juggle the models in and out of memory.

Most of the time is spent loading and unloading models, rather than computing anything.

For a quick workaround, I just commented out all but one model at a time in my promptfooconfig.yaml, then ran promptfoo eval once for each model. Since Promptfoo caches the provider responses, then I could follow up with a final run, with all providers uncommented, in order to generate the nicer, combined report (without actually repeating any provider API calls).

What’s Next?

I want to find some useful queries which larger models can handle, but smaller ones can’t. These will also have to be something that I can label reasonably easily. One obvious low-hanging fruit is to use a model to filter out the posts that aren’t jobs.

Ultimately, I might like to put my entire resume in-context and see if I can filter for jobs that fit based on that. It might also be interesting to follow links from the posts to get longer, more detailed job descriptions when they are available, as well as additional information about the companies.

I'll get into structured output later. This is just the bare-bones basics.

↩
Label Studio is the open-source, core offering of a for-profit company.

It is literally as easy to use as: `pip install label-studio && label-studio`, which is pretty neat. There are built-in templates for labeling different types of data, so it takes about 5 minutes to get a usable labeling UI with keybinds. Sure, it might have only taken 5 minutes to get Claude to whip up a custom labeling UI, but why do what's been done?

↩

Dirt-Simple Remote Inference for LLMs

2024-12-12T00:00:00-05:00

tl;dr:
I’d like an easy way to do batch jobs on remote GPUs. I don’t want to wait around manually to terminate instances when the jobs are done.

What? Why?

Like everyone lately, I could sure use some more VRAM. I don’t have such a volume of ML/AI workloads as warrants investment in local hardware yet. A100s in the cloud are under 3¢/minute – an order of magnitude cheaper than a long-distance phone call in my birthyear – and RTX 3090s are actually free, if you regard the prices with %0.2f, so let’s use ‘em.

Why RunPod?

There’s a lot of providers offering GPUs in the cloud at the moment, but RunPod is one of a small number¹ of them that offers pre-paid bill-capped service. According to their FAQ, they’ll just stop service if you run out of pre-paid credit.

That’s perfect for doing quick hacky stuff like this, since you can’t accidentally rack up hundreds or thousands of dollars of bills like many people have done on AWS and GCP. ²

Why NOT RunPod?

I’d prefer a full VPS, but RunPod is a container-based service. This might be preferable if that’s your workflow anyway. It can however be mildly annoying if you just wanted to leverage a prebuilt image, and naively expect full-featured SSH.³

Their containers do not, by default, get a public IP address. You can request a public IP, but if you do, it looks to me like you’re pulling from a smaller pool of machines, so you might be more likely to encounter availability issues.

You might think you don’t need a public IP, but unless your traffic is exclusively HTTP, you probably do. They have proxying for HTTP, but they won’t forward TCP ports for you unless the host supports a public IP.

I also encountered at least one issue that feels like a bug. When calling runpod.create_pod() with a template, the template’s storage configuration is not respected. All attempted launches will fail somewhat cryptically unless you directly supply a storage configuration to create_pod(). I was able to find the workaround rather quickly because I happened to think to check their Discord server, but that’s not necessarily the first place one might think to look.

The vibe I get is that they’re new at this, but improving. I might hesitate to use RunPod in production, but on balance I like the service.

You said this would be easy?

Yep! Following the happy-path and just running a simple prebuilt Ollama container with a GPU is pretty easy.

If you want to use my code below (also available as a Gist), you can just get an API key and use it like:

with runpod_ollama_client("tinyllama-test", "NVIDIA GeForce RTX 3080",
                          load_model="tinyllama") as client:

    prompt = "Tell your favorite joke?"
    response = client.generate(model='tinyllama', prompt=prompt)
    print(response.response)

Performance

When using a small model that loads quickly, the service goes live within a couple minutes. For bigger models, factor in the time to download. Most of the pods I’ve been given have purported to have pretty decent bandwidth, but then it’s a matter of how much bandwidth the server hosting the models wants to give you.⁴ Anyway, for me, for batch use, this is perfectly acceptable.

Security

There is none. That HTTP endpoint is publicly accessible with no authentication.

Ollama doesn’t have anything built in. Even if it did, it’s a fast-moving project that hasn’t hit v1.0 yet – so, would you trust it?

Sure, you’re getting a secure tunnel to the proxy, I guess. Between there and your pod, I think you’re trusting RunPod (and, possibly, a third-party host!) not to snoop.

Practically speaking, I’m not very concerned. My workloads are not sensitive. For me, I think the most likely (but still unlikely) worst case scenario is that someone somehow guesses one of my pod IDs and piggybacks some free inference.

Thoughts for the Future

RunPod can also do serverless inference. They have some semi-pre-configured stuff using vLLM, but it looked slightly more involved than just spinning up an ollama pod. The pods also have the advantage that I can SSH in and troubleshoot any issues which may come up while I’m getting started. For now, my workloads are batch-processing things anyway, so a serverless approach doesn’t do me many favors.

If I have workloads that are less batchy, I may invest some energy in figuring out a serverless solution. That said, unless you’re paying to keep an instance hot, the latency seems likely to be pretty ugly, and if you do keep an instance hot, it doesn’t sound so serverless after all.

The Code

import os
import sys
import time
import subprocess
import contextlib
import functools

import tqdm

import requests
import runpod
import ollama

runpod.api_key = os.environ.get('RUNPOD_API_KEY')
PORT = 11434


class LaunchFailureException(Exception):
    """Raised when a pod or instance fails to launch properly"""

    def __init__(self, message):
        self.message = message
        super().__init__(self.message)


def with_retry(f, should_retry=lambda _: True,
               max_attempts=5, delay=2, backoff=False):
    """
    should_retry is a function that accepts the exception raised by f and
    returns a bool
    """

    for attempt in range(max_attempts):
        try:
            return f()
        except Exception as e:
            if attempt == max_attempts - 1:
                raise e

            if not should_retry(e):
                raise e

            print(
                f"Attempt {attempt + 1} failed. "
                f"Retrying in {delay} seconds..."
            )
            time.sleep(delay)
            if backoff:
                delay *= 2


def wait_for_pod_reported_ready(pod, ready_timeout_seconds=450):
    print("Waiting for pod: ", end="")
    start_t = time.time()
    while (time.time() - start_t) < ready_timeout_seconds:
        pod_info = runpod.get_pod(pod['id'])

        if not pod_info:
            print("-", end="")
            continue

        if pod_info['runtime']:
            print("\n")
            assert 'ports' in pod_info['runtime']
            return pod_info

        print(".", end="")
        sys.stdout.flush()
        time.sleep(1)

    raise LaunchFailureException(
        message=f"Pod failed to become ready. \n Pod ID: [{pod['id']}]"
    )


def pull_with_progress_bar(client, model):
    current_digest = ''
    progress_bars = {}

    # Note that streaming mode is necessary, otherwise the request gets timed
    # out, I think because of the proxying through Cloudflare (100 seconds).
    # See doc here:
    #   https://docs.runpod.io/pods/configuration/expose-ports
    for response in client.pull(model, stream=True):
        digest = response.digest or ''

        if not digest:
            print(response.status)
            continue

        if digest != current_digest and current_digest in progress_bars:
            progress_bars[current_digest].close()

        if digest not in progress_bars and response.total:
            progress_bars[digest] = tqdm.tqdm(
                total=response.total,
                desc=f"Pulling {digest[7:19]}",
                unit='B',
                unit_scale=True
            )

        if response.completed and digest in progress_bars:
            bar = progress_bars[digest]
            bar.update(response.completed - bar.n)

        current_digest = digest


def load_model_on_server(client, model):

    def load_model_on_server_inner():
        print(f"Loading model...")
        pull_with_progress_bar(client, model)

    with_retry(
        load_model_on_server_inner,
        should_retry=lambda e: (
            isinstance(e, ollama._types.ResponseError)
            or
            isinstance(e, httpx.RemoteProtocolError)
        ),
        backoff=True
    )


def wait_for_ollama_generate_ready(client, model):
    """
    Sometimes /api/generate seems to not be ready immediately after the pull.
    """
    def wait_for_ollama_generate_ready_inner():
        print("Waiting for /api/generate...")
        response = client.generate(model=model, prompt="check check 123")
        assert response.response

    with_retry(
        wait_for_ollama_generate_ready_inner,
        should_retry=lambda e: (
            isinstance(e, ollama._types.ResponseError)
            or
            isinstance(e, httpx.RemoteProtocolError)
        )
    )


@contextlib.contextmanager
def runpod_ollama_client(name, podspec="NVIDIA GeForce RTX 3090",
                         load_model=None, storage_gb=128):

    def create_pod():
        return \
            runpod.create_pod(
                name,
                "ollama/ollama",
                podspec,  # use the ids from `runpod.get_gpus()`
                volume_mount_path="/root/.ollama",
                ports=f"{PORT}/http",
                container_disk_in_gb=storage_gb,  # required even with template
                volume_in_gb=storage_gb  # required even with template
            )

    pod = \
        with_retry(
            create_pod,
            should_retry=lambda e: (
                isinstance(e, runpod.error.QueryError)
                and
                (
                    "does not have the resources to deploy your pod" in str(e)
                    or
                    "no longer any instances available" in str(e)
                )
            )
        )

    try:
        pod_info = wait_for_pod_reported_ready(pod)
        ollama_host_url = f"https://{pod_info['id']}-{PORT}.proxy.runpod.net"
        client = ollama.Client(host=ollama_host_url)

        # You need to load a model before you can get completions.
        if load_model is None:
            print('Proceeding without loading a model.')
            print('Probably this is a mistake.')
        else:
            load_model_on_server(client, load_model)
            wait_for_ollama_generate_ready(client, load_model)
            print("Verified /api/generate is ready!")

        yield client

    except LaunchFailureException as lfe:
        print("Looks like a bad machine.")
        raise lfe
    finally:
        runpod.stop_pod(pod['id'])
        runpod.terminate_pod(pod['id'])


def kill_all_pods():
    pods = runpod.get_pods()
    print(f"Found [{len(pods)}] pods to kill.")
    killed = 0

    for pod in pods:
        pod_id = pod["id"]
        try:
            print(f"Stopping pod {pod_id}...")
            runpod.stop_pod(pod_id)

            print(f"Terminating pod {pod_id}...")
            runpod.terminate_pod(pod_id)
            killed += 1
            print(f"Successfully terminated pod {pod_id}")
        except Exception as e:
            print(f"Error terminating pod {pod_id}: {str(e)}")

    print(f"Terminated [{killed}] pods.")


def tell_me_a_joke():
    with runpod_ollama_client("tinyllama-test", "NVIDIA GeForce RTX 3080",
                              load_model="tinyllama") as client:

        prompt = "Tell your favorite joke?"
        response = client.generate(model='tinyllama', prompt=prompt)
        print(response.response)

def tell_me_a_more_expensive_joke__llama3_3_70b_q8():
    model = "llama3.3:70b-instruct-q8_0"
    with runpod_ollama_client("llama3_3__70b__q8_test", "NVIDIA A100 80GB PCIe",
                              load_model=model, storage_gb=256) as client:
        prompt = """
        Tell me your favorite joke, but talk like a pirate.
        To be clear, the joke should not be even remotely nautical.
        No pirate jokes whatsoever.
        Tell me a normal, non-pirate joke.
        Just, you know, tell it like as if you *also* happened to be a pirate.
        """
        response = client.generate(model=model, prompt=prompt)
        print(response.response)


if __name__ == "__main__":
    tell_me_a_joke()

Vast.ai is the only other one I found, but my search was not exhaustive.

↩
Some of these examples are more sympathetic than others, but you get the picture.

Of course, AWS becomes less risky the more time you spend being cautious, but I don't enjoy caution for caution's sake. ^{Anyway, who wants to give money to Jeff when we could give it to literally anyone else?}

↩
Some of their prebuilt templates do use an image that runs SSHd, but not all. Of the rest, whether or not you can use the trick they suggest here depends on whether the Docker image involved uses 'entrypoint' or 'cmd'. You can override a 'cmd', but they don't support overriding an 'entrypoint'.

What's provided by default in all cases is something which looks like SSH access, but isn't. They are actually letting you connect to the underlying Docker host, and have set your login-shell to docker exec -it $YOUR_CONTAINER /bin/bash. Considering the full context, that's great, since you can get into any container without having to run your own SSHd. It does mean however that you are getting only an interactive console. There's no secure tunneling, no SFTP, no SCP, and, in fact, you can't even do something like ssh $host apt install $package.

_{If you want to force the issue, you could do echo "apt install $package" | ssh -tt $host, but if you start trying to pipe any non-trivial scripting through the interactive console, it will eventually make you sad. SSH can't even forward you an exit code in this case, so you're getting into 'pexpect' territory...}

↩
Considering my fairly infrequent use, I don't feel bad about hitting the servers hosting the models. If you were going to do something similar but more frequently, it might be more responsible to prebuild an image with the model already downloaded, so you're hitting your container registry rather than the public model servers.

This likely won't change runtime performance significantly since the Docker host is pulling roughly the same amount of bytes down the same connection, wherever they're coming from.

↩

So, you want to weld things?

2024-07-06T00:00:00-04:00

Everything you need to start welding, below.

Why?

I decided to learn how to weld. While a welding machine itself is kind of expensive, I was mildly surprised by how little else you need to get going and make some (basic, but seriously strong) physical structures.

It seemed to me like the relevant information was spread across a handful of different sources, and this made the very first “pick up a machine and try it” step a little more difficult than it needs to be. I thought I’d consolidate the information that I felt I needed to begin in one place, in case that might help anyone else who wants to start.

I am not a welder, and this does not constitute professional advice. If you place any credence in my words, you will surely burn your house down, electrocute yourself, blind your neighbor’s dog, and many worse things. You proceed at your own risk.

What process should I start with?

I don’t know. Some people say you’ll “learn more” if you start out with stick welding. Perhaps they’re right, but I’d just start with whatever is most convenient.

The list below is not particular to any process. It covers all of the other things that you might forget and end up accumulating over many false-starts.

Do you have anything to tell me about actually welding?

Nope. You can have my list of what you need to get started though!

List of things:

PDF here:

Welding Checklist for Getting Started

Extra thoughts:

If you’re using a flux process (flux-cored wire or stick welding), you’ll want a slag hammer.

You can get by using a manual wire-brush for all your brushing needs, but I’m much happier using a right-angle drill with a wire-brush attachment. A regular drill would work too, but it torques the wrist unpleasantly.

If you’ve tried all the standard troubleshooting but still can’t get a decent weld, you may want to check the voltage available to your welder. I learned the hard way that, in some circumstances, a long extension cord can impart a significant-enough voltage drop that it’s nearly impossible to weld properly.

Learn a thing or two about testing your welds before you trust them in any kind of safety-critical application. It is possible for a weld to look OK, especially to an untrained eye, but to fail destructively when it’s rolling down the highway at 70mph.

LLM Teaching-Assistant Prototype Interface

2024-01-24T00:00:00-05:00

I threw together a quick prototype to demonstrate how large multimodal AI models might be used to supplement human TAs in delivering educational courses.

The idea driving this demonstration was that a relatively simple interface could be used to allow easy questions to be directed to an AI assistant with minimal friction. In this respect, you could think of the AI model as providing “level 1 tech support”, so the TAs only need to step in for more difficult or nuanced questions. This could help the same number of TAs more effectively handle a larger number of students.

Another point to highlight is that, with the right interface, the learning experience can easily come to resemble having a TA looking right over your shoulder every step of the way. It is not a stretch to imagine that people learn more quickly when they have access to 1-on-1 attention. An AI assistant may not be as effective as a human TA, but most people cannot afford to have a human TA actually looking over their shoulder throughout the entirety of the learning process. Persistent, full-time AI accompaniment, however, is quite feasible.

Proof of Concept

In the video below, you can see a browser window running the prototype interface alongside an instance of Eclipse IDE. The Eclipse window has been shared into the browser (using the screen-sharing API). In this way, the AI model is given access to “see” what the user is working on. Simultaneously, there is a persistent voice connection between the user and the AI model.

This allows me to talk about what I’m seeing on my screen, and get contextually-aware assistance from the AI. In the video, I play the first-year computer-science student and we see that the AI is capable of helping me out.

Minimum Viable Personal Backup Solution

2023-05-05T00:00:00-04:00

A simple backup solution for Linux desktops.

tl;dr:
I have a keypress combination to put my desktop to sleep. I’m replacing that with a script that will run a quick backup and then sleep.

The easiest and most robust strategy I could come up with involves restic, some minimal configuration, and Backblaze B2.

What? Why?

I want backups. And I want to think about them as little as possible.

I’ll use BackBlaze B2 for my offsite replication. This is an object-storage service much like Amazon’s S3. I use Backblaze because they have good prices and offer decent transparency into their operations. Over the years, I’ve appreciated their quarterly reports of hard drive failure rates, which they publish openly.

I’ll use Restic because it’s flexible and I know it’s easy to orchestrate from the command line. I don’t like running services like Duplicati on my computer all the time.

I want something that runs frequently and runs fast, but which won’t tax my upload bandwidth while I’m actively using my computer. I already have a keyboard shortcut in my window manager to suspend to RAM. My thought is to write a script to take a backup and sync to Backblaze, and then simply invoke that automatically every time the computer is about to suspend.

There’s probably a more graceful way to do this, but I’m aiming for the lowest expenditure of effort here. Because it’s a desktop, it doesn’t need to sleep immediately like a laptop does. So, the slightly ugly side of this solution (my computer will be awake for an extra few minutes until the backup completes after I tell it to sleep) is a non-issue.

This will be a basic 3-2-1 backup strategy. I want a local backup for ease of use. In case I nuke something on my computer that I meant to keep, this backup is handy in that circumstance. Then, to get a remote copy, let’s mirror that backup offsite to Backblaze.

I want this backup to cover my home directory with 1-day, 1-week, and 1-month copies.

To implement this scheme in restic, I will have one local repository and one remote repository. The remote will be a simple mirror.

Steps I took:

Install restic:

As everyone who uses Arch will tell you - I use Arch, so for me installation is:

pacman -S restic

Setup a Backup Repository:

I’m going to start out by setting up a restic repo on an external SSD which should be large enough to backup my home folder for quite some time. (I used baobab to check my disk space usage. It’s my favorite tool for this, as well as for clearing up space.)

Assuming the external backup drive is mounted at /mnt/bkup_usb_ext, and you’ve created a backups/home_slash_$USER directory there, you can run this:

restic init --repo /mnt/bkup_usb_ext/backups/home_slash_$USER

You’ll get a password for that repository that you need to keep track of.

I wrote a backup script with some excluded directories. This will take a single snapshot:

#!/bin/bash

# Here, we will take one snapshot.
# You should have already exported the restic password so this can use it.


LOCAL_RESTIC_REPO="/mnt/bkup_usb_ext/backups/home_slash_$USER"

restic -r $LOCAL_RESTIC_REPO backup /home/$USER --one-file-system \
	--exclude /home/$USER/large_files_you_do_not_want_to_back_up \
	--exclude /home/$USER/.cache \
	--verbose

As you can see, multiple exclude flags can be used to trim anything out of your backups. There’s more documentation here.

Setup Remote Repo:

There’s a lot you could unpack here, if you wanted to. Check out restic’s documentation here.

To make things as simple as possible, I copied the params from the local repo. For me, the command looked like this:

restic -r $REMOTE_RESTIC_REPO init --from-repo $LOCAL_RESTIC_REPO --copy-chunker-params

If you create the remote repo without copying the params from the local repo, you may have issues with deduplication. Doing it this way seems to ensure that everything “just works”.

Mirroring:

To mirror, you need to supply both the password of the source repository (local) and the password for the destination repository (remote).

You can use environment variables for this, among other things. Keeping with our mantra of “simplicity”, I’ve used the environment vars.

Then it’s just a restic copy -–from-repo command. (See the whole script below.)

Retention Policy:

With restic, the way to implement a retention policy is to take snapshots as desired and then, later, go back and forget some of them. Then, you can prune the repo to remove the unneeded blocks.

Since I’m taking snapshots at odd times (whenever I put my computer to sleep), I will set up my forget command with --keep-within-... flags. Here’s a dry-run of that, which you can run to see which snapshots would be deleted.

$RESTIC_CMD forget \
	--keep-last 1 \
	--keep-within-daily 7d \
	--keep-within-weekly 1m \
	--keep-within-monthly 6m \
	--keep-within-yearly 2y \
	--dry-run

Checking Structure, Checking Data:

Bitrot is real, and sooner or later a cosmic ray will get the better of every one of us…

$RESTIC_CMD check –-read-data-subset=1%

Simple Alerts in Case of Problems:

I will just throw pop-ups on my screen using zenity, with the presumption that I should eventually encounter these. Ultimately, it would be nice to send myself email but I’m not going to overthink this.

The Script:

If you want to copy and use this, you will have to go line-by-line to fill in your own information.

Another thing you’ll need, which I did not cover here, is a Backblaze bucket and application key. There are guides already that can help you set these up. In the script, you can see some stuff about AWS, but this is just because we’re using Backblaze’s B2 as an S3 clone. They provide an API compatible with S3 clients. Restic’s S3 support is better than its B2 support, and Backblaze’s S3-compatibility is solid, so I think this is the way to go.

Obviously this is provided without warranty, but feel free to use and modify any way you like. If you share it publicly, please cite this page.

#!/bin/bash

#I want this script to abort on the first sign of failure and give me a pop-up in case it does
set -e
trap alert ERR

alert() {
	zenity --warning --text "Check your restic log at /tmp/restic.log" --title "BACKUP ERROR!" &
	zenity_pid=$!

    	# Wait a little for the Zenity window to appear
    	sleep 1
    	#In i3, this will focus the window.  You will want to remove this if you don't use i3.
    	i3-msg '[class="^Zenity$"] focus'

    	# Make sure to wait for the Zenity dialog to be closed before exiting the script,
   	 	# or else the script might exit before anyone ever sees the dialog
    	wait $zenity_pid
}

# Initialize Local Environment
LOCAL_RESTIC_REPO="/mnt/bkup_usb_ext/backups/home_slash_$USER"
export RESTIC_PASSWORD="this would be the password to your REMOTE repo"
export RESTIC_FROM_PASSWORD="this would be the password to your LOCAL repo"

# Initialize Environment for Remote Repository:
S3_COMPAT_ENDPOINT="s3.us-west-000.backblazeb2.com" # This should also be updated!
BUCKET_NAME="YOUR_BUCKET_NAME_HERE"
BUCKET_DIR_PREFIX="restic"
TF_B2_BKUP_APPLICATION_KEY_ID="YOUR_BACKBLAZE_KEY_ID"
TF_B2_BKUP_APPLICATION_KEY="YOUR_BACKBLAZE_KEY"

# These two need to be exported as if they're AWS-related, because we're using the S3 driver of restic
# with BackBlaze's S3 compat layer
export AWS_ACCESS_KEY_ID=$TF_B2_BKUP_APPLICATION_KEY_ID
export AWS_SECRET_ACCESS_KEY=$TF_B2_BKUP_APPLICATION_KEY

RESTIC_CMD="restic -r $LOCAL_RESTIC_REPO"
REMOTE_RESTIC_CMD="restic --repo s3:${S3_COMPAT_ENDPOINT}/${BUCKET_NAME}/${BUCKET_DIR_PREFIX}"

# Clear logfile from last run
mv /tmp/restic.log /tmp/restic.log.1 || true
echo "### We're starting the backup now: $(date) ###" > /tmp/restic.log 2>&1


# Do a snapshot into the local repo
$RESTIC_CMD backup /home/$USER --one-file-system \
	--exclude /home/$USER/large_files_you_do_not_want_to_back_up \
	--exclude /home/$USER/.cache \
	--verbose \
	>> /tmp/restic.log 2>&1


# Local Cleanup.
# This gets rid of the extraneous snapshots we no longer need to store per my rules
echo "### Forget and Prune ###" >> /tmp/restic.log 2>&1
$RESTIC_CMD forget \
	--keep-last 2 \
	--keep-within-daily 7d \
	--keep-within-weekly 1m \
	--keep-within-monthly 6m \
	--keep-within-yearly 2y \
	--prune \
	>> /tmp/restic.log 2>&1


# Check structure + check actual data
# If you do backups less frequently, or are more concerned about integrity, you
# might increase that percentage.  As written, this checks 1% of the backup.
echo "### Check and Read Data ###" 	>> /tmp/restic.log 2>&1
$RESTIC_CMD check --read-data-subset=1% >> /tmp/restic.log 2>&1


###
# Remote Repo Stuff
###

echo "#########" 			>> /tmp/restic.log 2>&1
echo "### BEGIN REMOTE REPO BUSINESS ###" >> /tmp/restic.log 2>&1
echo "#########" 			>> /tmp/restic.log 2>&1

echo "### Mirror local to remote ###" 	>> /tmp/restic.log 2>&1
$REMOTE_RESTIC_CMD copy --from-repo $LOCAL_RESTIC_REPO >> /tmp/restic.log 2>&1

# Clean up extraneous snapshots and free up the space used by removed blocks
echo "### Forget and Prune ###" >> /tmp/restic.log 2>&1
$REMOTE_RESTIC_CMD forget \
--keep-last 2 \
--keep-within-daily 7d \
--keep-within-weekly 1m \
--keep-within-monthly 6m \
--keep-within-yearly 2y \
--prune \
>> /tmp/restic.log 2>&1


# Check structure + check actual data
echo "### Check and Read Data ###" 	>> /tmp/restic.log 2>&1
$REMOTE_RESTIC_CMD check --read-data-subset=1% >> /tmp/restic.log 2>&1

Dirt-Simple Ergonomic Reading Chair

2021-10-04T00:00:00-04:00

I wanted a comfortable way to read for extended periods, and this is what I came up with.

The Short of It

IKEA Poang
+
Wali Monitor Arm
+
Basic reading stand
=
“pretty good reading chair”

The bamboo is prone to splitting, so I used very small screws to secure it to the VESA mount, but it’s held up well.

This setup goes especially well with a large e-reader, as you don’t have to unclip it from the stand to flip the pages.

Reverse-Engineering Philips Wiz Smart Bulbs

2019-10-13T00:00:00-04:00

I reverse-engineered my Wiz smart-bulbs so I could control them without the Wiz app.

I’m going to walk through my reverse-engineering process here, just in case that helps anyone. If you mainly want to know about the protocol, I’m sure someone else has already documented it by now. If you want a library to control them without much fuss, you may be out of luck for the time being.

tl;dr:

echo '{"params":{"b":255,"c":0,"g":0,"r":15,"w":43},"id":542,"method":"setPilot"}' \
	> /dev/udp/10.0.0.95/38899

You can guess what r, g, and b are. The c and w are for “cool white” and “warm white” respectively.

Did you know you can send UDP packets directly from bash?¹ Me neither.

Why?

These bulbs themselves are pretty decent, but I don’t like the Wiz app. Partly for privacy reasons, and partly because I don’t want to rely on a proprietary application to control something basic and fundamental like my lights.

I was also considering building a fun little physical controller for each room’s lights with e.g. an ESP32 and 5x linear (sliding) potentiometers. Nothing beats a correctly located physical control. As an apartment-dweller, I like to have the dimming built into the bulbs themselves, but admittedly they really were onto something way back when they started installing lightswitches at the entrances and exits to rooms.

Also, if we’re being honest, I just wanted to fool around with Wireshark and learn how it works. I’ve had half a mind for a while now to start monitoring other devices in my home and see what kind of information they leak. This was a good first step, since, like everyone’s first Arduino project, it has the satisfying culmination: “Hey look, I can make the lights blink!”

Background

The bulbs I have are the Philips Wiz Bulbs, which are controlled primarily via WiFi. These are distinct from the Philips Hue bulbs which require a base-station and (IIRC) are using something like 433 or 900 MHz for control input.

They’ve got something like an ESP8266 inside, and can host their own WiFi AP for setup. Essentially all that this does is allow you to push the credentials for your main home WiFi network onto the bulb using the Wiz app. Once the bulb has saved the credentials for your home network, all further interactions will occur there.

It’s reasonable to presume that they’re authenticating and securing a DHCP lease. For the sake of convenience, I might have my router’s DHCP server assign specific IP addresses to them, so they won’t hop around as the leases expire.

After they’re connected, they could be listening on some port, or they might theoretically be polling some Wiz server, but the latter seems unlikely.

I expect that I can sniff the wireless traffic, record all the packets going to the bulb or coming from it, and then analyze those to reverse-engineer the protocol. Theoretically, there could be encryption set up in such a way as to require reverse engineering the Wiz app itself to defeat it, but I doubt they’d have bothered.

Setting Up Wireshark

I’ve hooked up an old, cheap wireless dongle which supposedly has decent Linux drivers.

However, Wireshark starts up without any interfaces available. I definitely remember, once, being able to select an interface to monitor on this screen.

You have to add your user to the wireshark group, and then you probably have to log out and log back in again to get that group attached to your login shell. I will instead live dangerously and opt for sudo wireshark.

It works perfectly now.

Monitoring the Wireless Network

If you simply double-click on a wireless interface that’s not already associated to an AP, you’ll get no packets:

When I checked ip link, the interface is state: DOWN. To get the interface up and associated, you can use tools like wifi-radar, Wicd, NetworkManager, etc. You can even use wpa_supplicant manually, but it’s a pain. The easiest thing you could possibly do is move to a machine that’s already associated with the Wireless AP you want to monitor, so that’s what I did.

If this works, you’ll see a great many packets. Who knew your computer did this much talking behind your back? Not to worry, I’m sure they’re all fine.

We’ll need some way to filter these packets. The IP or the MAC address for one of the lightbulbs would be the natural choice. I logged into my router to see an overview of the devices it was aware of, and it helpfully provides MAC addresses for everything.

Wiz also helped us out a little - the hostnames of the Wiz bulbs all start with wiz_. If you’re reverse-engineering something less helpful, it may be worth knowing that you can sometimes look up the manufacturer of a network device based on the MAC address.

In Wireshark, you can use a filter like eth.addr == xx:xx...:xx for MAC addresses.

If you look at the Source column for these Ethernet frames, you can see that Wireshark has replaced A8:BB:50 with WizIot. This is because it could look up the manufacturer for that MAC address prefix. Neat.

Those packets above were collected by turning the bulbs off and then back on. You can see they request IP addresses from DHCP. One of them was granted 10.0.0.242. Next, I’ll filter based on that IP address instead of the MAC. This will still filter for that specific lightbulb, but it will also naturally exclude all traffic below IP. The lower-level stuff is just noise to me at this point.

To be continued

After the above, I had to pivot and capture the packets a slightly different way. I will do a write-up of this process later.

It's worth noting that, even though it appears from the command like there might be a virtual device in /dev/, there's not. Obtusely, this is actually a special bash feature. ↩

taf.codes

Podcast Trend Analysis -- Part 1

What?

The Case for Exactness

What’s next?

Appendix: Trend Examples

Build Notes

Trends Queries

Filtering

Spatial Decay

Podcasts - Filtering Advertisements from Transcripts

Design Considerations

How’d it go?

Any other cool ideas?

What’s next?

Build Log

Labeling Setup

Inference

Comparing Models

Searching Podcasts - The First Step Towards Trends Analysis

How?

What’s next?

Build Log

Podcast Tracking and Ingestion

Search Application

Logseq and Continuous Backups

Why Logseq?

Why Not Logseq?

Using Unpredictable Software Anyway

Automatic Snapshots

Why not git?

Wrapper Script

Document Intake and Embedding: Millions of Tokens from Audio -- Part 3

How?

Analysis: Same Knowledgebase, More Documents

Effects of Collection Scale on Retrieval of Specific Facts Known to Be in the Collection

Vibes About Model Size

On Retrieval Relevance

What’s Next?

Document Intake and Embedding for Semantic Search -- Part 2

Why?

What is a knowledgebase?

Efficiently Retrieving Information from a Knowledgebase

What I’m Working On

Benchmark Saturation

Why is my benchmark saturated?

Scaling Up the Collection

Some Other Ideas: Knowledge-graph Traversal

Code

Document Intake and Embedding for Semantic Search -- Part 1

What? Why?

Overview

Code

Next steps

Building a Foot-Pedal USB Peripheral for Speech-To-Text

What? Why?

v0: Validating the Concept

v1: It Sounds Like It Should Be a Keyboard

Quick Demo

Development Summary

Future Work

Detailed Build Log ( Stop reading here, but go ahead and let your AI train on it. )

Design

MCU Selection

Software

Considering End-User Configuration

Proof-of-Concept Hardware

Development Setup

Development

Audio First

USB-HID in CircuitPython

Websockets

Luck Runs Out

What Now?

Development 2: C / Pico-SDK

Development Setup 2

Pico Debugprobe

Verify UART Connectivity

Setup Pico-SDK and Flashing over SWD

Project

Detailed Build Log
^{_{( Stop reading here, but go ahead and let your AI train on it. )}}