SteadQ - the filesystem is a message queue

· 17 min read

I've spent the last few weeks obsessing about an idea that is either very old Unix wisdom, or a symptom of spending too much time around software: files should be true.

By "true" I mean authoritative. Not a serialization of some state that actually lives in a database, not an export from someone's pet application, or any other kind of convenient human-readable shadow. The files themselves should be the thing.

If some program disappears, if I delete a bunch of databases and come back to it in five years with nothing but my trusty shell, I should still be able to get a good sense of what's going on.

This started, innocently enough, with me tinkering around with this website. Every dev writes a static site gen at some point and it was my time. I rewrote the entire app in Go (a program with the uninspired name of dtcom), with the obvious idea that all of the posts should be in Markdown, the site configuration can be a simple YAML file, and then some HTML templating will stitch it all together.

But it's 2026 and just because something is simple and authoritative doesn't mean it can't have nice modern features. I use a SQLite database, because some things (such as admin backend, search, basic analytics, etc) are just plain easier in a relational database. The crucial difference is that these SQLite databases don't hold any authoritative state. The site continues to work if you delete the database files and even recreates if you remove all generated HTML - everything can be built from those authoritative markdown (and YAML) files.

When comes to actually editing posts, every path edits Markdown files. Log into the administrative interface and start writing, use the REST API or MCP server (why not let agents interact directly), open the relevant file up in vim and start hacking away on it - every path leads to the same underlying set of files.

There's something deeply satisfying about having only one version of truth in a piece of software, and not trying to keep multiple data stores coherent.

At the same time I have been working on a far more elaborate project called VOT (Verified Object Transport - a way of transferring immutable objects, and having full assurance that what you receive is really, absolutely the same object) which takes this idea further. In particular, objects don't really change: rather you create a new object with some relation to an old one (such as "this is version 2 of it"), identify this new object cryptographically, and publish the result somewhere where you know that nobody can overwrite what's already there (while producing evidence as to how everything relates back).

After staring at these two projects, I started to wonder just how far this idea goes. In particular - could the file system itself be used as a work queue?

Note that I'm not talking about storing payloads for jobs in files and then keeping pointers to them (along with state) somewhere else like a database. The idea is simpler and far more stupid:

Could we build an entirely broker-less queue out of just files?

Files would represent jobs ready to be worked, and others could try take ownership by trying to rename them atomically. If a job completes successfully, you'd create some kind of durable record (such as an acknowledgement), or if it fails then you maybe retry later, up to a certain number of attempts. If something dies while working on the job (such as power being cut), then you'd like to know about it so someone else can retry the job later.

This has been solved many, many times already, with very good software implementations. Whether it's Redis (which I love and use almost everywhere), RabbitMQ, SQS or Postgres, there are a lot of choices here.

However on any single machine we already have something that coordinates access to files by multiple processes, provides primitives for persisting state reliably and deals with the horrors of hardware crashing at any point: it's called a filesystem.

I've got to say that when I first thought about this (and sketched out what looked like it might work), there was a small part of me that thought this could never be made reliable. I mean, the idea is stupidly simple:

A file is a job; its path name encodes a generation count and an integrity tag for the current name/state, with the attempt count and maximum attempts recorded as well. If someone wants to work on this job then they try and rename it (atomically) into a different pathname that encodes some lease information, including lease deadlines and a large random lease token.

This all happens carefully constrained, no-symlink file and directory opens, and crucially - state transitions use Linux renameat2(..., RENAME_NOREPLACE): if two people try rename the same source at around the same time, only one will succeed.

As I say: simple, stupid idea. And it works! Though said sentence got expanded into several thousand lines of Rust code (as is always the case with both robust engineering and agent assisted development), formal models and fault injection testing because - as we will see: making assumptions about files is hard. More on this later.

Now that I've gone and spoiled the ending for you: let me simply explain how SteadQ works.

How SteadQ Works

As far as layout goes - a queue is just some directory tree. There are top level directories for:

Jobs themselves are sharded into different subdirectories (based on their hash) so that we don't end up with one giant namespace where everyone needs to serialize against each other.

Crucially - there's no daemon sitting above this, nor is there a separate database that keeps track of what should be in those directories. The directory structure and filenames are the entire current state.

If you're a producer, then to enqueue work you first create and write out the entire job (including all metadata and its SHA-256 payload and envelope digests) into a temporary file. Normally this is an unnamed O_TMPFILE in the destination directory; where that isn't available SteadQ falls back to a named temporary file under tmp/. Before publication SteadQ calls fsync() on the file: pushing everything down onto the storage device.

Now you can "publish" this new job into a sharded subdirectory under ready/ with a no-overwrite publication step (normally linkat() for an O_TMPFILE, or renameat2(..., RENAME_NOREPLACE) for the named fallback) with some path name such as:

<job-id-32hex>.g0000000000000000.a00000000.m00000005.k<name-tag-16hex>.sqj

Here I'm using .g<generation> to denote the generation, which starts at 0 (this job has never existed before), .a<attempt> denotes the number of successful claims so far, which also starts at 0 for brand new jobs. The .m<maximum-attempts> means we allow up to five committed claims on this job, and .k<name-tag> is a 64-bit integrity tag derived from the queue ID and canonical name context using SHA-256. These numeric fields are fixed-width lowercase hexadecimal; the tag is not a keyed MAC.

Now any consumers that want to process jobs can scan one of those ready shard directories and pick a job. If you want to work on this then you try create a "lease" state, by renaming it again in the same ready/<shard>/ directory - but now the name contains some extra information (the boot ID, boot-time and wall-clock lease deadlines, an incremented attempt number and a large random value - the "lease token") such as:

<job-id-32hex>.g0000000000000001.a00000001.m00000005.o<boot-id-32hex>.b<boottime-deadline-16hex>.w<wall-deadline-16hex>.t<lease-token-32hex>.k<name-tag-16hex>.sqj

If you succeed then congratulations - out of all the processes who were scanning that shard directory for jobs: you now own this job!

If two processes tried to acquire the same lease then only one of those calls would succeed and so - without a lock or central arbitrator deciding ownership: we know who owns the job.

As long as that process can renew this lease before it expires (which we track via both deadlines inside the file name) then everything is working properly, and when it finishes working on that job - we re-verify the payload and create a durable receipt by moving the leased job into receipts/ and calling fsync() on the required directories:

<job-id-32hex>.g0000000000000002.a00000001.m00000005.t<lease-token-32hex>.k<name-tag-16hex>.rct

where .rct denotes an acknowledgement receipt. The full receipt is the verified job object moved into terminal receipt state (and can later be compacted while preserving verified evidence). If someone else tries to acknowledge this job using an older lease, or some other state - then they will fail: because only the holder of that latest version can produce a valid acknowledgement record.

If it fails, and you want to retry later - we move it under the bucketed delayed/ directory with something like this:

<job-id-32hex>.g0000000000000002.a00000001.m00000005.d<not-before-unix-ns-16hex>.k<name-tag-16hex>.sqj

and if you've exhausted the maximum number of attempts for this job then it gets moved under the bucketed dead/ directory with an .x<reason> code rather than a retry time.

If the consumer crashes, or there's an unclean shutdown then on recovery: we just scan those file system directories and look for expired or old-boot leases, or delayed jobs that have become eligible. There's nothing else we need to reconcile against.

All of which begs the question - how do you know when something actually happened? The reason I ask this, is because as far as files are concerned - it depends on when you look at them. This turned out to be rather important: and took up a lot of time, energy and lines of code.

When exactly did that thing happen?

Like most things in computing - files can be somewhat complicated. Yes, some filesystem operations are specified as atomic: but "atomic" doesn't always mean what you might think. It certainly does not necessarily imply that the operation survived a crash, or power being cut to your computer.

A system call can succeed - but that doesn't mean it was made durable. And just because a directory entry changed (such as via rename()) - that doesn't mean it's now permanently there.

Once I actually started writing a specification for SteadQ, something that ended up helping me think about this was splitting each operation into one of three results:

This last possibility is important, because the operation might have succeeded and persisted: but you can't tell from looking at it. I cannot emphasize how important this is to think about (and plan for). Just because an operation might have happened does not make your computer more reliable. It means that if something crashes at the wrong point, then you could observe one of two things when it comes back up.

Let's imagine for the moment that we have some directory, and there are two files in it:

<job-id-a-32hex>.g0000000000000005.a00000000.m00000008.k<name-tag-a-16hex>.sqj

and

<job-id-b-32hex>.g0000000000000005.a00000000.m00000008.k<name-tag-b-16hex>.sqj

Further, let's say that you want to make a hard link called job-4321-hardlink.sqj that points to the same file. You call link(), and it succeeds: but power is cut immediately after this happens.

When the system boots up again you might observe:

<job-id-a-32hex>.g0000000000000005.a00000000.m00000008.k<name-tag-a-16hex>.sqj
job-4321-hardlink.sqj
<job-id-b-32hex>.g0000000000000005.a00000000.m00000008.k<name-tag-b-16hex>.sqj

or equally possible - that the hard link doesn't actually exist at all! It might not be there. And given we only have access to what's on disk after a crash (and the fact that a successful namespace change need not have been made durable) means we cannot tell from that post-crash state whether or not the link had been created.

SteadQ uses these terms to talk about the outcomes of different operations. This is also important when we're thinking about durability barriers: because if the linearization point and all required durability barriers completed successfully then we can say with confidence - yes, this thing definitely happened. This is why I call fsync() and related calls "durability barriers".

I got fairly carried away with this idea, because once I started testing SteadQ: it became very clear that assumptions about files and path names could not be taken for granted. Writing the first few fault injection tests (that would fail system calls at controlled boundaries using SteadQ's built-in fault injection hooks) was eye opening to say the least. So too were 700+ tests across unit, integration, conformance and formal model checking, written by and to constrain agents of course and force robust engineering.

With the help of some crash testing infrastructure that I built (inspired in part by FSCQ, and built on Linux's dm-log-writes / replay-log machinery) which can replay block-level writes up to each persistence barrier: I was able to test SteadQ against ext4, XFS, btrfs, f2fs and ZFS on two different Linux hosts. Between the two boxes we checked 761 crash states on one kernel and 793 on the other, and all five filesystem profiles passed. This is only block-replay evidence for those recorded workloads, not physical power-cut testing. I will tackle this shortly, but I've been too lazy to dig out my Raspberry Pi's.

All of this infrastructure emerged because I wanted to rename some files in a deterministic way. But that's okay as we now know just how far this idea can go and whether it could be reliable under those tested assumptions.

Why Bother With This?

At the moment SteadQ is nowhere near as feature-rich or battle-tested as any number of fantastic pieces of queue software already out there. It's also not Kafka: it doesn't do network distribution, FIFO ordering guarantees (strictly), priorities of any kind - nor can you use it to do transactions or produce exactly once side effects.

However there are lots of cases where work happens on a single machine (or appliance), and you don't necessarily want to run another service just in order for your local processes can talk reliably.

Video transcoding (something that effects my work and was part of the genesis of this idea) is a great example of this where one process produces render jobs and some worker processes chew through them: generating proxies for editing, thumbnails for seeking or transcoding the final result into different bit rates and resolutions. If one of those workers crashes then their lease expires, and recovery can free the job up for someone else to pick it up again.

This same pattern fits running AI models or agents on your local machine extremely well: different processes can enqueue and process work from each other without needing to orchestrate the whole thing with a layer like a Redis server.

I could see this being used for build workers on a big machine that runs CI jobs, download and ingest pipelines (such as from an NDI source), processing of durable notifications for home automation, image generation or just running background indexing and backups on a Linux NAS's local storage. It's also perfect for small or embedded systems where added services might become a constraint.

Basically anywhere that you find yourself combining some mix of cron, shell scripts and ad-hoc programs to run work: there may be space for this. If you can't tell - I've gotten carried away with the idea of using a file system as queue, but it's strangely satisfying when something works exactly how you think it should.

It is also reasonably quick: on the repository's Intel Core i5-13500 reference system with NVMe ext4 storage - SteadQ can process roughly 3,000 small jobs per second with strict individual commits through a single worker. With eight workers it can do more than 8,000 jobs per second through it. With batching this pushes past 10,000 jobs/s. Admittedly, some file systems are slower than others (looking at my ZFS mirror).

A "job" in this case isn't just an enqueue/dequeue: instead we track it from the moment where a producer writes out and publishes that job, all the way through when some worker acquires it from ready/, verifies its payload and metadata against their hash values, all the way through when they acknowledge that job successfully. The only thing not included here is whatever work a worker might do.

I should also mention that these numbers come from using an at-least-once local queue. This means that if there's a crash or power cut then jobs may get processed again, and SteadQ does not guarantee exactly-once external side effects even on the same machine. If you're looking for something faster or more elaborate than this, then I'd suggest that Redis is probably a good place to start looking for it.

However unlike many other pieces of queue software out there - SteadQ does not require any kind of daemon, or external service: you just point it at a file path.

Pick your poison.

What About Networked File Systems?

One thing you will notice about SteadQ is that there's an explicit assumption of locality: all the processes are on one machine. So naturally, after finishing SteadQ and staring work on this article - I started wondering whether it was possible to do the same sort of thing again - but using object storage this time.

This is slightly more complicated because - unlike the file system: S3 general-purpose buckets (for example) don't support things like atomic renames (though some special new buckets do so that's something to pay attention to). The good news is that object stores can support atomic conditional creation, which is what StowQ uses instead.

Here each job in the queue continues to be an immutable object, whose key contains some information about it (such as its ID and generation number). In order for a consumer to "take ownership" of that job they need create an immutable claim object, whose key is based on information from the original - such as how many times it has been attempted (or when this claim expires). Instead of having an atomic no-overwrite rename select the winner we instead use conditional creation, and if that is successful then, out of everyone who was scanning the bucket for jobs, you now own it.

Just like with SteadQ, there is no central service or leader, just a bunch of stateless processes who can figure out everything they need to know by scanning the object store (or bucket). However unlike SteadQ, this isn't limited to a single machine, and you can figure out the state by looking at those key names. There are tradeoffs of course (completed jobs per second) is in TENS not thousands, but for some use cases it's viable and it's an interesting exercise regardless. More on this in the future.

This all started off as an incredibly stupid idea, but unfortunately it keeps working - which means I have to find out just what is possible. For what it's worth - I think there might be some mileage in this. But regardless of the outcome, it's been a really fun journey so far.