Race-Conditions on houdeshell.dev

Software that I

Mon, 01 Jun 2026 00:00:00 +0000

I bounce between Mac, Windows, and Linux daily. Honestly, the tools matter more than the OS at this point. Great software is great software, and I’ve been lucky enough to build a career on top of things other people built well.

This is the stuff I actually reach for. Not a “best of” list, not sponsored, not comprehensive. Just software that makes me better at what I do. Or at least makes the work more fun.

Software Development #

IDE

JetBrains Rider #

Cross-platform .NET IDE. Fast, smart, and doesn’t need Visual Studio’s weight to get the job done.

IDE

Visual Studio #

The OG. Heavy, but when you need the full .NET debugging experience, nothing else comes close.

editor

VS Code #

Extension ecosystem is unmatched. Somehow Electron done right.

editor

Neovim #

Vim reborn. Lua config, LSP native, and a plugin ecosystem that won’t quit. :wq is a lifestyle.

version control

Git #

The version control system that won. Love it or hate it, you can’t ship without it.

version control

lazygit #

Terminal UI for git that makes interactive rebases feel like cheating.

JSON

jq #

sed for JSON. Once you learn the syntax, you’ll pipe everything through it.

database

JetBrains DataGrip #

SQL IDE that actually understands your schema. Autocomplete that works across joins.

database

DBeaver #

Universal database tool that actually works. Connect to anything, query everything.

database

SSMS #

Microsoft SQL Management Studio. If you’re in SQL Server land, you already know.

database

PostgreSQL #

The database that keeps getting better. Extensions, JSON support, and rock-solid reliability.

containers

Docker #

“Works on my machine” became “works on every machine.” Changed how we ship software.

Languages #

backend

C# #

Pays the bills. Span<T> and ArrayPool mean the hot paths stay in managed code instead of dropping to C.

web

TypeScript #

JavaScript with adult supervision. The type system is stranger and better than people expect, right up until someone reaches for any.

services

Go #

Boring on purpose, which is the whole point. You’ll write if err != nil ten thousand times and ship something that runs for years.

systems

Rust #

The borrow checker is a reviewer who never gets tired and is never wrong. Miserable for a week, then that entire class of bug is gone.

the metal

C #

Still the substrate. Every abstraction you like is a polite lie told on top of it.

the metal

C++ #

Powerful enough to build anything, sharp enough to take a finger off. Worth it when the alternative is worse.

AI #

agent

Claude Code #

AI pair programmer in your terminal. It’s writing this page right now.

agent

VS Code Agents #

Native agent mode inside the editor. Let the assistant plan, edit files, and run commands without leaving VS Code or babysitting a separate CLI.

agent

Codex #

OpenAI’s open-source coding agent. Terminal-native, sandboxed by default, and you can swap the model behind it whenever.

platform

Azure AI Foundry #

Microsoft’s AI app stack. Build agents, point them at your data, ship them on Azure. The enterprise wrapper for everything you’d otherwise glue together yourself.

Infrastructure #

orchestration

Kubernetes #

Container orchestration that makes you mass produce YAML for a living.

orchestration

K3s #

All of Kubernetes in a single binary. Perfect for homelab, edge, and production deployments.

IaC

Terraform / OpenTofu #

Infrastructure as code. OpenTofu if you prefer the open-source fork without the license drama.

networking

Tailscale #

WireGuard-based mesh VPN that just works. Connect everything without opening a single port.

observability

Grafana #

Dashboards for everything. Pairs with Prometheus, Loki, and Tempo for the whole observability stack.

Kubernetes

k9s #

Terminal UI for Kubernetes. Makes cluster management feel like a video game. Way faster than raw kubectl.

Kubernetes

Lens #

The Kubernetes IDE. When you want to point and click your way through a cluster without shame.

it's complicated

Cloudflare #

You love them. You hate them. Your DNS is already there. Their edge network is everywhere, their pricing is unbeatable, and their product sprawl is terrifying. Stockholm syndrome as a service.

Shell & Terminal #

shell

Bash #

The shell that’s been there since before you were born. Simple, portable, everywhere.

shell

Zsh #

Bash’s cooler sibling. Tab completion, globbing, and plugin support that actually makes the terminal enjoyable.

prompt

Starship #

Cross-shell prompt written in Rust. Fast, pretty, and infinitely configurable.

multiplexer

tmux #

Terminal multiplexer. SSH into a box, detach, come back tomorrow. Your session is still there.

coreutils

eza #

Modern replacement for ls. Colors, git status, and tree view out of the box. Written in Rust, obviously.

Utilities / Apps #

macOS

AltTab #

Why does macOS think a minimized window is an inactive window and not let you switch to it? This fixes that.

About

Sat, 11 Apr 2026 00:00:00 +0000

// bit herder. pointer wrecker. yaml apologist.

I’m a bit herder, a pointer wrecker, and HTTP is my best friend. 0xA3 & 0x7C are also my friends, but not 0x08.

I have strong opinions about most of this and I hold every one of them loosely. Show me better data and I’ll change my mind on the spot, without being weird about it. Ideas are supposed to adapt. So are the people holding them.

I spend my days somewhere between writing code and making sure other people can write better code. I love distributed systems, low-level embedded devices, and anything that has a content-type. I also have an unhealthy relationship with YAML, but that’s a Kubernetes problem.

The day job #

During the day, I lead engineering and operations at EnergyCAP, a SaaS platform that helps organizations track utility spend, energy consumption, and sustainability goals across massive building portfolios. We’re talking $100B+ in utility bill spend tracked across 350,000+ sites. Government, education, healthcare, commercial real estate. If it has a meter, we probably manage it.

I started here as a developer in 2009 and have been building ever since: through senior roles, into leadership, and eventually into the VP seat where I get to shape both the tech and the teams behind it. Some days that means architecture decisions. Most days it means removing blockers so smart people can do their best work. And yes, sometimes it’s just meetings about meetings.

My philosophy is simple: technology should serve people. Build teams where people feel safe to be wrong, give them the tools to be great, and get out of the way.

Things I geek out about #

Distributed systems, container orchestration, database performance tuning, developer tooling, process dumps, leadership that doesn’t suck, making engineers’ lives better, and the eternal quest for a perfect terminal setup.

I also talk at things #

I like getting on stage and nerding out about the things I’ve learned the hard way. My talks tend to fall into a few buckets, and yes, I have strong opinions about all of them.

AI & the developer experience #

The AI wave hit and I leaned in. I’ve talked about using LLMs as debugging partners, building RAG pipelines that actually know your business context, the ethical lines we should be drawing as we hand more work to machines, and what happens to the junior-to-senior pipeline when the tools eat the entry-level work.

We Broke the Way Junior Engineers Become Senior Engineers
My Rubber Duck is a Large Language Model
Unleashing the Power of the AI Wizards: Retrieval-Augmented Generation Spells

Kubernetes & cloud native #

I was late to the Kubernetes party, and I’ll tell you all about it. From workshops where we break clusters on purpose, to navigating the 900+ services in the CNCF landscape, to making the case that cloud-native principles work even when you’re not in the cloud.

Kubernetes: The Hard Parts, Learned the Easy Way (workshop)
Your Pod Is Pending and Nobody Knows Why (workshop)
Kubernetes Chronicles: Late to the Party, Big on Adventure!
K3s: Half the Size, Twice as Awesome (workshop)
Orchestrating Machine Learning Workloads with Kubernetes
Exploring the Cloud Native Landscape
Cloud Native is Only for the Cloud, Right?
Nomad: Orchestration Doesn't Start with a K

Performance & databases #

I’ve spent an unreasonable amount of time staring at query plans and disassembly. These talks cover squeezing performance out of SQL Server, using DMVs like cheat codes, chasing allocations below the level of Big-O, and that one time we took a 10-hour process down to 10 minutes.

The Allocation Is the Problem
Your Benchmark Is Lying to You
ReArchitecting Data: 10 Hours to 10 Minutes
Mastering SQL Server Performance Optimization (workshop)
SQL Server DMVs That Give Me Superpowers
Achieving Continuous High Performance with Query Store
Practical High Performance: C# Edition
Intrinsics in .NET: Start Somewhere

Production war stories & leadership #

Production breaks. Systems fail. The interesting part is what you do next, and how often the retry you added to be safe is the thing that took you down. I also talk about the human side: what I’ve learned (and gotten wrong) leading engineering teams.

Your Retry Is the Outage
The Postmortem Is Not the Point
The Corpse Knows: Debugging Production with Process Dumps
Bug Squashing with Process Dumps
Recovery by Design: A Postmortem Adventure
Joining the Cloud: Our Journey
TIL as a CTO

The wildcard #

Sometimes you just want to build something fun with your hands.

Paper Circuits: Origami for a New Generation

Let’s talk #

I drink a bit too much coffee while rambling on about technology. If you’re willing to have a conversation, I’m ready to buy you a cup.

email: chris@houdeshell.dev
github: choudeshell
linkedin: choudeshell
bluesky: @choudeshell.bsky.social

Now

Mon, 01 Jun 2026 00:00:00 +0000

The Continuation That Never Woke Up

Sat, 05 Sep 2026 10:00:00 -0400

An order got stuck halfway through post processing. Somebody re-ran it. It went through fine. Ticket closed.

A few weeks later it happened again, and we did the same thing, because the same thing worked.

That is a great way to never fix anything.

The pipeline is not complicated. Two enrichment jobs run, a validation job reads what they wrote, a notification goes out at the end. Most orders go through without anyone ever thinking about them.

The stuck ones had nothing visibly wrong. No exception. No failed job. Nothing red in the dashboard, nothing in the error logs. Just a record sitting in an intermediate status waiting on a validation step that was never going to run.

And the retry always worked, which is the perfect disguise. Intermittent, self healing, no stack trace, no customer screaming. Nobody escalates that. It goes on the pile with the flaky tests and the one integration that times out on Mondays.

I looked at it three times over about two months and gave up twice.

When I finally got it, the answer had been in our logs the whole time. Naming the exact batch id. At a log level we had deliberately filtered out.

The lock comes off at the red dot. The continuation is not in storage until the purple one.

TL;DR #

Symptom: a batch stuck in Awaiting forever. The batch it was waiting on already Succeeded. The job inside it sits in the created set and never gets a state applied.

Cause: a continuation created nested inside an enclosing batch, whose antecedent was created with the static BatchJob.StartNew. The antecedent commits and starts running immediately. The continuation does not exist in storage until the enclosing batch commits, which can be hundreds of milliseconds later. If the antecedent finishes inside that window, the wakeup is lost, and nothing retries it.

Fix: one word.

diff-var enrichmentBatch = BatchJob.StartNew(enrich =>
+var enrichmentBatch = batch.StartNew(enrich =>

Go grep your logs for this. Warn level, from Hangfire.Batches.BatchContinuationsSupportAttribute:

textCould not start a continuation batch '<batchId>' for batch '<batchId>': it does not exist or about to expire

If that string is in there, this is happening to you right now.

One caveat before any of the rest. Hangfire.Pro is a commercial, closed source product. Everything below about its internals came out of decompiling the shipped assembly, so it is implementation detail and it can change between releases. Version under test: Hangfire.Pro 3.0.5, Hangfire.Core 1.8.23, SQL Server storage.

What it looks like in the dashboard #

Batches, then Awaiting Batches. There is a batch that has been sitting there for two days.

Open it. The Created tab lists the ValidationJob that never went anywhere. Now go find the enrichment batch it was waiting on.

Succeeded. Hours ago.

Those two facts cannot both be right. A continuation still waiting, on something that already finished.

The clue I had the whole time #

There is exactly one log line for this, and it names the batch id:

textCould not start a continuation batch '...' for batch '...': it does not exist or about to expire

It is at Warn, from Hangfire’s own namespace, and that is the whole problem. Hangfire is chatty at Info, so the standard move is to push Hangfire.* up to Error or drop it on the floor. We had done that. Sensible at the time.

So the library had been telling us exactly which batch was broken, by id, for months, into a filter.

I am not going to pretend that felt good.

If you read nothing else here, go run that grep. It takes a minute and it either finds nothing or it ruins your afternoon in a useful way.

Batches, quickly, for people who have only used plain Hangfire #

Plain Hangfire gives you BackgroundJob.Enqueue and BackgroundJob.ContinueJobWith. A job, and a job that runs after a job.

Hangfire.Pro adds batches. A batch is a group of jobs treated as one unit, so you can say “when all of these finish, do this” without tracking individual job ids. BatchJob.StartNew makes one. BatchJob.ContinueBatchWith makes a batch that waits on another batch. Batches nest, and a parent is not done until its children are.

Here is the part that matters later: Pro has no tables of its own. On SQL Server a batch is a pile of rows in the generic Hash, Set and List tables, keyed by convention.

Key	Table	Holds
`batch:{id}`	Hash	`CreatedAt`, `Description`, and `ParentId` if nested
`batch:{id}:state`	Hash	single field `Data`, current state as JSON
`batch:{id}:states`	List	state history
`batch:{id}:created` / `:pending` / `:processing` / `:succeeded` / `:finished`	Set	job ids by state
`batch:{id}:created:batches` and the same `:batches` suffixes	Set	nested batch ids by state
`batch:{id}:continuations`	Hash	continuations with non default options
`batch:{id}:continuations:succeeded` / `:finished`	Set	registered continuation batch ids
`batches:started` / `:awaiting` / `:awaiting-job` / `:completed` / `:succeeded` / `:deleted`	Set	the per state indexes the dashboard reads, `Score` is the Unix seconds timestamp of entry

States are Created, Started, Awaiting, Completed, Succeeded, Deleted. Learn that key layout. It is the only reason the SQL at the end of this post is possible.

The code #

Simplified, but this is the shape. One outer batch holds the enrichment and validation pipeline plus some unrelated shipping, tax and inventory work, and a single continuation notifies downstream once all of it is done.

csharppublic string EnqueueOrderPostProcessing(OrderPayload payload)
{
 var batchId = BatchJob.StartNew(batch =>
 {
 EnqueueEnrichmentAndValidation(batch, payload);

 // ... several more batches created here for shipping, tax and inventory work ...
 var shippingBatch = BatchJob.StartNew(b => b.Enqueue<ShippingJobs>(x => x.Recalculate(payload)));
 var taxBatch = BatchJob.ContinueBatchWith(shippingBatch, b => b.Enqueue<TaxJobs>(x => x.Recalculate(payload)));
 BatchJob.ContinueBatchWith(taxBatch, b => b.Enqueue<InventoryJobs>(x => x.Reserve(payload)));
 });

 BatchJob.ContinueBatchWith(batchId, batch =>
 {
 batch.Enqueue<INotifier>(n => n.NotifyDownstream(payload));
 });

 return batchId;
}

private void EnqueueEnrichmentAndValidation(IBatchAction batch, OrderPayload payload)
{
 foreach (var chunk in payload.OrderIds.Chunk(BatchSize))
 {
 var chunkList = chunk.ToList();

 // BUG IS HERE: static BatchJob.StartNew commits immediately, and these jobs start running now
 var enrichmentBatch = BatchJob.StartNew(enrich =>
 {
 enrich.Enqueue<PricingJob>(j => j.Run(chunkList));

 if (_featureFlags.IsEnabled(FeatureFlag.FraudCheck)) // a database round trip, per chunk
 {
 enrich.Enqueue<FraudCheckJob>(j => j.Run(chunkList));
 }
 });

 // ...but this continuation is a CHILD of the outer batch, so it is not written to
 // storage until the enclosing BatchJob.StartNew commits
 batch.ContinueBatchWith(enrichmentBatch, validate =>
 {
 validate.Enqueue<ValidationJob>(j => j.Run(chunkList));
 });
 }
}

Look at those two calls sitting next to each other. BatchJob.StartNew and batch.ContinueBatchWith.

They read like siblings.

What I thought it was first #

Chunking. Obviously chunking.

The failures clustered on bigger orders, bigger orders meant more chunks, and more chunks meant more of everything. I spent a while convinced the loop was racing itself, that two chunks were somehow stepping on each other’s batch ids. Added logging around the chunk boundaries. Found nothing, because there was nothing.

Then the feature flag, because IsEnabled does a database round trip and it is inside the loop. Wrong again, but closer than I knew at the time. The flag call was not the cause. It was making the window wider.

What finally moved it was reading the warning properly. Not “some continuation failed” but “it does not exist”. Present tense. Storage was asked for a batch and there was nothing there.

That is a strange thing to be true about a batch you are in the middle of creating. Unless it has not been created yet.

Two ways to make a batch #

This is the whole post, so it is worth being slow about.

BatchJob.StartNew(...), the static facade, goes to BatchJobClient.Create. It opens its own storage connection, takes a distributed lock on the new batch id, writes everything, commits. Immediately. The jobs inside are enqueued and a worker can pick one up the instant the call returns. By the time your next line of C# runs, that batch might already be executing.

batch.StartNew(...) and batch.ContinueBatchWith(...), called on the IBatchAction you are currently inside, create a nested batch. BatchAction.Create stamps ParentId into the hash and queues the child in memory. Then in BatchFactory.Create:

csharpcontext.Transaction.SetRangeInHash("batch:" + context.BatchId, context.Batch);
if (!context.Batch.ContainsKey("ParentId"))
{
 _stateMachine.ApplyState(...);
}

Read the condition. A nested batch does not get its state applied there. All of it goes into the outer batch’s transaction, and none of it exists in storage until the enclosing BatchJob.StartNew returns.

Same verb. Same shape where you type it. Completely different commit semantics, and nothing in the type system is going to mention it.

One thing that is not obvious #

An aside, because it cost me an afternoon: IBatchAction only declares BatchId. The reason batch.StartNew(...) compiles at all is that it inherits IBackgroundJobClient, IBatchJobClient and IBackgroundJobClientV2. You will not work that out from the interface declaration.

Pro is not naive about this #

Worth saying, because the obvious version of this race is handled. When a batch is created in BatchAwaitingState, BatchContinuationsSupportAttribute.OnStateElection calls AddContinuation. Paraphrased from the decompiled source:

csharpusing (context.Connection.AcquireDistributedBatchLock(parentBatchId, TimeSpan.FromMinutes(1)))
{
 using var tx = context.Connection.CreateWriteTransaction();

 tx.AddToSet("batch:" + parentBatchId + ":continuations:succeeded", context.BatchId);

 var parentState = context.Connection.GetBatchState(parentBatchId);
 if (parentState == null)
 throw new InvalidOperationException("Can not find antecedent batch ... to create a continuation.");

 if (parentState.IsFinal)
 {
 context.CandidateState = ShouldStartContinuation(parentState, options)
 ? new BatchStartedState { Reason = "Antecedent batch was already finished" }
 : new BatchDeletedState { Reason = "Continuation condition was not satisfied" };
 }

 tx.Commit(); // registration is committed here, separately, and the lock is released
}

It takes a lock on the antecedent. It re-reads the antecedent’s state under that lock. If the antecedent already finished, it starts the continuation right there rather than waiting on a wakeup that is never coming.

Somebody thought about this.

The catch is what IsFinal actually sets. A candidate state. The continuation’s own BatchAwaitingState row gets written afterwards by BatchStateMachine.ApplyState, into the outer batch’s transaction.

Which is still open.

The lost wakeup #

The other half. When the antecedent goes final, OnStateApplied calls ExecuteContinuationsIfExist, which walks the continuation sets and calls into the state changer for each id:

csharpif (_process.ChangeState(new BatchStateChangeContext(
 context.Storage, context.Connection, continuationBatchId,
 new BatchStartedState { Reason = "Antecedent batch has finished" },
 BatchAwaitingState.StateName)) == null)
{
 _logger.Warn("Could not start a continuation batch '" + continuationBatchId +
 "' for batch '" + context.BatchId + "': it does not exist or about to expire");
}

And BatchStateChanger.ChangeState opens with this:

csharpvar hash = context.Connection.GetAllEntriesFromHash("batch:" + context.BatchId);
if (hash == null || hash.Count == 0)
{
 return null;
}

No hash, no batch. Return null, log a warning, move on.

The sequence #

Put the halves together:

Under the lock on the enrichment batch, the validation continuation is registered into batch:{enrichment}:continuations:succeeded and committed. Enrichment is not final yet, so the candidate state stays Awaiting. The lock is released.
The outer transaction keeps building. More chunks. A feature flag lookup per chunk. Three more batch creations for shipping, tax and inventory. Tens to hundreds of milliseconds of database round trips.
The enrichment jobs, running since step 1, finish. Enrichment goes Succeeded. ExecuteContinuationsIfExist finds the validation batch id in the set and calls ChangeState.
batch:{validation} is not in storage. It is inside the outer transaction, uncommitted. ChangeState returns null, the warning fires, and nothing ever retries it. Continuations fire once, off the antecedent’s state transition, and the antecedent is now final forever.
The outer transaction commits and writes the validation batch as Awaiting. It stays that way permanently, ValidationJob sitting in its created set having never had a state applied.

The registration and the wakeup are serialized against each other by a lock on the antecedent. But the lock comes off before the continuation’s own state row is committed.

That gap is the bug.

And it is not a microsecond gap. It is however long the rest of the enclosing batch build takes. Ours was a per chunk feature flag call plus three more batch creations, which is why this showed up sometimes instead of never. A tighter loop hides it for years. A slower database makes it constant.

So my chunking theory was not wrong, exactly. It was one layer off. More chunks did cause it, but not by racing each other. They just made the window longer.

Three outcomes that look the same and are not #

Two of these are not this bug, and I checked the wrong one first.

What you see	What it means
Continuation stuck in Awaiting forever, antecedent already final	The lost wakeup above
Continuation in Deleted, reason “Continuation condition was not satisfied”	The antecedent finished, but not in a state matching the continuation options
Continuation in Started, reason “Antecedent batch was already finished”	The race Pro does handle, caught under the lock

The middle row deserves its own paragraph, because it catches a lot of people and it is not a race at all. ContinueBatchWith defaults to BatchContinuationOptions.OnlyOnSucceededState. One permanently failed job anywhere in the antecedent means the antecedent never reaches Succeeded, and your continuation is not stuck. It is deleted. Quietly, with a reason string you only see if you open that batch in the dashboard.

If you are hunting a continuation that never ran and it is not in Awaiting, look in Deleted before you assume you have a race.

The fix #

diff- var enrichmentBatch = BatchJob.StartNew(enrich =>
+ var enrichmentBatch = batch.StartNew(enrich =>

Now the antecedent and the continuation are both nested children of the same outer batch, created in one transaction. BatchStateMachine.SortContinuations handles exactly this:

csharpelse if (createdItem is BatchedBatch batchedBatch &&
 batchedBatch.InitialState is BatchAwaitingState awaiting)
{
 if (createdBatchIds.Contains(awaiting.ParentBatchId) && !seenBatchIds.Contains(awaiting.ParentBatchId))
 {
 AddBatchContinuation(awaiting.ParentBatchId, batchedBatch); // deferred until after its antecedent
 }
}

A continuation whose antecedent is a sibling in the same batch gets deferred until after that antecedent’s state is applied. The enrichment jobs are enqueued in the same commit, so they cannot start before the continuation exists.

The window closes because there is nothing left in it.

The trade off is that the enrichment jobs start marginally later, after the outer commit rather than during it. Which is what you wanted anyway.

The rule worth remembering #

Do not mix batch scopes.

Either both the antecedent and the continuation are top level, each a static BatchJob.* call committing independently. That is safe, because AddContinuation re-checks the antecedent under a lock and the antecedent is already durable by then. Or both are nested in the same batch, which is the case SortContinuations was written for. The broken shape is specifically a nested continuation whose antecedent is an external, already committed, already running batch.

Hangfire.Pro 2.1.0’s release notes list “Continuations now work properly, when antecedent job/batch and continuation have the same batch” as a fixed item. The same batch pattern is something the vendor deliberately made work.

Digging out the wreckage #

Everything in this section is a last resort. It is for a system that already has orphans in it, not maintenance, and raw SQL bypasses the batch:{id}:lock distributed lock, so do it with the job servers stopped or in a genuinely quiet window. Default schema is HangFire.

Finding the orphans #

Awaiting batches, with the state of whatever each one is waiting on:

sqlSELECT a.Value AS BatchId,
 DATEADD(SECOND, CAST(a.Score AS BIGINT), '19700101') AS AwaitingSinceUtc,
 pid.ParentBatchId,
 COALESCE(ps.Value, '*** PARENT GONE ***') AS ParentState,
 (SELECT COUNT(*) FROM HangFire.[Set] s2
 WHERE s2.[Key] = 'batch:' + a.Value + ':created') AS CreatedJobs
FROM HangFire.[Set] a
LEFT JOIN HangFire.Hash st ON st.[Key] = 'batch:' + a.Value + ':state' AND st.Field = 'Data'
CROSS APPLY (SELECT JSON_VALUE(st.Value, '$.ParentBatchId') AS ParentBatchId) pid
LEFT JOIN HangFire.Hash ps ON ps.[Key] = 'batch:' + pid.ParentBatchId + ':state' AND ps.Field = 'Data'
WHERE a.[Key] = 'batches:awaiting'
ORDER BY a.Score;

Read those results carefully, because this is where you can do real damage. A batch whose parent is gone, or whose parent is already BatchSucceededState, BatchCompletedState or BatchDeletedState, is orphaned and will never start. A batch whose parent is BatchStartedState is legitimately waiting and is doing its job.

Deleting everything in Awaiting will destroy live work. Do not do that.

Getting the payload back #

The payload is still readable, so you can recover what those jobs were meant to do before cleaning anything up:

sqlSELECT DISTINCT j.Id, j.Arguments
FROM HangFire.[Set] s
JOIN HangFire.Job j ON j.Id = TRY_CAST(s.Value AS INT)
WHERE s.[Key] LIKE 'batch:%:created'
 AND j.InvocationData LIKE '%ValidationJob%';

Cleaning up without deleting #

For the cleanup itself, do not hard delete. Backdate ExpireAt on the batch’s Hash, Set and List rows and on its row in batches:awaiting, then let Hangfire’s own ExpirationManager sweep them. It runs delete top (@count) T ... where ExpireAt < @now, and JobExpirationCheckInterval defaults to 30 minutes.

There is a nice property in doing it that way. It self corrects. If you get one wrong and mark a batch that is actually still alive, its next state change calls PersistBatch, which clears ExpireAt again and rescues it from your cleanup script.

Leave the orphaned Job rows alone. CoreBackgroundJobFactory creates every job with ExpireAt set to creation plus 30 days, and a job in a batch that never started never has a state applied, so that expiry never gets cleared. They age out on their own, and they are not sitting in a queue in the meantime.

Prior art, such as it is #

I went looking for other people hitting this before I opened the decompiler. There is less out there than you would expect.

The closest is a forum thread, Batch Continuation Stuck. The reporter got impressively close: they narrowed it to “Job 1 in Batch 1 starts at the exact same time that Job 2 in Batch 2 is created”, with database timestamps colliding at the millisecond. Nobody from the project ever answered. No resolution.

HangfireIO/Hangfire#2080, “Hangfire.Pro multiple continuations - batches can be stuck in the Started state”, is open, but it is about jobs stuck Enqueued and fetched job counts. Different animal. HangfireIO/Hangfire#1035, “Continuations stuck in Awaiting State”, is a 2018 report against Hangfire 1.6.17 with a screenshot and not much else.

Pro 3.0.1/2.3.3 and 3.0.2/2.3.4 did ship nested batch fixes: empty nested batches remaining Started, nested batch jobs not triggering batch updates, a lock ordering deadlock when attaching nested batch continuations. None of them is this. 3.0.5 is a CSS only patch.

There is a decent reason for the silence, and it is not that nobody hit this. Pro is closed source and paid, so Pro bugs go to a support email rather than a public tracker. Whatever conversations happened, happened in someone’s inbox. Searching that warning string finds nothing because the people who hit it had nowhere public to put it, which is its own small argument about paid closed source infrastructure, and I will leave it there.

What I actually take from this #

The library is not doing anything indefensible. It guards the obvious race on purpose, with a lock and a re-read, and it guards it correctly. The failure only surfaces at the seam between two APIs that read as equivalent at the call site and are not underneath.

BatchJob.StartNew and batch.StartNew differ by four characters. One means “write this now and let the workers have it”. The other means “add this to the transaction I am already building”. Nothing where you type it tells you which is which, and both compile, and both look completely reasonable in review. I reviewed it. It looked completely reasonable.

Anyway. Go grep for Could not start a continuation batch.

Finding It Isn't Fixing It

Thu, 20 Aug 2026 09:00:00 -0400

GitHub found the broken component about three hours into the August 17 outage. They were another four hours and thirty-nine minutes from done.

The status timeline has it at 16:36 UTC: “identified the problematic component and have taken corrective actions.” Then 17:30, Git Operations degraded again. 19:13, they partially disabled authentication token retries. Resolved 21:15. Seven hours forty-seven minutes, and most of it came after they knew what was wrong.

Nobody deployed anything. From the writeup, about this one and the one before it: “Neither outage was caused by a code or configuration change. Both incidents were capacity failures at their core.” Load balancers in Central US saturated on a new traffic peak, an Istio sidecar hit its concurrency limit without scaling, and four HAProxy nodes went with it.

Then the retries piled on. The Copilot token service normally sits around 7-9K RPS. During the incident it took 70-100K. That’s not user traffic. That’s retry logic doing what it was told.

The growth in that post is the context. 1.4 billion commits in April, 2.9 billion in August. Azure went from 12% of platform load in May to 58%. Move that fast and something in the stack is going to be the piece that didn’t scale.

Coming back up is a different engineering problem than not going down, and it usually gets less attention.

Prevention gets the budget: headroom, redundancy, review gates. The recovery path gets a quarterly drill where everything cooperates. GitHub’s remediation list is mostly the second category. Retry budgets, variable timeouts, isolating critical systems, removing shared dependencies.

Measure how long it takes you to come back. That number matters as much as the one on your status page.

Davit: A Native UI for Apple's Container Tool

Mon, 15 Jun 2026 10:00:00 -0400

I’ve spent the last week living in Apple’s container tool from the terminal. It’s good. But some days I want a window, a list of what’s running, and a button. Davit is that window.

Davit's dashboard — container services stopped

It’s a native SwiftUI app. No Electron, no web view stuffed in a title bar. That alone put it ahead of most of the desktop container tooling I’ve tried.

The part I like: it doesn’t shell out to the container binary and scrape the output. It talks to the same platform the CLI does, straight over XPC. So the dashboard, the live CPU chart, the container list, the log tail, the filesystem browser all read from the source instead of parsing text a human was supposed to read.

The screenshot above is the first thing I saw, and it made me laugh. “Container services stopped,” with a Start button. That’s the container system start dance from my last post, except now it’s a button I don’t have to remember.

Install is a one-liner:

brew install wouterdebie/tap/davit

If you don’t already have Apple’s container platform, Davit can install it for you.

It’s not GUI-only, either. The same binary does davit run, davit build, and a full davit compose up/down/logs/exec with docker-compatible flags. So I can click around when I want to look, and drop to the same tool in a script when I don’t.

Apple silicon, and you’ll want the container platform on the box. It’s young, and it’s MIT. Worth a look.

Helpful macOS Container Tool Commands

Sat, 13 Jun 2026 10:00:00 -0400

I’ve been living in Apple’s container tool for the last few days. A handful of commands come up a lot, and other team members have asked how, so I’m writing them down. Here are the three I reach for most.

container machine run --root runs your default machine as root.
usermod -aG sudo $user adds your user to the sudoers group once you’re root (install sudo first).
container machine logs shows the logs for your default machine.

Running Ubuntu on Apple's Container

Fri, 12 Jun 2026 14:00:00 -0400

After getting Apple’s container running, the thing I actually wanted was a normal Ubuntu box to poke around in. That took one more step than I expected: systemd.

A container runs whatever its CMD is as PID 1. The stock ubuntu:26.04 image doesn’t ship an init, so you boot into a bare shell and not much else. If you want services, units, and a machine that behaves like a real Ubuntu install, systemd has to be the thing running as init. So you build your own image.

Here’s the Dockerfile. Most of it is the well-worn systemd-in-a-container cleanup: install systemd, then delete the units that make no sense inside a container so it doesn’t sit there trying to start hardware that isn’t there.

dockerfileFROM ubuntu:26.04

ENV container container

RUN apt update -y && apt upgrade -y && apt install -y systemd init
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == \
systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/local-fs.target.wants/*; \
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
rm -f /lib/systemd/system/basic.target.wants/*;\
rm -f /lib/systemd/system/anaconda.target.wants/*;

VOLUME [ "/sys/fs/cgroup" ]
CMD ["/usr/sbin/init"]

Build it, create a machine from it, and run:

❯ container build -t local/devel -f devel.dockerfile
[+] Building 356.1s (7/7) FINISHED
 => [resolver] fetching image docker.io/library/ubuntu:26.04
 => [internal] load build definition from Dockerfile
 => [linux/arm64 1/3] RUN apt update -y && apt upgrade -y && apt install -y systemd init
 => [linux/arm64 2/3] RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do ... )
 => exporting to oci image format
local/devel:latest

❯ container machine create --name devel --set-default local/devel
devel

❯ container machine run
chris.houdeshell@devel:/Users/chris.houdeshell$

That last line is the Ubuntu shell. You’re in.

What the flags are doing, quickly:

container build -t local/devel -f devel.dockerfile: -t tags the image as local/devel; -f points at the Dockerfile by name, since I didn’t name it Dockerfile.
container machine create --name devel --set-default local/devel: builds a VM named devel from that image. --set-default makes it the machine everything else targets, so I don’t have to spell out the name on every command.
container machine run: boots the default machine and drops me into its shell.

One thing worth knowing: that build ran just shy of six minutes. The tool is young and the build path isn’t quick yet, so don’t assume it hung.

Apple's Container Tool

Fri, 12 Jun 2026 10:00:00 -0400

Apple has a CLI for running Linux containers on macOS now. It’s called container, and the part I care about is that it doesn’t need Docker Desktop.

The design choice that makes it interesting: there’s no single shared Linux VM that every container piles into. Each container gets its own lightweight VM. So you get real isolation for free, boots are quick, and it all sits on top of Apple’s Containerization Swift package. Images are plain OCI, so the registries and the pull/build/push workflow you already know carry over. The requirements are stricter than I’d like, though: Apple silicon, macOS 26 or newer.

You install it from a signed package on the releases page, then start the background service. The service is where I tripped.

Run anything before it’s up and you get this:

❯ container machine create --name dev --set-default fedora:latest
Error: interrupted: "XPC connection error: Connection invalid"
Ensure container system service has been started with `container system start`.

❯ container system start
Launching container-apiserver...
Testing access to container-apiserver...
Verifying machine API server is running...
No default kernel configured.
Install the recommended default kernel from [https://github.com/kata-containers/kata-containers/releases/download/3.28.0/kata-static-3.28.0-arm64.tar.zst]? [Y/n]: y
Installing kernel...

XPC connection error: Connection invalid is not a bug. The system service just isn’t running. Start it with container system start, say yes to the default kernel, and you’re good.

Docker Desktop on a Mac has always felt like more machinery than I needed. This is lighter. Give it a spin.

A Single Border and a Better Radius

Wed, 10 Jun 2026 10:00:00 -0400

I read the release notes. I just opened a Finder window and it feels better, and it took me a second to figure out why.

It’s two things. There’s now a single vertical border between the sidebar and the content pane, so the two regions actually feel like two regions instead of one dark expanse where the divider was only implied by a shift in shade. And the inner corners got a slightly larger, more consistent radius that lines up with the outer window. That’s it. No new feature, nothing to learn.

The smallest possible line in the right place does more than any amount of gradient or shadow. One border to say “this is a boundary,” one radius to make the corners agree with each other, and the window stops feeling vague.

Good detail work is invisible until you go back to the old version and feel how much you weren’t seeing.