Knative on houdeshell.dev

Software that I

Mon, 01 Jun 2020 00:00:00 +0000

I bounce between Mac, Windows, and Linux daily. Honestly, the tools matter more than the OS at this point. Great software is great software, and I’ve been lucky enough to build a career on top of things other people built well.

This is the stuff I actually reach for. Not a “best of” list, not sponsored, not comprehensive. Just software that makes me better at what I do. Or at least makes the work more fun.

Software Development

IDE

JetBrains Rider

Cross-platform .NET IDE. Fast, smart, and doesn’t need Visual Studio’s weight to get the job done.

IDE

Visual Studio

The OG. Heavy, but when you need the full .NET debugging experience, nothing else comes close.

editor

VS Code

Extension ecosystem is unmatched. Somehow Electron done right.

editor

Neovim

Vim reborn. Lua config, LSP native, and a plugin ecosystem that won’t quit. :wq is a lifestyle.

version control

Git

The version control system that won. Love it or hate it, you can’t ship without it.

version control

lazygit

Terminal UI for git that makes interactive rebases feel like cheating.

JSON

jq

sed for JSON. Once you learn the syntax, you’ll pipe everything through it.

database

JetBrains DataGrip

SQL IDE that actually understands your schema. Autocomplete that works across joins.

database

DBeaver

Universal database tool that actually works. Connect to anything, query everything.

database

SSMS

Microsoft SQL Management Studio. If you’re in SQL Server land, you already know.

database

PostgreSQL

The database that keeps getting better. Extensions, JSON support, and rock-solid reliability.

containers

Docker

“Works on my machine” became “works on every machine.” Changed how we ship software.

AI

agent

Claude Code

AI pair programmer in your terminal. It’s writing this page right now.

orchestration

JetBrains Air

Agent orchestration in IDE form. Run Claude, Codex, Gemini CLI, and Junie at the same time. Each one sandboxed, all of them working while you do something else.

agent

Codex

OpenAI’s open-source coding agent. Terminal-native, sandboxed by default, and you can swap the model behind it whenever.

platform

Azure AI Foundry

Microsoft’s AI app stack. Build agents, point them at your data, ship them on Azure. The enterprise wrapper for everything you’d otherwise glue together yourself.

Infrastructure

orchestration

Kubernetes

Container orchestration that makes you mass produce YAML for a living.

orchestration

K3s

All of Kubernetes in a single binary. Perfect for homelab, edge, and production deployments.

IaC

Terraform / OpenTofu

Infrastructure as code. OpenTofu if you prefer the open-source fork without the license drama.

networking

Tailscale

WireGuard-based mesh VPN that just works. Connect everything without opening a single port.

observability

Grafana

Dashboards for everything. Pairs with Prometheus, Loki, and Tempo for the whole observability stack.

Kubernetes

k9s

Terminal UI for Kubernetes. Makes cluster management feel like a video game. Way faster than raw kubectl.

Kubernetes

Lens

The Kubernetes IDE. When you want to point and click your way through a cluster without shame.

it's complicated

Cloudflare

You love them. You hate them. Your DNS is already there. Their edge network is everywhere, their pricing is unbeatable, and their product sprawl is terrifying. Stockholm syndrome as a service.

Shell & Terminal

shell

Bash

The shell that’s been there since before you were born. Simple, portable, everywhere.

shell

Zsh

Bash’s cooler sibling. Tab completion, globbing, and plugin support that actually makes the terminal enjoyable.

prompt

Starship

Cross-shell prompt written in Rust. Fast, pretty, and infinitely configurable.

multiplexer

tmux

Terminal multiplexer. SSH into a box, detach, come back tomorrow. Your session is still there.

coreutils

eza

Modern replacement for ls. Colors, git status, and tree view out of the box. Written in Rust, obviously.

Utilities / Apps

macOS

AltTab

Why does macOS think a minimized window is an inactive window and not let you switch to it? This fixes that.

About

Sat, 11 Apr 2020 00:00:00 +0000

// bit herder. pointer wrecker. yaml apologist.

I’m a bit herder, a pointer wrecker, and HTTP is my best friend. 0xA3 & 0x7C are also my friends, but not 0x08.

I spend my days somewhere between writing code and making sure other people can write better code. I love distributed systems, low-level embedded devices, and anything that has a content-type. I also have an unhealthy relationship with YAML, but that’s a Kubernetes problem.

The day job

During the day, I lead engineering and operations at EnergyCAP, a SaaS platform that helps organizations track utility spend, energy consumption, and sustainability goals across massive building portfolios. We’re talking $100B+ in utility bill spend tracked across 350,000+ sites. Government, education, healthcare, commercial real estate. If it has a meter, we probably manage it.

I started here as a developer in 2009 and have been building ever since: through senior roles, into leadership, and eventually into the VP seat where I get to shape both the tech and the teams behind it. Some days that means architecture decisions. Most days it means removing blockers so smart people can do their best work. And yes, sometimes it’s just meetings about meetings.

My philosophy is simple: technology should serve people. Build teams where people feel safe to be wrong, give them the tools to be great, and get out of the way.

Things I geek out about

Distributed systems, container orchestration, database performance tuning, developer tooling, process dumps, leadership that doesn’t suck, making engineers’ lives better, and the eternal quest for a perfect terminal setup.

I also talk at things

I like getting on stage and nerding out about the things I’ve learned the hard way. My talks tend to fall into a few buckets, and yes, I have strong opinions about all of them.

AI & the developer experience

The AI wave hit and I leaned in. I’ve talked about using LLMs as debugging partners, building RAG pipelines that actually know your business context, and the ethical lines we should be drawing as we hand more work to machines.

My Rubber Duck is a Large Language Model
Unleashing the Power of the AI Wizards: Retrieval-Augmented Generation Spells

Kubernetes & cloud native

I was late to the Kubernetes party, and I’ll tell you all about it. From lightweight K3s workshops to navigating the 900+ services in the CNCF landscape, to making the case that cloud-native principles work even when you’re not in the cloud.

Kubernetes Chronicles: Late to the Party, Big on Adventure!
K3s: Half the Size, Twice as Awesome (workshop)
Orchestrating Machine Learning Workloads with Kubernetes
Exploring the Cloud Native Landscape
Cloud Native is Only for the Cloud, Right?
Nomad: Orchestration Doesn't Start with a K

Performance & databases

I’ve spent an unreasonable amount of time staring at query plans. These talks cover squeezing performance out of SQL Server, using DMVs like cheat codes, and that one time we took a 10-hour process down to 10 minutes.

ReArchitecting Data: 10 Hours to 10 Minutes
Mastering SQL Server Performance Optimization (workshop)
SQL Server DMVs That Give Me Superpowers
Achieving Continuous High Performance with Query Store
Practical High Performance: C# Edition
Intrinsics in .NET: Start Somewhere

Production war stories & leadership

Production breaks. Systems fail. The interesting part is what you do next. I also talk about the human side: what I’ve learned (and gotten wrong) leading engineering teams.

Bug Squashing with Process Dumps
Recovery by Design: A Postmortem Adventure
Joining the Cloud: Our Journey
TIL as a CTO

The wildcard

Sometimes you just want to build something fun with your hands.

Paper Circuits: Origami for a New Generation

Let’s talk

I drink a bit too much coffee while rambling on about technology. If you’re willing to have a conversation, I’m ready to buy you a cup.

email: chris@houdeshell.dev
github: choudeshell
linkedin: choudeshell
twitter: @choudeshell

Now

Sat, 02 May 2026 00:00:00 +0000

~/$ cat now.md

last updated: May 2, 2026

Focus

Production AI agents at /\$DAYJOB/. Rolling out internal agent infrastructure with real customer data behind it. Compliance edges, eval suites, on-call playbooks. The fun stuff and the boring stuff in equal measure.
AI document extraction. Pulling structured data out of utility bills and the kind of PDFs that look like they were faxed in 2003. Two-phase validation: deterministic checks first (totals match, dates parse, units make sense), then a nondeterministic pass with a model when the deterministic layer can’t reach a verdict. The hybrid catches more than either side alone.
The Flue stack on Kubernetes. Self-hosted serverless for agents. Three posts in the can, more in the queue.
Cost-aware Kubernetes. Watching cluster spend like a hawk while AI workloads scale up unpredictably. There’s a talk hiding in here somewhere.

Stack This Week

TypeScript 7 beta. The Go rewrite is real. Compile feels like Go.
Flue + Knative. Self-hosted agent harness on K8s.

Reading

More LLM eval papers than I expected to.
Internal architecture decision records. More than I want to admit.

Thinking

An internal /\$DAYJOB/ session on the agent rollout.

Flue, Part 3: Self-Hosting Agents with Knative

Sat, 02 May 2026 18:00:00 -0400

Part 1 was the framework pitch. Part 2 was the agent I wanted to build. This one is for the people who can’t (or won’t) ship that agent to Cloudflare Workers.

The reasons vary. Compliance says all customer data stays in our VPC. Security says no third-party serverless on the data path. Procurement won’t sign another vendor contract this fiscal year. We already run Kubernetes for everything else, so why are we paying someone to schedule containers for us? All valid.

Self-hosting a Flue agent on Kubernetes is straightforward. The trick is doing it without rebuilding the things managed serverless gives you for free: HTTP routing, autoscaling, scale-to-zero, ingress, TLS. That’s where Knative earns its keep.

Why Knative

Knative is the K8s-native answer to serverless. You hand it a container image and a port. It hands you back a public URL, request-based autoscaling, scale-to-zero when nothing is calling, and TLS through cert-manager. The whole “deploy a webhook handler” thing collapses into a single Kubernetes resource.

For a Flue agent specifically:

The agent boots in a few hundred milliseconds. Cold start is fine.
Webhook traffic is bursty. Scale-to-zero saves real money.
Inbound traffic is HTTP only. No need for a service mesh.

OpenFAAS works too. So does spinning up a Deployment plus Service plus Ingress plus HPA by hand. I keep coming back to Knative because the Service resource is one file and the operational story (events, observability, autoscaling) is built in.

The Dockerfile

Flue ships as a Node CLI plus a function bundle. For deployment, build a small image with the function code and the SDK’s HTTP server in front:

FROM node:22-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev

FROM node:22-alpine
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY agent.ts jira.ts ./
COPY skills ./skills
ENV NODE_ENV=production
EXPOSE 8080
CMD ["npx", "flue", "serve", "--host", "0.0.0.0", "--port", "8080", "agent.ts"]

Two-stage build to keep the runtime image small. The skills directory comes along since the agent reads context files at runtime. flue serve is the SDK’s HTTP entrypoint that turns export default into a webhook handler at /.

If you’re running a Bun-flavored Flue agent, swap the base image for oven/bun:1-alpine and the CMD for bun run agent.ts. Same shape, fewer layers.

Push to whatever registry you’re using:

docker build -t registry.internal/agents/jira-triage:0.1.0 .
docker push registry.internal/agents/jira-triage:0.1.0

The Knative Service

The Service resource is the whole thing.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
 name: jira-triage
 namespace: agents
spec:
 template:
 metadata:
 annotations:
 autoscaling.knative.dev/min-scale: "0"
 autoscaling.knative.dev/max-scale: "10"
 autoscaling.knative.dev/target: "5"
 spec:
 timeoutSeconds: 90
 containerConcurrency: 1
 containers:
 - image: registry.internal/agents/jira-triage:0.1.0
 ports:
 - containerPort: 8080
 envFrom:
 - secretRef:
 name: jira-triage-secrets
 resources:
 requests:
 cpu: 100m
 memory: 256Mi
 limits:
 cpu: 1000m
 memory: 1Gi

The annotations are doing real work:

min-scale: 0 lets the agent drop to zero replicas when nothing is calling it. That is the entire point of running this on Knative instead of a plain Deployment.
max-scale: 10 caps blast radius. If a JIRA automation rule misfires and floods the webhook, it won’t take down the cluster.
target: 5 plus containerConcurrency: 1 tell Knative each pod handles one in-flight request at a time, and to scale up once five concurrent requests are queued.

timeoutSeconds: 90 is the upper bound for a single request. Most agent runs finish in 10 to 30 seconds. Ninety gives a comfortable cushion for the slow case where the model is grinding through a long classification context. If you regularly need longer than that, you’ve left “webhook handler” territory and want a workflow runner like Argo or a queue.

containerConcurrency: 1 is the safe default. Flue agents tend to hold an open session and a model connection per request. Sharing one Node process across multiple in-flight model calls works, but the concurrency math gets complicated and you trade simplicity for a small CPU win. Start at 1, tune later if cost matters.

Secrets

Secrets stay out of the Service manifest. Drop them in a Secret and envFrom them in:

apiVersion: v1
kind: Secret
metadata:
 name: jira-triage-secrets
 namespace: agents
type: Opaque
stringData:
 ANTHROPIC_API_KEY: "sk-ant-..."
 JIRA_HOST: "yourorg.atlassian.net"
 JIRA_EMAIL: "triage-bot@yourorg.com"
 JIRA_TOKEN: "..."

If you’re running External Secrets Operator or Vault Secrets Operator, point at the upstream store instead of stuffing keys into a YAML you’ll commit by accident. The Flue agent doesn’t care where the env vars come from. It just reads them.

The webhook URL

Knative gives the Service a public URL on whatever domain your cluster operator has configured. You’ll see it in kubectl get ksvc -n agents:

kubectl get ksvc jira-triage -n agents

NAME URL READY
jira-triage https://jira-triage.agents.example.internal True

Point your JIRA automation rule at that URL with whatever auth you want on it. Knative supports request authentication through the standard ingress controllers, so if you want HMAC verification, JWT, or a static bearer token, the Service is just an HTTP server and you wire it in upstream.

For HMAC (JIRA can sign webhooks), I add a small middleware layer in the Flue handler itself rather than at the ingress. That way the Service stays portable and the auth lives next to the agent code.

Scaling, the operator’s view

Knative does its own pod scaling, but the autoscaler still has to live somewhere. The defaults are reasonable, and if your team is already running KPA (the Knative Pod Autoscaler), you don’t need to do anything. If you’d rather use HPA, swap the annotation:

autoscaling.knative.dev/class: "hpa.autoscaling.knative.dev"

KPA reacts faster to bursts (sub-second). HPA is steadier and integrates with your existing custom metrics pipeline. For agent workloads I’d default to KPA: bursts of webhook traffic show up first as concurrency, which KPA scales on natively.

What you give up

Cold starts. The first request after a quiet period will pay 1 to 3 seconds for the Node process to come up. That’s fine for JIRA webhooks. It would be painful for a chatbot facing humans. If you can’t tolerate cold starts, set min-scale: 1 and pay for one always-warm pod.

Operational ownership. Cloudflare hands you “the function ran, here’s the log, here’s the trace.” With Knative you own the autoscaler tuning, the ingress, the cert rotation, the registry, and the K8s upgrade cycle. If your team already runs that, the marginal cost is small. If not, you might be reaching for the wrong tool.

Egress quotas. Knative defaults to no egress filtering. If you’re in a regulated environment, you’ll want NetworkPolicies in front of the agent’s namespace so it can only reach the model API and your JIRA host. The default is “open egress,” which is probably fine for the prototype and definitely not fine for prod.

Observability

The agent already returns structured JSON for each invocation (the action it took, the reasoning summary, the issue key). On Knative, every request shows up in kubectl logs and the queue-proxy sidecar exposes Prometheus metrics:

revision_request_count per Service revision
request_latency histogram
concurrent_requests gauge

Pair that with a ServiceMonitor and you get a dashboard for free. Add a structured logger inside the agent (pino, console.log of structured objects, whatever your cluster’s log pipeline can parse) and you’ve got correlated logs and metrics without any custom infra.

For traces: enable Knative’s OpenTelemetry support, or bolt on the OpenTelemetry SDK inside the Flue handler. Whichever your tracing stack already prefers.

When this is the right tool

Self-host the Flue agent on Knative when:

Your data plane has to stay in your VPC.
You already run Kubernetes and have an SRE team that knows it.
You have multiple agents and want a uniform deployment story across them all.
Webhook traffic is bursty enough that “always running on a VM” is overkill, but consistent enough that you’d rather not pay per-request to a third party.

Reach for managed serverless when:

You don’t have a K8s team, or you have one and they’re already at capacity.
You want one less thing to babysit.
The data is fine to leave the building.

Both shapes work. Both are real. The Flue agent code doesn’t change between them, which is the part I keep finding satisfying: you can develop on Workers, prove the agent out in the small, and self-host the same code in your cluster the day someone in security says “no, that data stays put.”

Flue, Part 2: Stop Triaging JIRA By Hand

Sat, 02 May 2026 15:00:00 -0400

Part 1 was the framework pitch. This is the agent I actually wanted to build with it.

The setup is something I keep seeing across teams: JIRA triage is brutal. Tickets pile up in the new-issues queue. An EM or tech lead spends an hour every morning sorting them. Someone classifies, someone re-classifies, someone routes, someone re-routes. Half of them get assigned to the wrong team and bounce around for two days before landing.

The agent I’m going to walk through does that triage work. It reads each new ticket, classifies it, picks a team and a person, and either assigns it or comments on the ticket asking for human eyes. It never guesses. If it isn’t confident, the human gets a structured handoff instead of a wrong assignee.

The plan

Two skills:

classify-ticket: reads the ticket and picks a category, domain, and priority. Self-reports confidence.
route-ticket: given that classification, picks a team and a specific assignee. Also self-reports confidence.

Then a tiny piece of glue:

If both skills come back high-confidence, assign + label, log, done.
Otherwise, comment on the ticket with the agent’s reasoning, label needs-triage, leave the assignee blank.

The whole thing runs on a JIRA webhook. New issue created? Webhook fires, Flue runs, two API calls back to JIRA, response sent. Total wall-clock time: a few seconds.

The agent

import type { FlueContext } from '@flue/sdk/client';
import * as v from 'valibot';
import * as jira from './jira';

export const triggers = { webhook: true };

const Confidence = v.picklist(['high', 'medium', 'low']);

const Classification = v.object({
 category: v.picklist([
 'bug',
 'feature-request',
 'support',
 'security',
 'infra',
 'docs',
 'duplicate',
 'spam',
 ]),
 domain: v.picklist([
 'frontend',
 'backend',
 'database',
 'auth',
 'billing',
 'reporting',
 'kubernetes',
 'unknown',
 ]),
 priority: v.picklist(['low', 'medium', 'high', 'critical']),
 reasoning: v.string(),
 confidence: Confidence,
});

const Routing = v.object({
 team: v.string(),
 assignee: v.union([v.string(), v.null()]),
 reasoning: v.string(),
 confidence: Confidence,
});

export default async function ({ init, payload, env }: FlueContext) {
 const { issue } = payload;

 const agent = await init({ model: 'anthropic/claude-opus-4-7' });
 const session = await agent.session();

 const classification = await session.skill('classify-ticket', {
 args: {
 summary: issue.fields.summary,
 description: issue.fields.description,
 reporter: issue.fields.reporter.displayName,
 labels: issue.fields.labels ?? [],
 },
 result: Classification,
 });

 const routing = await session.skill('route-ticket', {
 args: {
 classification,
 summary: issue.fields.summary,
 description: issue.fields.description,
 },
 result: Routing,
 });

 const confident =
 classification.confidence === 'high' &&
 routing.confidence === 'high' &&
 routing.assignee != null;

 if (confident) {
 await jira.assignIssue(env, issue.key, routing.assignee!);
 await jira.applyLabels(env, issue.key, [
 `category/${classification.category}`,
 `domain/${classification.domain}`,
 `priority/${classification.priority}`,
 `team/${routing.team}`,
 ]);
 return { action: 'assigned', issue: issue.key, ...routing };
 }

 return await escalate(env, issue.key, classification, routing);
}

async function escalate(
 env: any,
 key: string,
 c: v.InferOutput<typeof Classification>,
 r: v.InferOutput<typeof Routing>,
) {
 const body = [
 `Triage agent escalated this ticket for human review.`,
 ``,
 `*Classification (${c.confidence}):*`,
 `- Category: ${c.category}`,
 `- Domain: ${c.domain}`,
 `- Priority: ${c.priority}`,
 `- Reasoning: ${c.reasoning}`,
 ``,
 `*Suggested routing (${r.confidence}):*`,
 `- Team: ${r.team}`,
 `- Assignee: ${r.assignee ?? 'unsure'}`,
 `- Reasoning: ${r.reasoning}`,
 ].join('\n');

 await jira.addComment(env, key, body);
 await jira.applyLabels(env, key, ['needs-triage']);
 return { action: 'escalated', issue: key };
}

Two skill calls, one branch. The whole file fits on a single screen.

The JIRA helper is small enough to inline. Three calls: assign, comment, label.

// jira.ts
const base = (env: any) => `https://${env.JIRA_HOST}/rest/api/3`;
const headers = (env: any) => ({
 Authorization: `Basic ${btoa(`${env.JIRA_EMAIL}:${env.JIRA_TOKEN}`)}`,
 'Content-Type': 'application/json',
});

export async function assignIssue(env: any, key: string, accountId: string) {
 await fetch(`${base(env)}/issue/${key}/assignee`, {
 method: 'PUT',
 headers: headers(env),
 body: JSON.stringify({ accountId }),
 });
}

export async function addComment(env: any, key: string, text: string) {
 await fetch(`${base(env)}/issue/${key}/comment`, {
 method: 'POST',
 headers: headers(env),
 body: JSON.stringify({
 body: {
 type: 'doc',
 version: 1,
 content: [
 { type: 'paragraph', content: [{ type: 'text', text }] },
 ],
 },
 }),
 });
}

export async function applyLabels(env: any, key: string, labels: string[]) {
 await fetch(`${base(env)}/issue/${key}`, {
 method: 'PUT',
 headers: headers(env),
 body: JSON.stringify({
 update: { labels: labels.map((l) => ({ add: l })) },
 }),
 });
}

The skills

A Flue skill is a directory. It has a prompt template and zero or more context files the model reads before answering. That’s where the “advanced reasoning” actually lives, in the context the model gets to look at before it commits to a structured output.

Here is skills/classify-ticket/prompt.md:

You are a triage assistant for our engineering org. Classify the JIRA ticket
below.

Use these context files to understand the categories, domains, and our
priority rubric:

- /context/categories.md
- /context/domains.md
- /context/priority.md
- /context/historical-examples.md

Process:

1. Read the ticket and the context files.
2. Pick the single most accurate category and domain.
3. Pick a priority based on /context/priority.md, not based on how loudly
 the reporter complains.
4. Set confidence based on how clean the match is. If you have to choose
 between two domains, that is "medium" at best. If the ticket is too
 vague to map confidently, that is "low".
5. In `reasoning`, write 2 to 3 sentences explaining the call. This is
 what a human will read if you escalate.

Never guess. We would rather have a human triage an ambiguous ticket than
auto-assign a wrong one.

The ticket:
- Summary: {{ args.summary }}
- Description: {{ args.description }}
- Reporter: {{ args.reporter }}
- Labels: {{ args.labels }}

Three things matter in that prompt. The structured output schema enforces that the model picks from a finite set of categories. The “never guess” instruction backed by the confidence field gives it permission to bail out. The context files give it the org-specific knowledge that the base model can’t have.

skills/route-ticket/ is the same shape. Its own prompt, its own context:

/context/teams.md: what each team owns, with examples.
/context/people.md: who’s on each team, what they specialize in, and who’s currently OOO.
/context/ownership-by-domain.md: table of domain to primary team, with edge cases.

The OOO file is the under-rated piece. If the model’s first pick is on vacation, it should fall back to the team’s secondary, not barrel through with a stale assignment that nobody picks up for a week.

What “confident” actually means here

The confidence field isn’t magic. It’s the model self-reporting after reading the prompt and the context files. It’s wrong sometimes. Two things make it less wrong:

Two paths, one decision. I run both skills before deciding. A “high” on classification but “medium” on routing means we ask a human, even though the model thinks it has half the answer. In practice, “the routing is shaky” is the most common reason to escalate, and I want that signal preserved instead of averaged away.

No null assignees on the happy path. If the routing skill returns assignee: null (it knows the team but not the person), I treat that as not-confident even if the model self-reports “high”. The shape of the result is doing some of the gating work for me.

You could tighten this further. Numeric confidence with explicit thresholds, an explicit “needs more info” exit code from the skill, retries with deeper context. I haven’t needed those yet. The two-skill, two-confidence pattern catches the bulk of bad routings before they happen.

The escalation comment is the product

When the agent isn’t sure, the JIRA comment is where it earns its keep. Look at what it writes:

Triage agent escalated this ticket for human review.

*Classification (medium):*
- Category: support
- Domain: billing
- Priority: medium
- Reasoning: Customer reports a charge they don't recognize. Could be
 a duplicate from the recent retry storm or a real billing bug. Treating
 as support until billing confirms which one.

*Suggested routing (low):*
- Team: billing
- Assignee: unsure
- Reasoning: Billing team owns this domain, but they're rotating an
 on-call this week and I can't tell from /context/people.md who is
 actually picking things up today. Recommend an EM eyeball this and
 route.

A human reading that has all the work pre-loaded. The category guess. The domain guess. The reasoning. The reason it stopped short. They’re not triaging from scratch. They’re auditing a draft.

That’s the actual UX win. Even when the agent fails, it fails in a way that saves the human time.

Deploying it

Flue runs in a few places. For something this small I’d put it on Cloudflare Workers. The request flow is “JIRA webhook, 30 seconds of model calls, JIRA REST API.” No long-running compute, no filesystem, just a function that eats a webhook.

Point JIRA’s automation rule at your worker URL, configure secrets in Cloudflare for the JIRA token and the Anthropic key, push with wrangler deploy. The whole thing has fewer moving parts than the legacy “triage Slack channel” my team used to run.

Where this stops working

It is not a replacement for an EM. It is not a replacement for a tech lead with context. The agent is good at the volume case: routine tickets, clear ownership, well-documented domains. It will struggle at the long tail:

Cross-team tickets that legitimately span two domains.
Tickets where the customer is using the wrong vocabulary for the system.
Anything political. Reorgs, ownership disputes, “you broke our launch.”

The escalation path is what catches those. As long as the agent is willing to stop and ask, the long tail goes to a human anyway. The agent’s job isn’t to be right about everything. It’s to clear the boring 70% so humans can focus on the interesting 30%.

That’s the same pattern I keep finding for these harness-based agents in general. They don’t replace the senior person. They eat the toil that the senior person hates doing, and they hand back the genuinely hard cases with a useful first draft. On net, an excellent trade.

Flue vs Mastra: TypeScript Agents in 2026

Sat, 02 May 2026 09:00:00 -0400

I’ve been on a kick lately, building small agents in TypeScript instead of Python. Two frameworks keep showing up in my browser tabs: Flue and Mastra. They’re both betting on the same thing: TypeScript is the right language to ship AI in. They just disagree about almost everything else.

That disagreement is the interesting part.

The harness vs the workflow

Flue’s pitch is one line: Agent = Model + Harness. The framework gives you a programmable harness. Sandbox, filesystem, bash, sessions, skills with structured output. You drop in a model. You get something that looks a lot like Claude Code or Codex, except you wrote it and you deploy it.

Here’s a Flue agent that triages a GitHub issue:

import type { FlueContext } from '@flue/sdk/client';
import * as v from 'valibot';

export default async function ({ init, payload }: FlueContext) {
 const agent = await init({ model: 'anthropic/claude-opus-4-7' });
 const session = await agent.session();

 const triage = await session.skill('triage', {
 args: { issueNumber: payload.issueNumber },
 result: v.object({
 severity: v.picklist(['low', 'medium', 'high', 'critical']),
 reproducible: v.boolean(),
 summary: v.string(),
 }),
 });

 return triage;
}

The whole shape is: init the agent, open a session, call a skill, get a typed result back. The skill is a directory of prompts and tools that the harness orchestrates. Your code never has to babysit a tool-call loop. The framework is doing it.

Mastra goes the other way. The core abstractions are agents, workflows, and RAG. You compose them, evaluate them, and deploy them as APIs next to your Next.js or Hono app.

A roughly equivalent Mastra agent:

import { Agent } from '@mastra/core';
import { z } from 'zod';

export const triageAgent = new Agent({
 name: 'issue-triage',
 instructions: 'You triage GitHub issues. Be concise.',
 model: { provider: 'ANTHROPIC', name: 'claude-opus-4-7' },
});

export async function triage(issueNumber: number) {
 const result = await triageAgent.generate(
 `Triage issue #${issueNumber}`,
 {
 output: z.object({
 severity: z.enum(['low', 'medium', 'high', 'critical']),
 reproducible: z.boolean(),
 summary: z.string(),
 }),
 },
 );

 return result.object;
}

Different shape entirely. You declare the agent up front, you call .generate() with a structured output schema, you get an object back. There’s no harness loop driving sessions and skills. There’s just a model with tools, and you wire it into your app like any other service.

Where they diverge

	Flue	Mastra
Mental model	Agent = Model + Harness	Agent = LLM with tools
Primitives	Sessions, skills, sandboxes	Agents, workflows, RAG
What you bring	Skill prompts and tools	Instructions, tools, eval suites
Deploy targets	Node, Cloudflare Workers, GitHub Actions	Anywhere TypeScript runs, plus Mastra Cloud
Replaces	Dosu, Greptile, CodeRabbit	Honestly, half of LangChain
Tagline	“Stop renting someone else’s agent”	“Python trains, TypeScript ships”

If you’re building something that acts (writes files, runs commands, reads a repo, works through a task across many tool calls), Flue’s harness model is doing more for you. If you’re embedding an agent into an existing app and the bigger problem is glue, eval, and observability, Mastra has more of the production scaffolding ready to go.

I don’t think one is winning yet. They might both be right for different jobs.

The thing nobody’s saying out loud

Both of these projects assume something I would have laughed at five years ago: that TypeScript is a serious language for building AI infrastructure. Not a frontend duct tape. Not a glue layer. The actual substrate.

A couple weeks back, Microsoft shipped the TypeScript 7.0 beta, and the punchline is that the compiler is no longer written in TypeScript. It’s written in Go. Native code, shared memory parallelism, the whole thing. They’re claiming about 10x faster than 6.0 on real codebases. The language server, too. The editing experience is reportedly night and day.

Read that twice. The TypeScript team looked at the language they invented, decided it wasn’t fast enough to compile itself, and rewrote the toolchain in Go. That’s a remarkable thing to do.

It also reframes everything. If tsc is fast enough that incremental compiles feel like Go’s build times, the cost-of-iteration argument against TypeScript for serious systems work mostly evaporates. The other arguments (no real concurrency, GC pauses, V8 quirks) are runtime arguments, and most agent code is I/O bound on a model API anyway.

So you get a typed, npm-friendly, JIT’d runtime with a now-fast toolchain, and the LLM ecosystem has decided to publish first-class SDKs for it. That’s a pretty good place for a framework to land.

Flue and Mastra are both betting on it. They might both be right.

What I’d actually pick

For a coding agent, an issue triage bot, anything that needs to do work in a sandbox: Flue. The harness model is the right level of abstraction for that kind of work, and “drop your code in a Cloudflare Worker, point a webhook at it” is a stupidly nice deployment story.

For a chat-shaped assistant living inside a real product, with eval suites and traces and the whole observability story: Mastra. The framework gets out of your way and the tooling around it is more mature.

For everything else: keep an eye on both. The TypeScript-as-AI-substrate bet is going to play out fast, and a 10x faster compiler landing the same year is not a coincidence I would have predicted.

SFTPGo on Kubernetes: Azure Blob Without the $219/Month SFTP Tax

Sat, 11 Apr 2026 00:00:00 +0000

Someone needed SFTP access to Azure Blob Storage. Simple enough, right? Azure even has native SFTP support built into storage accounts now. Problem solved.

Until you look at the price tag.

The Azure SFTP Tax

Azure’s native SFTP support for Blob Storage costs $0.30 per hour. That’s roughly $219 per month. Just for the endpoint. Just for it to exist. You haven’t transferred a single file yet. That’s before storage costs, before transaction costs, before anything useful happens.

Oh, and it requires hierarchical namespace (Azure Data Lake Storage Gen2), which means you can’t just flip it on for an existing standard Blob Storage account. You need ADLS Gen2 from the start, or you’re creating a new storage account.

For a high-throughput enterprise workload moving terabytes a day, $219/month is a rounding error. For the rest of us who just need to let a vendor or partner drop some files into a blob container? That’s an absurd premium for what amounts to a protocol adapter.

SFTPGo: The Alternative

SFTPGo is an open-source, fully featured SFTP server written in Go. It supports virtual folders backed by local filesystem, S3, Google Cloud Storage, and Azure Blob Storage. It handles SSH keys, password auth, per-user quotas, bandwidth limits, and a web admin UI. It’s also a single binary that runs great in a container.

The plan: run SFTPGo on Kubernetes, point virtual folders at Azure Blob Storage, and skip the $219/month endpoint tax entirely.

The Kubernetes Setup

Here’s what the deployment looks like. Nothing exotic. A Deployment, a couple of Services, a ConfigMap, some Secrets, and a PostgreSQL database for SFTPGo’s user/config store.

Namespace

Keep it tidy:

apiVersion: v1
kind: Namespace
metadata:
 name: sftpgo

The Secrets

Three secrets. Azure Blob credentials, PostgreSQL connection, and SSH host keys.

apiVersion: v1
kind: Secret
metadata:
 name: sftpgo-azure
 namespace: sftpgo
type: Opaque
stringData:
 SFTPGO_AZBLOB_CONTAINER: "your-container-name"
 SFTPGO_AZBLOB_ACCOUNT_NAME: "yourstorageaccount"
 SFTPGO_AZBLOB_ACCOUNT_KEY: "your-storage-account-key"
---
apiVersion: v1
kind: Secret
metadata:
 name: sftpgo-db
 namespace: sftpgo
type: Opaque
stringData:
 SFTPGO_DATA_PROVIDER__DRIVER: "postgresql"
 SFTPGO_DATA_PROVIDER__NAME: "postgresql://sftpgo:your-db-password@sftpgo-db:5432/sftpgo?sslmode=disable"

If you’re using Managed Identity for the Azure side (and you should be if you can), you can skip the account key and rely on the pod’s identity. But that’s a topic for another post.

SFTPGo reads its data provider config from environment variables using the SFTPGO_DATA_PROVIDER__ prefix. The double underscore maps to the nested JSON structure. Neat trick that saves you from templating JSON.

The host keys get their own secret. Generate them once, store them in the cluster, and every pod (current and future replicas) uses the same keys. If you skip this step and let SFTPGo generate keys on startup, every pod restart hands clients a different fingerprint. That WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! message trains users to blindly accept key changes, which is the opposite of what SSH security is for.

# Generate the keys locally
ssh-keygen -t ed25519 -f id_ed25519 -N ""
ssh-keygen -t rsa -b 4096 -f id_rsa -N ""

# Create the secret
kubectl create secret generic sftpgo-hostkeys -n sftpgo \
 --from-file=id_ed25519 \
 --from-file=id_rsa

# Clean up local copies
rm id_ed25519 id_ed25519.pub id_rsa id_rsa.pub

The ConfigMap

SFTPGo has a mountain of configuration options. The data provider config is handled by the env vars in the secret above, so the ConfigMap just needs the SFTP, HTTP, and telemetry bindings:

apiVersion: v1
kind: ConfigMap
metadata:
 name: sftpgo-config
 namespace: sftpgo
data:
 sftpgo.json: |
 {
 "sftpd": {
 "bindings": [{ "port": 2022, "address": "" }],
 "host_keys": [
 "/var/lib/sftpgo/id_ed25519",
 "/var/lib/sftpgo/id_rsa"
 ]
 },
 "httpd": {
 "bindings": [{
 "port": 8080,
 "address": "",
 "enable_web_admin": true,
 "enable_web_client": true
 }]
 },
 "telemetry": {
 "bind_port": 10000,
 "bind_address": "",
 "enable_profiler": false,
 "auth_user_file": ""
 }
 }

Port 2022 for SFTP, port 8080 for the web admin, port 10000 for the Prometheus metrics endpoint. PostgreSQL connection details come from the sftpgo-db secret, so they stay out of the ConfigMap where they belong.

PostgreSQL

SFTPGo needs a database for users, folders, and configuration. PostgreSQL is the right choice here. It handles concurrent access without drama, and if you ever want to scale SFTPGo beyond a single replica, you’ll need it.

A simple StatefulSet gets the job done:

apiVersion: v1
kind: Secret
metadata:
 name: sftpgo-pg
 namespace: sftpgo
type: Opaque
stringData:
 POSTGRES_USER: "sftpgo"
 POSTGRES_PASSWORD: "your-db-password"
 POSTGRES_DB: "sftpgo"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: sftpgo-db
 namespace: sftpgo
spec:
 serviceName: sftpgo-db
 replicas: 1
 selector:
 matchLabels:
 app: sftpgo-db
 template:
 metadata:
 labels:
 app: sftpgo-db
 spec:
 containers:
 - name: postgres
 image: postgres:16-alpine
 ports:
 - containerPort: 5432
 envFrom:
 - secretRef:
 name: sftpgo-pg
 volumeMounts:
 - name: pgdata
 mountPath: /var/lib/postgresql/data
 resources:
 requests:
 cpu: 100m
 memory: 256Mi
 limits:
 cpu: 500m
 memory: 512Mi
 volumeClaimTemplates:
 - metadata:
 name: pgdata
 spec:
 accessModes: ["ReadWriteOnce"]
 resources:
 requests:
 storage: 5Gi
---
apiVersion: v1
kind: Service
metadata:
 name: sftpgo-db
 namespace: sftpgo
spec:
 clusterIP: None
 selector:
 app: sftpgo-db
 ports:
 - port: 5432
 targetPort: 5432

A headless Service gives SFTPGo a stable sftpgo-db:5432 endpoint to connect to. The StatefulSet’s volumeClaimTemplates handle persistent storage so your data survives pod restarts.

If you’re already running a managed PostgreSQL instance (Azure Database for PostgreSQL, RDS, etc.), just point the connection string at that and skip this entirely.

The Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
 name: sftpgo
 namespace: sftpgo
 labels:
 app: sftpgo
spec:
 replicas: 1
 selector:
 matchLabels:
 app: sftpgo
 template:
 metadata:
 labels:
 app: sftpgo
 spec:
 containers:
 - name: sftpgo
 image: drakkan/sftpgo:latest
 ports:
 - containerPort: 2022
 name: sftp
 - containerPort: 8080
 name: http
 - containerPort: 10000
 name: metrics
 envFrom:
 - secretRef:
 name: sftpgo-azure
 - secretRef:
 name: sftpgo-db
 volumeMounts:
 - name: config
 mountPath: /etc/sftpgo/sftpgo.json
 subPath: sftpgo.json
 - name: hostkeys
 mountPath: /var/lib/sftpgo/id_ed25519
 subPath: id_ed25519
 readOnly: true
 - name: hostkeys
 mountPath: /var/lib/sftpgo/id_rsa
 subPath: id_rsa
 readOnly: true
 resources:
 requests:
 cpu: 100m
 memory: 128Mi
 limits:
 cpu: 500m
 memory: 256Mi
 volumes:
 - name: config
 configMap:
 name: sftpgo-config
 - name: hostkeys
 secret:
 secretName: sftpgo-hostkeys
 defaultMode: 0600

No PVC needed. Config comes from a ConfigMap, credentials from Secrets, host keys from a Secret, and user data lives in PostgreSQL. The Deployment is completely stateless, which means Kubernetes can reschedule pods freely and scaling replicas is trivial (more on that below).

The Service

The Service is internal only. ClusterIP, not LoadBalancer. We don’t want the admin UI anywhere near the public internet.

apiVersion: v1
kind: Service
metadata:
 name: sftpgo
 namespace: sftpgo
spec:
 type: ClusterIP
 selector:
 app: sftpgo
 ports:
 - name: sftp
 port: 2022
 targetPort: 2022
 protocol: TCP
 - name: http
 port: 8080
 targetPort: 8080
 protocol: TCP
 - name: metrics
 port: 10000
 targetPort: 10000
 protocol: TCP

To access the admin UI, use kubectl port-forward:

kubectl port-forward -n sftpgo svc/sftpgo 8080:8080

Then open http://localhost:8080/web/admin in your browser. This keeps the admin UI off the network entirely. No Ingress, no TLS cert to manage, no authentication proxy to configure. Just a direct tunnel from your machine to the pod when you need it.

The Ingress

SFTP needs to be reachable from the outside, so we expose only the SFTP port. Most ingress controllers support TCP/UDP proxying alongside HTTP. With nginx-ingress, you configure it through a TCP services ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
 name: tcp-services
 namespace: ingress-nginx
data:
 "2222": "sftpgo/sftpgo:2022"

This tells the ingress controller to listen on port 2222 externally and forward TCP traffic to sftpgo:2022 in the sftpgo namespace. Make sure your ingress controller’s Service and Deployment are configured to expose this port. You’ll need to add it to the controller’s --tcp-services-configmap arg and open the port on the controller’s Service:

# Patch the ingress-nginx controller Service to expose port 2222
# (or add it to your existing Service definition)
spec:
 ports:
 - name: sftp-proxy
 port: 2222
 targetPort: 2222
 protocol: TCP

Your clients connect with sftp -P 2222 user@your-ingress-ip. The admin UI stays completely internal. The only thing exposed to the internet is the SFTP port, which is exactly the attack surface you want: SSH protocol with key-based auth. Not a web UI with a login form.

Configuring the Azure Blob Virtual Folder

Once everything is running, port-forward into the admin UI and create a user:

kubectl port-forward -n sftpgo svc/sftpgo 8080:8080

Open http://localhost:8080/web/admin. The default admin credentials are in the SFTPGo docs (change them immediately, obviously).

The fun part: virtual folders. When you create or edit a user, you can map a virtual path to an Azure Blob container. In the SFTPGo admin:

Go to your user’s settings
Under Virtual Folders, add a new folder
Set the mapped path (e.g., /uploads)
Choose Azure Blob Storage as the filesystem
Fill in the container name and credentials (or let the env vars handle it)

Now when a client connects via SFTP and cd /uploads, they’re reading and writing directly to Azure Blob Storage. The client has no idea. They just see files and directories.

You can also do this via the REST API if you’re automating user provisioning:

curl -X POST http://localhost:8080/api/v2/users \
 -H "Content-Type: application/json" \
 -u admin:password \
 -d '{
 "username": "vendor-uploads",
 "password": "a-strong-password-please",
 "permissions": { "/": ["*"] },
 "virtual_folders": [
 {
 "name": "azure-uploads",
 "mapped_path": "/uploads",
 "virtual_path": "/uploads",
 "filesystem": {
 "provider": 3,
 "azblobconfig": {
 "container": "vendor-data",
 "account_name": "yourstorageaccount",
 "account_key": { "payload": "base64-key-here" }
 }
 }
 }
 ]
 }'

Provider 3 is Azure Blob Storage in SFTPGo’s config. Provider 1 is S3, 2 is Google Cloud Storage, 0 is local. The kind of magic number situation that makes you love (and occasionally curse) open source software.

What You Get

For the cost of a small pod running on your existing cluster, you now have:

SFTP access to Azure Blob Storage without the $219/month Azure tax
Per-user access control with SSH keys or passwords
Virtual folder mappings so different users see different containers or prefixes
Bandwidth throttling and quotas per user
Audit logging of every connection and file operation
A web UI for managing users via kubectl port-forward

Scaling Up: Multiple Replicas

A single SFTPGo pod works fine until it doesn’t. Maybe you need high availability. Maybe you have enough concurrent connections that one pod is sweating. Maybe you just don’t want a single point of failure sitting between your vendors and their file drops.

The good news: because we set this up with PostgreSQL for state and Secrets for host keys, the Deployment is already stateless. Scaling is just:

spec:
 replicas: 3

That’s it. No migration, no rearchitecture. Every replica mounts the same host keys from the sftpgo-hostkeys Secret, so clients get a consistent SSH fingerprint regardless of which pod they land on. User data, virtual folder configs, and quota tracking all live in PostgreSQL, so every replica sees the same state.

SFTPGo also supports a shared event system through the data provider. When one instance updates a user, other instances pick up the change. No manual cache invalidation, no restart required.

Load Distribution

SFTP is a TCP protocol, so Kubernetes’ default round-robin Service routing handles connection distribution automatically. Each new SFTP connection lands on a different pod. Connections are sticky for their duration (TCP, after all), so a long-running transfer won’t bounce between pods mid-stream.

If you’re using the nginx-ingress TCP proxy from earlier, the same applies there. Each new inbound connection on port 2222 gets routed to the ClusterIP Service, which distributes across replicas.

One thing to note: SFTP connections are long-lived. CPU and memory might look low even under heavy transfer load because the bottleneck is usually network I/O, not compute. If you want autoscaling, you’ll need custom metrics rather than CPU. The monitoring section below covers exactly how to set that up.

Monitoring with Prometheus and Grafana

Running an SFTP server without monitoring is like deploying to production without logs. You technically can, but the first time a vendor calls asking why their upload failed three hours ago, you’ll wish you hadn’t.

SFTPGo has a built-in Prometheus metrics endpoint. We already enabled it in the config on port 10000. Hit /metrics on that port and you get the standard Prometheus exposition format with everything you’d want to know: active connections, bytes transferred, upload/download counts, error rates, and data provider health.

ServiceMonitor

If you’re running the Prometheus Operator (and if you’re on Kubernetes with Prometheus, you probably are), a ServiceMonitor makes scrape configuration automatic:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 name: sftpgo
 namespace: sftpgo
 labels:
 app: sftpgo
spec:
 selector:
 matchLabels:
 app: sftpgo
 endpoints:
 - port: metrics
 interval: 30s
 path: /metrics

That’s the whole thing. The Prometheus Operator picks this up, finds the sftpgo Service’s metrics port, and starts scraping. No need to edit Prometheus config files or restart anything.

If you’re running vanilla Prometheus without the Operator, add a scrape job to your prometheus.yml:

scrape_configs:
 - job_name: "sftpgo"
 kubernetes_sd_configs:
 - role: endpoints
 namespaces:
 names:
 - sftpgo
 relabel_configs:
 - source_labels: [__meta_kubernetes_endpoint_port_name]
 action: keep
 regex: metrics

What Gets Exposed

SFTPGo’s /metrics endpoint exports a solid set of counters and gauges. Some of the useful ones:

sftpgo_active_connections - current active SFTP connections per pod
sftpgo_active_transfers - in-progress file transfers
sftpgo_bytes_sent_total / sftpgo_bytes_received_total - cumulative transfer volume
sftpgo_uploads_total / sftpgo_downloads_total - file operation counts
sftpgo_uploads_errors_total / sftpgo_downloads_errors_total - failed transfers
sftpgo_data_provider_availability - whether the PostgreSQL connection is healthy

That last one is important. If the data provider goes down, SFTPGo can’t authenticate users. You want to know about that before your vendors do.

Grafana Dashboard

SFTPGo ships a pre-built Grafana dashboard that you can import directly. Grab it from the SFTPGo repo or import dashboard ID 16498 from Grafana’s dashboard marketplace.

If you’d rather build your own, here are a few PromQL queries worth pinning:

Active connections across all replicas:

sum(sftpgo_active_connections)

Transfer throughput (bytes/sec, 5-minute rate):

sum(rate(sftpgo_bytes_sent_total[5m]) + rate(sftpgo_bytes_received_total[5m]))

Upload error rate as a percentage:

sum(rate(sftpgo_uploads_errors_total[5m]))
 / sum(rate(sftpgo_uploads_total[5m])) * 100

Data provider health (alert if any pod loses DB connectivity):

min(sftpgo_data_provider_availability) == 0

That last query is a good candidate for a Prometheus alert rule. If any replica can’t reach PostgreSQL, fire an alert. The min catches the case where one pod is healthy but another has lost its connection.

Tying It Back to Autoscaling

Remember the HPA note from the replicas section? sftpgo_active_connections is your custom metric for scaling. With the Prometheus Adapter, you can scale SFTPGo pods based on actual connection count instead of CPU:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
 name: sftpgo
 namespace: sftpgo
spec:
 scaleTargetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: sftpgo
 minReplicas: 2
 maxReplicas: 10
 metrics:
 - type: Pods
 pods:
 metric:
 name: sftpgo_active_connections
 target:
 type: AverageValue
 averageValue: "50"

When the average connection count per pod crosses 50, Kubernetes spins up another replica. When it drops, it scales back down. Way better than guessing based on CPU, which barely moves during SFTP transfers.

Central Logging

Metrics tell you what’s happening right now. Logs tell you what happened at 2am when that vendor’s automated upload script decided to authenticate 400 times in a row with the wrong password.

SFTPGo writes structured JSON logs to stdout by default, which is exactly what you want on Kubernetes. No sidecar containers, no log file rotation, no volume mounts for log directories. Your cluster’s log collector (Fluentd, Fluent Bit, Promtail, Vector, whatever you’re running) picks them up automatically from the container runtime.

A typical SFTPGo log line looks like this:

{
 "level": "info",
 "time": "2026-04-11T14:32:01.123Z",
 "sender": "sftpd",
 "message": "User logged in",
 "username": "vendor-uploads",
 "remote_address": "10.244.3.15:48212",
 "protocol": "SFTP",
 "connection_id": "abc123"
}

Every log entry includes the username, remote address, protocol, and connection ID. File operations include the file path and transfer size. Auth failures include the attempted username and the reason. All structured, all parseable, all ready to ship to wherever your logs land.

Configuring Log Format

SFTPGo defaults to JSON on stdout, but you can control the verbosity. Add a logger block to the ConfigMap if you want to tune it:

{
 "logger": {
 "enabled": true,
 "level": "info",
 "utc_time": true
 }
}

Keep it at info for production. debug is useful when you’re troubleshooting auth issues or virtual folder mappings, but it’s chatty. It logs every single SSH handshake step, every directory listing, every stat call. On a busy server, that’s a lot of log volume.

What to Query For

Once your logs are in Loki, Elasticsearch, or whatever your stack uses, these are the queries worth saving:

Failed authentication attempts are the first thing you want to see. Brute force attempts, expired credentials, misconfigured clients. In Loki with LogQL:

{namespace="sftpgo"} | json | level="error" | message=~".*authentication.*"

File transfer activity by user gives you an audit trail. Who uploaded what, when, and how big:

{namespace="sftpgo"} | json | message="Upload" | line_format "{{.username}} {{.virtual_path}} {{.elapsed_ms}}ms"

Connection patterns help you spot anomalies. A user that normally connects once a day suddenly hammering the server every 30 seconds is worth investigating:

sum by (username) (count_over_time({namespace="sftpgo"} | json | message="User logged in" [1h]))

Audit Trail

For compliance-heavy environments (and if you’re handling file transfers for enterprise partners, you’re probably in one), SFTPGo’s structured logs give you a complete audit trail out of the box. Every login, every file operation, every disconnect. Combined with the username and remote IP in each log entry, you can reconstruct exactly who did what and when.

Pipe these into a long-retention store (separate from your operational logs) and you’ve got audit coverage without bolting on a separate audit system. Set a retention policy that matches your compliance requirements and forget about it.

Things to Watch Out For

Keep the admin UI internal. The admin UI has full control over user accounts. Don’t Ingress it. Don’t LoadBalancer it. kubectl port-forward when you need it, close it when you don’t. If you absolutely need remote access, put it behind an auth proxy and a VPN. Not just one of those. Both.

PostgreSQL backups. You’re running a real database now, which means you need real backups. pg_dump on a cron, or use your managed PostgreSQL provider’s automated backups. The SFTPGo user database isn’t big, but losing it means recreating every user and virtual folder mapping from scratch.

Azure Blob isn’t a filesystem. Listing large directories can be slow because blobs use a flat namespace with prefix-based “directory” simulation. If your users are running ls on a container with 500,000 objects, they’re going to have a bad time.

The Math

	Azure Native SFTP	SFTPGo on K8s
Monthly endpoint cost	~$219	$0 (runs on existing cluster)
Requires ADLS Gen2	Yes	No
Multi-user support	Local users only	Full user management
Quotas/throttling	No	Yes
Audit logging	Azure Monitor	Built-in + syslog
Prometheus metrics	No	Native /metrics endpoint
Admin UI exposure	N/A	Internal only (port-forward)
Setup complexity	Toggle in portal	Deploy to K8s

The “setup complexity” column is doing some heavy lifting for Azure there. Toggling a switch is simpler. But $219/month simpler? For most workloads, no.

Wrapping Up

SFTPGo fills a gap that shouldn’t exist. SFTP is a 20+ year old protocol. Mapping it to object storage shouldn’t cost $2,600/year. Running your own is a few YAML files and a container image. The SFTPGo project is actively maintained, well-documented, and handles edge cases I haven’t even thought about yet.

If you’re already running Kubernetes, this is maybe 20 minutes of work. The hardest part is explaining to your finance team why you’re not using the Azure-native option.

Kubernetes for Cloud Engineers: Automating TLS with cert-manager

Sat, 04 Apr 2026 00:00:00 +0000

This is part two of a series for new operations and cloud engineers getting into Kubernetes. In part one we covered how to set default TLS certificates on your ingress controllers. That works, but it means you’re managing certs by hand. Let’s fix that.

Why cert-manager?

In the last post, we created TLS secrets manually. That’s fine for getting started. It’s not fine for production. Certificates expire. People forget to renew them. And then you’re getting paged at 3am because your site is serving a browser warning.

cert-manager is a Kubernetes-native certificate management controller. It watches your cluster for resources that need certificates, talks to a Certificate Authority (usually Let’s Encrypt), handles the challenge/response dance, stores the resulting cert as a Kubernetes secret, and renews it before it expires. All automatically. You configure it once and move on with your life.

It’s one of those tools that, once installed, you forget it’s there. That’s the highest compliment I can give infrastructure software.

Installing cert-manager

Two options. Pick whichever matches your deployment style.

Option 1: Helm (recommended)

helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
 --version v1.20.0 \
 --namespace cert-manager \
 --create-namespace \
 --set crds.enabled=true

# NAME: cert-manager
# NAMESPACE: cert-manager
# STATUS: deployed

Option 2: Static manifests

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.20.0/cert-manager.yaml

Both methods install the same thing: the cert-manager controller, webhook, cainjector, and six CRDs. Those CRDs are the building blocks you’ll use for everything else.

# Verify everything is running
kubectl get pods -n cert-manager

# NAME READY STATUS
# cert-manager-5c4b5f7b9-xk2lq 1/1 Running
# cert-manager-cainjector-7f694c-m8p 1/1 Running
# cert-manager-webhook-7cd8c8-9tn2f 1/1 Running

Three pods running. That’s what you want to see.

Issuers: telling cert-manager where to get certificates

cert-manager doesn’t know anything about Let’s Encrypt out of the box. You need to tell it where to go and how to prove you own your domains. That’s what Issuers do.

There are two flavors:

Issuer is namespace-scoped. It can only issue certs for resources in the same namespace.
ClusterIssuer is cluster-scoped. It works across all namespaces.

For most setups, you want a ClusterIssuer. One config, whole cluster. If you have teams that need isolated certificate management per namespace, use an Issuer. But start simple.

Staging first. Always.

Let’s Encrypt has two environments: staging and production. They work identically, but staging issues certificates signed by a fake CA that browsers don’t trust. The certificates themselves are structurally real. They just won’t show a green lock.

Why does this matter? Because Let’s Encrypt production has rate limits. Tight ones. 50 certificates per registered domain per week. 5 duplicate certificates per week. If you mess up your config and keep retrying, you can lock yourself out for days.

Staging has much more generous limits. So you should always get your pipeline working against staging first, then switch to production once you know everything is wired up correctly.

Create a staging ClusterIssuer

# staging-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-staging
spec:
 acme:
 server: https://acme-staging-v02.api.letsencrypt.org/directory
 email: you@example.com
 privateKeySecretRef:
 name: letsencrypt-staging-account-key
 solvers:
 - http01:
 ingress:
 ingressClassName: nginx

kubectl apply -f staging-issuer.yaml
# clusterissuer.cert-manager.io/letsencrypt-staging created

# Check that it registered successfully
kubectl get clusterissuer letsencrypt-staging

# NAME READY AGE
# letsencrypt-staging True 30s

READY: True means cert-manager registered an account with Let’s Encrypt staging. If you see False, run kubectl describe clusterissuer letsencrypt-staging and read the events.

Create the production ClusterIssuer

Same thing, different ACME server URL:

# production-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-prod
spec:
 acme:
 server: https://acme-v02.api.letsencrypt.org/directory
 email: you@example.com
 privateKeySecretRef:
 name: letsencrypt-prod-account-key
 solvers:
 - http01:
 ingress:
 ingressClassName: nginx

Use separate privateKeySecretRef names for staging and production. The ACME accounts are completely independent. You can’t reuse keys between environments.

HTTP-01 vs DNS-01: how you prove domain ownership

When you request a certificate, Let’s Encrypt needs proof that you actually own the domain. There are two ways to do this, and which one you pick depends on your setup.

HTTP-01

Let’s Encrypt sends an HTTP request to http://yourdomain.com/.well-known/acme-challenge/<token>. cert-manager spins up a temporary pod and Ingress (or HTTPRoute) to serve the response. Once validated, the temp resources get cleaned up.

This is the simpler option. It works if:

Port 80 is publicly reachable on your cluster
You don’t need wildcard certificates

It does not work if:

You’re behind a firewall with no public HTTP access
You need *.example.com certs

DNS-01

Instead of an HTTP request, Let’s Encrypt checks for a specific TXT record at _acme-challenge.yourdomain.com. cert-manager talks to your DNS provider’s API to create and clean up the record.

This is the only way to get wildcard certificates. It’s also the right choice for internal domains that aren’t publicly accessible. The tradeoff is more setup. You need to give cert-manager API credentials for your DNS provider, and DNS propagation delays can slow things down.

Built-in DNS providers: Cloudflare, Route53, Google Cloud DNS, Azure DNS, DigitalOcean, Akamai, ACMEDNS, and RFC-2136. There are webhook extensions for 30+ more.

Wiring it up with Ingress

If you’re using the traditional Ingress API, cert-manager makes this really clean. Add an annotation to your Ingress resource, include a tls block, and cert-manager handles the rest.

# my-app-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: my-app
 annotations:
 cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
 ingressClassName: nginx
 rules:
 - host: app.example.com
 http:
 paths:
 - pathType: Prefix
 path: /
 backend:
 service:
 name: my-app
 port:
 number: 80
 tls:
 - hosts:
 - app.example.com
 secretName: app-example-com-tls

That’s it. cert-manager sees the annotation, reads the tls block, creates a Certificate resource, kicks off the ACME challenge, and stores the signed certificate in the app-example-com-tls secret. Your ingress controller picks up the secret and starts serving HTTPS.

When the cert is 30 days from expiration, cert-manager renews it automatically.

Staging first, remember. Point the annotation at letsencrypt-staging until you see a valid (but untrusted) cert in your browser. Then swap it to letsencrypt-prod. You’ll need to delete the old secret so cert-manager issues a fresh one from production.

Use cert-manager.io/cluster-issuer for ClusterIssuers and cert-manager.io/issuer for namespace-scoped Issuers. Mix them up and nothing will happen. No error, no cert. Just silence. Ask me how I know.

Wiring it up with Gateway API

If you followed the first post in this series, you know Gateway API is where things are heading. cert-manager supports it, but it needs a little extra configuration.

Enable Gateway API support

Gateway API support is not on by default. You need to opt in.

helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
 --namespace cert-manager \
 --set crds.enabled=true \
 --set config.enableGatewayAPI=true

Make sure the Gateway API CRDs are installed in your cluster too:

kubectl apply --server-side \
 -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/standard-install.yaml

If you installed the CRDs after cert-manager was already running, restart it so it picks them up:

kubectl rollout restart deployment cert-manager -n cert-manager

Annotate your Gateway

The pattern is similar to Ingress. Add the annotation, reference a secret in the listener’s certificateRefs, and cert-manager fills in the rest.

# gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: my-gateway
 annotations:
 cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
 gatewayClassName: nginx
 listeners:
 - name: https
 hostname: app.example.com
 port: 443
 protocol: HTTPS
 tls:
 mode: Terminate
 certificateRefs:
 - name: app-example-com-tls

cert-manager sees the annotation and the listener config, creates a Certificate for app.example.com, and stores the result in the app-example-com-tls secret. The Gateway picks it up and starts terminating TLS.

Three things must be true for cert-manager to act on a Gateway listener:

The hostname is not empty
The TLS mode is Terminate (not Passthrough)
There’s at least one entry in certificateRefs

Miss any of those and cert-manager quietly ignores the listener.

Wildcard certificates with DNS-01

This is where DNS-01 earns its keep. Let’s say you want a single cert that covers *.example.com so every subdomain is automatically secured. HTTP-01 can’t do this. Only DNS-01 can.

I’ll use Cloudflare as the DNS provider since it’s common (and because I have a love/hate relationship with them that we’ve already discussed on the software page).

Step 1: Create a Cloudflare API token

In the Cloudflare dashboard, create an API token with these permissions:

Zone / DNS / Edit
Zone / Zone / Read

Scope it to the zone you need. Don’t use a global API key if you can avoid it.

Step 2: Store the token in your cluster

# The secret MUST be in the cert-manager namespace for ClusterIssuers
apiVersion: v1
kind: Secret
metadata:
 name: cloudflare-api-token
 namespace: cert-manager
type: Opaque
stringData:
 api-token: your-cloudflare-api-token-here

That namespace bit is important. When you use a ClusterIssuer, cert-manager looks for referenced secrets in the namespace where cert-manager itself is installed. Not the namespace of your Certificate. Not the namespace of your app. The cert-manager namespace. This trips up everyone at least once.

Step 3: Create a DNS-01 ClusterIssuer

# dns-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-prod-dns
spec:
 acme:
 server: https://acme-v02.api.letsencrypt.org/directory
 email: you@example.com
 privateKeySecretRef:
 name: letsencrypt-prod-dns-account-key
 solvers:
 - dns01:
 cloudflare:
 apiTokenSecretRef:
 name: cloudflare-api-token
 key: api-token

Step 4: Request the wildcard cert

# wildcard-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: wildcard-example-com
 namespace: default
spec:
 secretName: wildcard-example-com-tls
 issuerRef:
 name: letsencrypt-prod-dns
 kind: ClusterIssuer
 dnsNames:
 - "*.example.com"
 - example.com

Include both *.example.com and example.com in the dnsNames. The wildcard only covers subdomains, not the bare domain itself.

kubectl apply -f wildcard-cert.yaml
# certificate.cert-manager.io/wildcard-example-com created

# Watch it work
kubectl get certificate wildcard-example-com -w

# NAME READY SECRET AGE
# wildcard-example-com False wildcard-example-com-tls 5s
# wildcard-example-com True wildcard-example-com-tls 47s

DNS-01 is slower than HTTP-01 because of DNS propagation. Give it a minute or two. If it takes more than five minutes, start troubleshooting (see below).

When things go wrong

They will. Here’s how to figure out what happened.

cert-manager has a chain of resources that it creates when processing a certificate request. Follow the chain from top to bottom:

# 1. Is the Certificate ready?
kubectl get certificates -A

# 2. What does the CertificateRequest say?
kubectl get certificaterequest -A
kubectl describe certificaterequest <name> -n <namespace>

# 3. Is the Issuer healthy?
kubectl get clusterissuer
kubectl describe clusterissuer <name>

# 4. For Let's Encrypt: check the ACME Order and Challenge
kubectl get orders -A
kubectl get challenges -A
kubectl describe challenge <name> -n <namespace>

# 5. Check the cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager --tail=100

Most problems fall into a handful of categories:

Challenge not reachable (HTTP-01). Port 80 isn’t open, or the temporary solver Ingress isn’t being picked up by your controller. Check that ingressClassName in your solver config matches your actual ingress controller.

DNS propagation timeout (DNS-01). The TXT record was created but Let’s Encrypt can’t see it yet. If your cluster’s DNS resolver is slow, you can point cert-manager at public resolvers:

helm upgrade cert-manager oci://quay.io/jetstack/charts/cert-manager \
 --namespace cert-manager \
 --set 'extraArgs={--dns01-recursive-nameservers-only,--dns01-recursive-nameservers=1.1.1.1:53\,9.9.9.9:53}'

Secret in the wrong namespace. The number one DNS-01 headache. ClusterIssuers look for API token secrets in the cert-manager namespace. Not your app namespace. Not default.

Rate limited. You hit Let’s Encrypt production limits. Switch back to staging, wait for the rate limit window to pass (usually 7 days), fix whatever caused the excessive requests, and try again.

Issuer not ready. kubectl describe clusterissuer will tell you why. Usually it’s a bad email address, an unreachable ACME server, or a malformed privateKeySecretRef.

Nothing happens at all. Check the annotation name. cert-manager.io/cluster-issuer is not the same as cert-manager.io/issuer. Using the wrong one for your Issuer type produces zero errors and zero certificates. It’s the most frustrating failure mode because there’s nothing in the logs.

A note on certificate lifetimes

Let’s Encrypt has been shortening certificate lifetimes. They started at 90 days, moved to 64, and are now pushing toward 45-day certificates. This doesn’t matter much if you’re using cert-manager because it handles renewal automatically (default is 30 days before expiration). But it does mean that if cert-manager breaks and you don’t notice, you have less runway before things go red.

Monitor your certificates. At minimum, set up an alert on cert-manager pod health. Ideally, also alert on certificates that haven’t renewed within their expected window.

Wrapping up

Here’s the workflow once everything is configured:

Deploy an Ingress or Gateway with the cert-manager annotation
cert-manager creates a Certificate resource
cert-manager talks to Let’s Encrypt, completes the challenge
The signed certificate lands in a Kubernetes secret
Your ingress controller or gateway picks it up
cert-manager renews it before expiration

No cron jobs. No manual certbot renew. No calendar reminders. Just certificates that work.

If you’re running through this series from the beginning, you now have default TLS certs on your ingress controllers and automated certificate management with Let’s Encrypt. That’s a solid foundation.

Next up in this series: DNS automation with external-dns. Because if cert-manager removes the manual work from certificates, external-dns does the same thing for DNS records.

Kubernetes for Cloud Engineers: Default TLS Certificates

Sat, 04 Apr 2026 00:00:00 +0000

This is the first post in a series for new operations and cloud engineers getting started with Kubernetes. Whether you’re running K3s on a Raspberry Pi, EKS in AWS, AKS in Azure, or anything in between, the concepts are the same. Let’s get into it.

The problem

You’ve got a Kubernetes cluster. You’ve got an ingress controller routing traffic. You hit your app over HTTPS and… you get a browser warning about an untrusted self-signed certificate. Or worse, you’ve got ten services and you’re copy-pasting the same TLS secret into every single Ingress resource.

The fix: set a default TLS certificate on your ingress controller. One cert to rule them all. Any request that doesn’t match a more specific TLS config falls back to this default. Clean, simple, done.

Every controller does this slightly differently. Let’s walk through each one.

First, create your TLS secret

This part is the same regardless of which controller you’re running. You need a Kubernetes TLS secret containing your certificate and private key.

# Create the TLS secret from your cert and key files
kubectl create secret tls default-tls \
 --cert=tls.crt \
 --key=tls.key \
 -n default

# secret/default-tls created

Your cert file should contain the full chain: leaf → intermediate → root. If you’re using cert-manager or Let’s Encrypt, this is handled for you. If you’re doing it by hand, get the order right or you’ll be debugging trust chain errors at 2am.

Ingress Controllers (Legacy Ingress API)

Heads up: Ingress NGINX is retired. The kubernetes/ingress-nginx project, the one most of us grew up on, officially reached end-of-life on March 24, 2026. The repository is now read-only. No more releases, no bug fixes, no security patches. Your existing deployments won’t break overnight, but you’re flying without a safety net.

The Kubernetes Steering Committee and Security Response Committee issued a joint statement in January 2026 urging migration. The recommended path forward is Gateway API, which has been GA since October 2023 and now has 20+ conformant implementations.

If you’re starting fresh, skip to the Gateway API section below. If you’re migrating, check out ingress2gateway, the official migration tool that now supports 30+ annotation conversions.

Note: NGINX Inc.’s commercial ingress controller (F5/NGINX) is a completely separate codebase and remains actively maintained. Don’t confuse the two.

Ingress NGINX (kubernetes/ingress-nginx)

Even though it’s retired, you’ll encounter this in the wild for a while. The default cert is set via a controller argument.

With Helm:

# values.yaml
controller:
 extraArgs:
 default-ssl-certificate: "default/default-tls"

helm upgrade ingress-nginx ingress-nginx/ingress-nginx \
 -f values.yaml \
 -n ingress-nginx

Without Helm, patch the controller deployment directly:

# Add to the container args in the ingress-nginx-controller Deployment
spec:
 containers:
 - name: controller
 args:
 - /nginx-ingress-controller
 - --default-ssl-certificate=default/default-tls

The format is namespace/secret-name. Without this flag, NGINX generates a self-signed certificate for the catch-all server. That’s the browser warning you’re seeing.

Gotcha: If your Ingress resource has a tls: section but no secretName, NGINX uses this default cert and forces an HTTPS redirect. If the tls: section is missing entirely, it still serves the default cert but does not redirect. Use force-ssl-redirect: "true" in the ConfigMap if you want to enforce HTTPS everywhere.

HAProxy Ingress

There are two HAProxy ingress projects: the community haproxy-ingress and the one from HAProxy Technologies. Both support the same flag pattern.

With Helm (HAProxy Technologies):

# values.yaml
controller:
 defaultTLSSecret:
 secretNamespace: default
 secret: default-tls

With controller args (both projects):

--default-ssl-certificate=default/default-tls

helm upgrade haproxy-ingress haproxytech/kubernetes-ingress \
 --set controller.defaultTLSSecret.secretNamespace=default \
 --set controller.defaultTLSSecret.secret=default-tls \
 -n haproxy-ingress

Without a default cert configured, both HAProxy implementations generate a self-signed fake certificate. Same story as NGINX.

Note: HAProxy Technologies controller v1.11+ automatically enables QUIC (HTTP/3) when you set a default TLS cert. If you don’t want that yet, add --disable-quic.

Istio Ingress Gateway

Istio doesn’t use the Kubernetes Ingress API at all. It has its own Gateway CRD (not the same as Gateway API, I know, the naming is painful).

Important: The TLS secret must live in the same namespace as the Istio ingress gateway pod, typically istio-system.

kubectl create secret tls default-tls \
 --cert=tls.crt \
 --key=tls.key \
 -n istio-system

# secret/default-tls created

Then create the Gateway resource:

# istio-gateway.yaml
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
 name: default-gateway
spec:
 selector:
 istio: ingressgateway
 servers:
 - port:
 number: 443
 name: https
 protocol: HTTPS
 tls:
 mode: SIMPLE
 credentialName: default-tls
 hosts:
 - "*.example.com"

kubectl apply -f istio-gateway.yaml
# gateway.networking.istio.io/default-gateway created

Gotchas:

credentialName must exactly match the Kubernetes secret name. No namespace/name format here. It just looks in its own namespace.
Istio doesn’t have a single “default certificate” concept. You configure TLS per-server block on the Gateway. To make it act as a default, use a wildcard host like *.example.com.
Wrong namespace is the #1 debugging headache. If TLS isn’t working, check the secret namespace first.

Gateway API

Gateway API is the future. It’s the Kubernetes-native standard from SIG-Network, GA since October 2023, and every major ingress controller is building an implementation. If you’re starting fresh, start here.

The big difference: there’s no --default-ssl-certificate flag. TLS is configured declaratively on the Gateway resource itself, per-listener. Each HTTPS listener explicitly references a certificate via certificateRefs.

NGINX Gateway Fabric

NGINX Gateway Fabric is NGINX’s Gateway API implementation, completely separate from the retired Ingress NGINX project.

# gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: default-gateway
spec:
 gatewayClassName: nginx
 listeners:
 - name: http
 port: 80
 protocol: HTTP
 - name: https
 hostname: "*.example.com"
 port: 443
 protocol: HTTPS
 tls:
 mode: Terminate
 certificateRefs:
 - kind: Secret
 name: default-tls

kubectl apply -f gateway.yaml
# gateway.gateway.networking.k8s.io/default-gateway created

Then attach your HTTPRoutes to the HTTPS listener:

# route.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: my-app
spec:
 parentRefs:
 - name: default-gateway
 sectionName: https
 hostnames:
 - "app.example.com"
 rules:
 - backendRefs:
 - name: my-app
 port: 80

With cert-manager, add an annotation and cert-manager handles the rest:

# gateway.yaml: cert-manager will create and manage the secret
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: default-gateway
 annotations:
 cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
 gatewayClassName: nginx
 listeners:
 - name: https
 hostname: "app.example.com"
 port: 443
 protocol: HTTPS
 tls:
 mode: Terminate
 certificateRefs:
 - kind: Secret
 name: default-tls

HAProxy (Gateway API)

The beauty of Gateway API is that it’s a standard. The Gateway resource looks almost identical regardless of the underlying implementation. You just swap the gatewayClassName.

# gateway.yaml: same pattern, different class
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: default-gateway
spec:
 gatewayClassName: haproxy
 listeners:
 - name: https
 hostname: "*.example.com"
 port: 443
 protocol: HTTPS
 tls:
 mode: Terminate
 certificateRefs:
 - kind: Secret
 name: default-tls

kubectl apply -f gateway.yaml
# gateway.gateway.networking.k8s.io/default-gateway created

That’s it. Same spec. Same structure. The gatewayClassName tells Kubernetes which controller reconciles the resource. This is exactly why Gateway API is the future. You can swap implementations without rewriting your config.

Gotchas for both implementations:

Gateway API doesn’t have a global “default certificate” flag. Each HTTPS listener must explicitly reference a cert via certificateRefs. To cover everything, use a wildcard hostname.
The TLS secret must be in the same namespace as the Gateway by default. For cross-namespace references, create a ReferenceGrant.
TLS modes: Terminate means the gateway decrypts traffic (most common). Passthrough forwards encrypted traffic as-is. Use this with TLSRoute, not HTTPRoute.
NGINX Gateway Fabric currently supports only a single certificateRef per listener. If you need multiple certs, create multiple listeners.

Wrapping up

Here’s the cheat sheet:

Controller	Method	Format
Ingress NGINX (retired)	`--default-ssl-certificate`	`namespace/secret`
HAProxy Ingress	`--default-ssl-certificate`	`namespace/secret`
Istio	`credentialName` on Gateway server	secret name (same namespace)
Gateway API (any impl)	`certificateRefs` on listener	secret reference

If you’re starting fresh, go straight to Gateway API. Pick an implementation (NGINX Gateway Fabric, HAProxy, Envoy Gateway, whatever fits your stack), define your Gateway with a TLS listener, and move on.

If you’re migrating from Ingress NGINX, take a breath. Your cluster won’t explode tomorrow. But start planning the move. The ingress2gateway tool can automate most of the conversion, and the Gateway API spec is stable and well-documented.

Next up in this series: cert-manager and automated TLS. Creating secrets by hand is so 2024.

Engineering Managers Should Make Engineers Better

Sat, 07 Mar 2026 00:00:00 +0000

During my regularly scheduled lurking on Hacker News, I came across Don’t Become an Engineering Manager, making the case that senior engineers should stay IC. The ladder is flattening, Staff pays better, tech is moving too fast to step away. Probably right, but it bugs me.

It treats the EM role like a career trade. A thing you optimize for or against. And when you look at it that way, sure, the math is shaky right now. But that skips the question I actually care about: what does a good engineering manager do for the engineers on their team?

The actual job

The best EMs I’ve worked with spend most of their time on the unglamorous stuff. Not “shielding the team” in the vague r/LinkedInLunatics way. The specific stuff. Why has the deploy pipeline been flaky for two weeks and nobody’s fixed it? Two teams are about to build the same thing, so get them in a room. Product is asking for something unreasonable and the engineers are too polite to say so.

None of that shows up on a career ladder. It barely shows up in performance reviews. But it’s the difference between a team that ships and a team that fights its own org.

Making people better

The part of the job I care about most doesn’t come up in any of these “should you become an EM” debates. Are the engineers on your team getting better? Not in the promotion-packet sense. Is the person who joined six months ago actually understanding the system now? Is the senior who’s great at heads-down execution starting to think about what happens after they ship?

That work is mostly invisible. It’s a one-on-one that looks like a casual conversation. It’s asking “what would you do?” instead of giving the answer. It’s putting someone slightly past their comfort zone and making sure they don’t eat it.

So should you become an EM?

If you care about people, and you want to make those people better engineers, then yes. Engineers need you.