Kubernetes on houdeshell.dev

Software that I ❤

Mon, 01 Jun 2020 00:00:00 +0000

I bounce between Mac, Windows, and Linux daily — and honestly, the tools matter more than the OS at this point. Great software is great software, and I’ve been lucky enough to build a career on top of things that other people built well.

This is the stuff I actually reach for. Not a “best of” list, not sponsored, not comprehensive. Just software that makes me better at what I do — or at least makes the work more fun.

Software Development

IDE

JetBrains Rider

Cross-platform .NET IDE. Fast, smart, and doesn’t need Visual Studio’s weight to get the job done.

IDE

Visual Studio

The OG. Heavy, but when you need the full .NET debugging experience, nothing else comes close.

editor

VS Code

Extension ecosystem is unmatched. Somehow Electron done right.

editor

Neovim

Vim reborn. Lua config, LSP native, and a plugin ecosystem that won’t quit. :wq is a lifestyle.

Claude Code

AI pair programmer in your terminal. It’s writing this page right now.

version control

Git

The version control system that won. Love it or hate it, you can’t ship without it.

version control

lazygit

Terminal UI for git that makes interactive rebases feel like cheating.

JSON

jq

sed for JSON. Once you learn the syntax, you’ll pipe everything through it.

database

JetBrains DataGrip

SQL IDE that actually understands your schema. Autocomplete that works across joins.

database

DBeaver

Universal database tool that actually works. Connect to anything, query everything.

database

SSMS

Microsoft SQL Management Studio. If you’re in SQL Server land, you already know.

database

PostgreSQL

The database that keeps getting better. Extensions, JSON support, and rock-solid reliability.

containers

Docker

“Works on my machine” became “works on every machine.” Changed how we ship software.

Infrastructure

orchestration

Kubernetes

Container orchestration that makes you mass produce YAML for a living.

orchestration

K3s

All of Kubernetes in a single binary. Perfect for homelab, edge, and production deployments.

IaC

Terraform / OpenTofu

Infrastructure as code. OpenTofu if you prefer the open-source fork without the license drama.

networking

Tailscale

WireGuard-based mesh VPN that just works. Connect everything without opening a single port.

observability

Grafana

Dashboards for everything. Pairs with Prometheus, Loki, Tempo — the whole observability stack.

Kubernetes

k9s

Terminal UI for Kubernetes. Makes cluster management feel like a video game. Way faster than raw kubectl.

Kubernetes

Lens

The Kubernetes IDE. When you want to point and click your way through a cluster without shame.

it's complicated

Cloudflare

You love them. You hate them. Your DNS is already there. Their edge network is everywhere, their pricing is unbeatable, and their product sprawl is terrifying. Stockholm syndrome as a service.

Shell & Terminal

shell

Bash

The shell that’s been there since before you were born. Simple, portable, everywhere.

shell

Zsh

Bash’s cooler sibling. Tab completion, globbing, and plugin support that actually makes the terminal enjoyable.

prompt

Starship

Cross-shell prompt written in Rust. Fast, pretty, and infinitely configurable.

multiplexer

tmux

Terminal multiplexer. SSH into a box, detach, come back tomorrow. Your session is still there.

coreutils

eza

Modern replacement for ls. Colors, git status, and tree view out of the box. Written in Rust, obviously.

Utilities / Apps

macOS

AltTab

Why does macOS think a minimized window is an inactive window and not let you switch to it? This fixes that.

About

Sat, 11 Apr 2020 00:00:00 +0000

whoami

Chris Houdeshell

bit herder. pointer wrecker. yaml apologist.

stack: dotnet, postgres, k8s, azure, too-much-sql

location: State College, PA

coffee intake: dangerously high

github / linkedin / talks / email

I’m a bit herder, a pointer wrecker, and HTTP is my best friend. 0xA3 & 0x7C are also my friends, but not 0x08.

I spend my days somewhere between writing code and making sure other people can write better code. I love distributed systems, low-level embedded devices, and anything that has a content-type. I also have an unhealthy relationship with YAML, but that’s a Kubernetes problem.

The day job

During the day, I lead engineering and operations at EnergyCAP — a SaaS platform that helps organizations track utility spend, energy consumption, and sustainability goals across massive building portfolios. We’re talking $100B+ in utility bill spend tracked across 350,000+ sites. Government, education, healthcare, commercial real estate — if it has a meter, we probably manage it.

I started here as a developer in 2009 and have been building ever since — through senior roles, into leadership, and eventually into the VP seat where I get to shape both the tech and the teams behind it. Some days that means architecture decisions. Most days it means removing blockers so smart people can do their best work. And yes, sometimes it’s just meetings about meetings.

My philosophy is simple: technology should serve people. Build teams where people feel safe to be wrong, give them the tools to be great, and get out of the way.

Things I geek out about

Distributed systems, container orchestration, database performance tuning, developer tooling, process dumps, leadership that doesn’t suck, making engineers’ lives better, and the eternal quest for a perfect terminal setup.

I also talk at things

I like getting on stage and nerding out about the things I’ve learned the hard way. My talks tend to fall into a few buckets — and yes, I have strong opinions about all of them.

AI & the developer experience

The AI wave hit and I leaned in. I’ve talked about using LLMs as debugging partners, building RAG pipelines that actually know your business context, and the ethical lines we should be drawing as we hand more work to machines.

My Rubber Duck is a Large Language Model
Unleashing the Power of the AI Wizards: Retrieval-Augmented Generation Spells

Kubernetes & cloud native

I was late to the Kubernetes party — and I’ll tell you all about it. From lightweight K3s workshops to navigating the 900+ services in the CNCF landscape, to making the case that cloud-native principles work even when you’re not in the cloud.

Kubernetes Chronicles: Late to the Party, Big on Adventure!
K3s — Half the Size, Twice as Awesome (workshop)
Orchestrating Machine Learning Workloads with Kubernetes
Exploring the Cloud Native Landscape
Cloud Native is Only for the Cloud, Right?
Nomad — Orchestration Doesn't Start with a K

Performance & databases

I’ve spent an unreasonable amount of time staring at query plans. These talks cover squeezing performance out of SQL Server, using DMVs like cheat codes, and that one time we took a 10-hour process down to 10 minutes.

ReArchitecting Data: 10 Hours to 10 Minutes
Mastering SQL Server Performance Optimization (workshop)
SQL Server DMVs That Give Me Superpowers
Achieving Continuous High Performance with Query Store
Practical High Performance: C# Edition
Intrinsics in .NET: Start Somewhere

Production war stories & leadership

Production breaks. Systems fail. The interesting part is what you do next. I also talk about the human side — what I’ve learned (and gotten wrong) leading engineering teams.

Bug Squashing with Process Dumps
Recovery by Design: A Postmortem Adventure
Joining the Cloud: Our Journey
TIL as a CTO

The wildcard

Sometimes you just want to build something fun with your hands.

Paper Circuits: Origami for a New Generation

Let’s talk

I drink a bit too much coffee while rambling on about technology. If you’re willing to have a conversation — I’m ready to buy you a cup.

~ echo $CONTACT
email: chris@houdeshell.dev
github: choudeshell
linkedin: choudeshell
twitter: @choudeshell

Kubernetes for Cloud Engineers: Automating TLS with cert-manager

Sat, 04 Apr 2026 00:00:00 +0000

This is part two of a series for new operations and cloud engineers getting into Kubernetes. In part one we covered how to set default TLS certificates on your ingress controllers. That works, but it means you’re managing certs by hand. Let’s fix that.

Why cert-manager?

In the last post, we created TLS secrets manually. That’s fine for getting started. It’s not fine for production. Certificates expire. People forget to renew them. And then you’re getting paged at 3am because your site is serving a browser warning.

cert-manager is a Kubernetes-native certificate management controller. It watches your cluster for resources that need certificates, talks to a Certificate Authority (usually Let’s Encrypt), handles the challenge/response dance, stores the resulting cert as a Kubernetes secret, and renews it before it expires. All automatically. You configure it once and move on with your life.

It’s one of those tools that, once installed, you forget it’s there. That’s the highest compliment I can give infrastructure software.

Installing cert-manager

Two options. Pick whichever matches your deployment style.

Option 1: Helm (recommended)

helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
 --version v1.20.0 \
 --namespace cert-manager \
 --create-namespace \
 --set crds.enabled=true

# NAME: cert-manager
# NAMESPACE: cert-manager
# STATUS: deployed

Option 2: Static manifests

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.20.0/cert-manager.yaml

Both methods install the same thing: the cert-manager controller, webhook, cainjector, and six CRDs. Those CRDs are the building blocks you’ll use for everything else.

# Verify everything is running
kubectl get pods -n cert-manager

# NAME READY STATUS
# cert-manager-5c4b5f7b9-xk2lq 1/1 Running
# cert-manager-cainjector-7f694c-m8p 1/1 Running
# cert-manager-webhook-7cd8c8-9tn2f 1/1 Running

Three pods running. That’s what you want to see.

Issuers: telling cert-manager where to get certificates

cert-manager doesn’t know anything about Let’s Encrypt out of the box. You need to tell it where to go and how to prove you own your domains. That’s what Issuers do.

There are two flavors:

Issuer is namespace-scoped. It can only issue certs for resources in the same namespace.
ClusterIssuer is cluster-scoped. It works across all namespaces.

For most setups, you want a ClusterIssuer. One config, whole cluster. If you have teams that need isolated certificate management per namespace, use an Issuer. But start simple.

Staging first. Always.

Let’s Encrypt has two environments: staging and production. They work identically, but staging issues certificates signed by a fake CA that browsers don’t trust. The certificates themselves are structurally real. They just won’t show a green lock.

Why does this matter? Because Let’s Encrypt production has rate limits. Tight ones. 50 certificates per registered domain per week. 5 duplicate certificates per week. If you mess up your config and keep retrying, you can lock yourself out for days.

Staging has much more generous limits. So you should always get your pipeline working against staging first, then switch to production once you know everything is wired up correctly.

Create a staging ClusterIssuer

# staging-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-staging
spec:
 acme:
 server: https://acme-staging-v02.api.letsencrypt.org/directory
 email: you@example.com
 privateKeySecretRef:
 name: letsencrypt-staging-account-key
 solvers:
 - http01:
 ingress:
 ingressClassName: nginx

kubectl apply -f staging-issuer.yaml
# clusterissuer.cert-manager.io/letsencrypt-staging created

# Check that it registered successfully
kubectl get clusterissuer letsencrypt-staging

# NAME READY AGE
# letsencrypt-staging True 30s

READY: True means cert-manager registered an account with Let’s Encrypt staging. If you see False, run kubectl describe clusterissuer letsencrypt-staging and read the events.

Create the production ClusterIssuer

Same thing, different ACME server URL:

# production-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-prod
spec:
 acme:
 server: https://acme-v02.api.letsencrypt.org/directory
 email: you@example.com
 privateKeySecretRef:
 name: letsencrypt-prod-account-key
 solvers:
 - http01:
 ingress:
 ingressClassName: nginx

Use separate privateKeySecretRef names for staging and production. The ACME accounts are completely independent. You can’t reuse keys between environments.

HTTP-01 vs DNS-01: how you prove domain ownership

When you request a certificate, Let’s Encrypt needs proof that you actually own the domain. There are two ways to do this, and which one you pick depends on your setup.

HTTP-01

Let’s Encrypt sends an HTTP request to http://yourdomain.com/.well-known/acme-challenge/<token>. cert-manager spins up a temporary pod and Ingress (or HTTPRoute) to serve the response. Once validated, the temp resources get cleaned up.

This is the simpler option. It works if:

Port 80 is publicly reachable on your cluster
You don’t need wildcard certificates

It does not work if:

You’re behind a firewall with no public HTTP access
You need *.example.com certs

DNS-01

Instead of an HTTP request, Let’s Encrypt checks for a specific TXT record at _acme-challenge.yourdomain.com. cert-manager talks to your DNS provider’s API to create and clean up the record.

This is the only way to get wildcard certificates. It’s also the right choice for internal domains that aren’t publicly accessible. The tradeoff is more setup. You need to give cert-manager API credentials for your DNS provider, and DNS propagation delays can slow things down.

Built-in DNS providers: Cloudflare, Route53, Google Cloud DNS, Azure DNS, DigitalOcean, Akamai, ACMEDNS, and RFC-2136. There are webhook extensions for 30+ more.

Wiring it up with Ingress

If you’re using the traditional Ingress API, cert-manager makes this really clean. Add an annotation to your Ingress resource, include a tls block, and cert-manager handles the rest.

# my-app-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: my-app
 annotations:
 cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
 ingressClassName: nginx
 rules:
 - host: app.example.com
 http:
 paths:
 - pathType: Prefix
 path: /
 backend:
 service:
 name: my-app
 port:
 number: 80
 tls:
 - hosts:
 - app.example.com
 secretName: app-example-com-tls

That’s it. cert-manager sees the annotation, reads the tls block, creates a Certificate resource, kicks off the ACME challenge, and stores the signed certificate in the app-example-com-tls secret. Your ingress controller picks up the secret and starts serving HTTPS.

When the cert is 30 days from expiration, cert-manager renews it automatically.

Staging first, remember. Point the annotation at letsencrypt-staging until you see a valid (but untrusted) cert in your browser. Then swap it to letsencrypt-prod. You’ll need to delete the old secret so cert-manager issues a fresh one from production.

Use cert-manager.io/cluster-issuer for ClusterIssuers and cert-manager.io/issuer for namespace-scoped Issuers. Mix them up and nothing will happen. No error, no cert. Just silence. Ask me how I know.

Wiring it up with Gateway API

If you followed the first post in this series, you know Gateway API is where things are heading. cert-manager supports it, but it needs a little extra configuration.

Enable Gateway API support

Gateway API support is not on by default. You need to opt in.

helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
 --namespace cert-manager \
 --set crds.enabled=true \
 --set config.enableGatewayAPI=true

Make sure the Gateway API CRDs are installed in your cluster too:

kubectl apply --server-side \
 -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/standard-install.yaml

If you installed the CRDs after cert-manager was already running, restart it so it picks them up:

kubectl rollout restart deployment cert-manager -n cert-manager

Annotate your Gateway

The pattern is similar to Ingress. Add the annotation, reference a secret in the listener’s certificateRefs, and cert-manager fills in the rest.

# gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: my-gateway
 annotations:
 cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
 gatewayClassName: nginx
 listeners:
 - name: https
 hostname: app.example.com
 port: 443
 protocol: HTTPS
 tls:
 mode: Terminate
 certificateRefs:
 - name: app-example-com-tls

cert-manager sees the annotation and the listener config, creates a Certificate for app.example.com, and stores the result in the app-example-com-tls secret. The Gateway picks it up and starts terminating TLS.

Three things must be true for cert-manager to act on a Gateway listener:

The hostname is not empty
The TLS mode is Terminate (not Passthrough)
There’s at least one entry in certificateRefs

Miss any of those and cert-manager quietly ignores the listener.

Wildcard certificates with DNS-01

This is where DNS-01 earns its keep. Let’s say you want a single cert that covers *.example.com so every subdomain is automatically secured. HTTP-01 can’t do this. Only DNS-01 can.

I’ll use Cloudflare as the DNS provider since it’s common (and because I have a love/hate relationship with them that we’ve already discussed on the software page).

Step 1: Create a Cloudflare API token

In the Cloudflare dashboard, create an API token with these permissions:

Zone / DNS / Edit
Zone / Zone / Read

Scope it to the zone you need. Don’t use a global API key if you can avoid it.

Step 2: Store the token in your cluster

# The secret MUST be in the cert-manager namespace for ClusterIssuers
apiVersion: v1
kind: Secret
metadata:
 name: cloudflare-api-token
 namespace: cert-manager
type: Opaque
stringData:
 api-token: your-cloudflare-api-token-here

That namespace bit is important. When you use a ClusterIssuer, cert-manager looks for referenced secrets in the namespace where cert-manager itself is installed. Not the namespace of your Certificate. Not the namespace of your app. The cert-manager namespace. This trips up everyone at least once.

Step 3: Create a DNS-01 ClusterIssuer

# dns-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-prod-dns
spec:
 acme:
 server: https://acme-v02.api.letsencrypt.org/directory
 email: you@example.com
 privateKeySecretRef:
 name: letsencrypt-prod-dns-account-key
 solvers:
 - dns01:
 cloudflare:
 apiTokenSecretRef:
 name: cloudflare-api-token
 key: api-token

Step 4: Request the wildcard cert

# wildcard-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: wildcard-example-com
 namespace: default
spec:
 secretName: wildcard-example-com-tls
 issuerRef:
 name: letsencrypt-prod-dns
 kind: ClusterIssuer
 dnsNames:
 - "*.example.com"
 - example.com

Include both *.example.com and example.com in the dnsNames. The wildcard only covers subdomains, not the bare domain itself.

kubectl apply -f wildcard-cert.yaml
# certificate.cert-manager.io/wildcard-example-com created

# Watch it work
kubectl get certificate wildcard-example-com -w

# NAME READY SECRET AGE
# wildcard-example-com False wildcard-example-com-tls 5s
# wildcard-example-com True wildcard-example-com-tls 47s

DNS-01 is slower than HTTP-01 because of DNS propagation. Give it a minute or two. If it takes more than five minutes, start troubleshooting (see below).

When things go wrong

They will. Here’s how to figure out what happened.

cert-manager has a chain of resources that it creates when processing a certificate request. Follow the chain from top to bottom:

# 1. Is the Certificate ready?
kubectl get certificates -A

# 2. What does the CertificateRequest say?
kubectl get certificaterequest -A
kubectl describe certificaterequest <name> -n <namespace>

# 3. Is the Issuer healthy?
kubectl get clusterissuer
kubectl describe clusterissuer <name>

# 4. For Let's Encrypt: check the ACME Order and Challenge
kubectl get orders -A
kubectl get challenges -A
kubectl describe challenge <name> -n <namespace>

# 5. Check the cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager --tail=100

Most problems fall into a handful of categories:

Challenge not reachable (HTTP-01). Port 80 isn’t open, or the temporary solver Ingress isn’t being picked up by your controller. Check that ingressClassName in your solver config matches your actual ingress controller.

DNS propagation timeout (DNS-01). The TXT record was created but Let’s Encrypt can’t see it yet. If your cluster’s DNS resolver is slow, you can point cert-manager at public resolvers:

helm upgrade cert-manager oci://quay.io/jetstack/charts/cert-manager \
 --namespace cert-manager \
 --set 'extraArgs={--dns01-recursive-nameservers-only,--dns01-recursive-nameservers=1.1.1.1:53\,9.9.9.9:53}'

Secret in the wrong namespace. The number one DNS-01 headache. ClusterIssuers look for API token secrets in the cert-manager namespace. Not your app namespace. Not default.

Rate limited. You hit Let’s Encrypt production limits. Switch back to staging, wait for the rate limit window to pass (usually 7 days), fix whatever caused the excessive requests, and try again.

Issuer not ready. kubectl describe clusterissuer will tell you why. Usually it’s a bad email address, an unreachable ACME server, or a malformed privateKeySecretRef.

Nothing happens at all. Check the annotation name. cert-manager.io/cluster-issuer is not the same as cert-manager.io/issuer. Using the wrong one for your Issuer type produces zero errors and zero certificates. It’s the most frustrating failure mode because there’s nothing in the logs.

A note on certificate lifetimes

Let’s Encrypt has been shortening certificate lifetimes. They started at 90 days, moved to 64, and are now pushing toward 45-day certificates. This doesn’t matter much if you’re using cert-manager because it handles renewal automatically (default is 30 days before expiration). But it does mean that if cert-manager breaks and you don’t notice, you have less runway before things go red.

Monitor your certificates. At minimum, set up an alert on cert-manager pod health. Ideally, also alert on certificates that haven’t renewed within their expected window.

Wrapping up

Here’s the workflow once everything is configured:

Deploy an Ingress or Gateway with the cert-manager annotation
cert-manager creates a Certificate resource
cert-manager talks to Let’s Encrypt, completes the challenge
The signed certificate lands in a Kubernetes secret
Your ingress controller or gateway picks it up
cert-manager renews it before expiration

No cron jobs. No manual certbot renew. No calendar reminders. Just certificates that work.

If you’re running through this series from the beginning, you now have default TLS certs on your ingress controllers and automated certificate management with Let’s Encrypt. That’s a solid foundation.

Next up in this series: DNS automation with external-dns. Because if cert-manager removes the manual work from certificates, external-dns does the same thing for DNS records.

Kubernetes for Cloud Engineers: Default TLS Certificates

Sat, 04 Apr 2026 00:00:00 +0000

This is the first post in a series for new operations and cloud engineers getting started with Kubernetes. Whether you’re running K3s on a Raspberry Pi, EKS in AWS, AKS in Azure, or anything in between — the concepts are the same. Let’s get into it.

The problem

You’ve got a Kubernetes cluster. You’ve got an ingress controller routing traffic. You hit your app over HTTPS and… you get a browser warning about an untrusted self-signed certificate. Or worse, you’ve got ten services and you’re copy-pasting the same TLS secret into every single Ingress resource.

The fix: set a default TLS certificate on your ingress controller. One cert to rule them all. Any request that doesn’t match a more specific TLS config falls back to this default. Clean, simple, done.

Every controller does this slightly differently. Let’s walk through each one.

First — create your TLS secret

This part is the same regardless of which controller you’re running. You need a Kubernetes TLS secret containing your certificate and private key.

# Create the TLS secret from your cert and key files
kubectl create secret tls default-tls \
 --cert=tls.crt \
 --key=tls.key \
 -n default

# secret/default-tls created

Your cert file should contain the full chain: leaf → intermediate → root. If you’re using cert-manager or Let’s Encrypt, this is handled for you. If you’re doing it by hand — get the order right or you’ll be debugging trust chain errors at 2am.

Ingress Controllers (Legacy Ingress API)

Heads up: Ingress NGINX is retired. The kubernetes/ingress-nginx project — the one most of us grew up on — officially reached end-of-life on March 24, 2026. The repository is now read-only. No more releases, no bug fixes, no security patches. Your existing deployments won’t break overnight, but you’re flying without a safety net.

The Kubernetes Steering Committee and Security Response Committee issued a joint statement in January 2026 urging migration. The recommended path forward is Gateway API, which has been GA since October 2023 and now has 20+ conformant implementations.

If you’re starting fresh — skip to the Gateway API section below. If you’re migrating, check out ingress2gateway — the official migration tool that now supports 30+ annotation conversions.

Note: NGINX Inc.’s commercial ingress controller (F5/NGINX) is a completely separate codebase and remains actively maintained. Don’t confuse the two.

Ingress NGINX (kubernetes/ingress-nginx)

Even though it’s retired, you’ll encounter this in the wild for a while. The default cert is set via a controller argument.

With Helm:

# values.yaml
controller:
 extraArgs:
 default-ssl-certificate: "default/default-tls"

helm upgrade ingress-nginx ingress-nginx/ingress-nginx \
 -f values.yaml \
 -n ingress-nginx

Without Helm — patch the controller deployment directly:

# Add to the container args in the ingress-nginx-controller Deployment
spec:
 containers:
 - name: controller
 args:
 - /nginx-ingress-controller
 - --default-ssl-certificate=default/default-tls

The format is namespace/secret-name. Without this flag, NGINX generates a self-signed certificate for the catch-all server — that’s the browser warning you’re seeing.

Gotcha: If your Ingress resource has a tls: section but no secretName, NGINX uses this default cert and forces an HTTPS redirect. If the tls: section is missing entirely, it still serves the default cert but does not redirect. Use force-ssl-redirect: "true" in the ConfigMap if you want to enforce HTTPS everywhere.

HAProxy Ingress

There are two HAProxy ingress projects — the community haproxy-ingress and the one from HAProxy Technologies. Both support the same flag pattern.

With Helm (HAProxy Technologies):

# values.yaml
controller:
 defaultTLSSecret:
 secretNamespace: default
 secret: default-tls

With controller args (both projects):

--default-ssl-certificate=default/default-tls

helm upgrade haproxy-ingress haproxytech/kubernetes-ingress \
 --set controller.defaultTLSSecret.secretNamespace=default \
 --set controller.defaultTLSSecret.secret=default-tls \
 -n haproxy-ingress

Without a default cert configured, both HAProxy implementations generate a self-signed fake certificate — same story as NGINX.

Note: HAProxy Technologies controller v1.11+ automatically enables QUIC (HTTP/3) when you set a default TLS cert. If you don’t want that yet, add --disable-quic.

Istio Ingress Gateway

Istio doesn’t use the Kubernetes Ingress API at all. It has its own Gateway CRD (not the same as Gateway API — I know, the naming is painful).

Important: The TLS secret must live in the same namespace as the Istio ingress gateway pod — typically istio-system.

kubectl create secret tls default-tls \
 --cert=tls.crt \
 --key=tls.key \
 -n istio-system

# secret/default-tls created

Then create the Gateway resource:

# istio-gateway.yaml
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
 name: default-gateway
spec:
 selector:
 istio: ingressgateway
 servers:
 - port:
 number: 443
 name: https
 protocol: HTTPS
 tls:
 mode: SIMPLE
 credentialName: default-tls
 hosts:
 - "*.example.com"

kubectl apply -f istio-gateway.yaml
# gateway.networking.istio.io/default-gateway created

Gotchas:

credentialName must exactly match the Kubernetes secret name. No namespace/name format here — it just looks in its own namespace.
Istio doesn’t have a single “default certificate” concept. You configure TLS per-server block on the Gateway. To make it act as a default, use a wildcard host like *.example.com.
Wrong namespace is the #1 debugging headache. If TLS isn’t working, check the secret namespace first.

Gateway API

Gateway API is the future. It’s the Kubernetes-native standard from SIG-Network, GA since October 2023, and every major ingress controller is building an implementation. If you’re starting fresh — start here.

The big difference: there’s no --default-ssl-certificate flag. TLS is configured declaratively on the Gateway resource itself, per-listener. Each HTTPS listener explicitly references a certificate via certificateRefs.

NGINX Gateway Fabric

NGINX Gateway Fabric is NGINX’s Gateway API implementation — completely separate from the retired Ingress NGINX project.

# gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: default-gateway
spec:
 gatewayClassName: nginx
 listeners:
 - name: http
 port: 80
 protocol: HTTP
 - name: https
 hostname: "*.example.com"
 port: 443
 protocol: HTTPS
 tls:
 mode: Terminate
 certificateRefs:
 - kind: Secret
 name: default-tls

kubectl apply -f gateway.yaml
# gateway.gateway.networking.k8s.io/default-gateway created

Then attach your HTTPRoutes to the HTTPS listener:

# route.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: my-app
spec:
 parentRefs:
 - name: default-gateway
 sectionName: https
 hostnames:
 - "app.example.com"
 rules:
 - backendRefs:
 - name: my-app
 port: 80

With cert-manager — add an annotation and cert-manager handles the rest:

# gateway.yaml — cert-manager will create and manage the secret
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: default-gateway
 annotations:
 cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
 gatewayClassName: nginx
 listeners:
 - name: https
 hostname: "app.example.com"
 port: 443
 protocol: HTTPS
 tls:
 mode: Terminate
 certificateRefs:
 - kind: Secret
 name: default-tls

HAProxy (Gateway API)

The beauty of Gateway API is that it’s a standard. The Gateway resource looks almost identical regardless of the underlying implementation — you just swap the gatewayClassName.

# gateway.yaml — same pattern, different class
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: default-gateway
spec:
 gatewayClassName: haproxy
 listeners:
 - name: https
 hostname: "*.example.com"
 port: 443
 protocol: HTTPS
 tls:
 mode: Terminate
 certificateRefs:
 - kind: Secret
 name: default-tls

kubectl apply -f gateway.yaml
# gateway.gateway.networking.k8s.io/default-gateway created

That’s it. Same spec. Same structure. The gatewayClassName tells Kubernetes which controller reconciles the resource. This is exactly why Gateway API is the future — you can swap implementations without rewriting your config.

Gotchas for both implementations:

Gateway API doesn’t have a global “default certificate” flag. Each HTTPS listener must explicitly reference a cert via certificateRefs. To cover everything, use a wildcard hostname.
The TLS secret must be in the same namespace as the Gateway by default. For cross-namespace references, create a ReferenceGrant.
TLS modes: Terminate means the gateway decrypts traffic (most common). Passthrough forwards encrypted traffic as-is — use this with TLSRoute, not HTTPRoute.
NGINX Gateway Fabric currently supports only a single certificateRef per listener. If you need multiple certs, create multiple listeners.

Wrapping up

Here’s the cheat sheet:

Controller	Method	Format
Ingress NGINX (retired)	`--default-ssl-certificate`	`namespace/secret`
HAProxy Ingress	`--default-ssl-certificate`	`namespace/secret`
Istio	`credentialName` on Gateway server	secret name (same namespace)
Gateway API (any impl)	`certificateRefs` on listener	secret reference

If you’re starting fresh — go straight to Gateway API. Pick an implementation (NGINX Gateway Fabric, HAProxy, Envoy Gateway, whatever fits your stack), define your Gateway with a TLS listener, and move on.

If you’re migrating from Ingress NGINX, take a breath. Your cluster won’t explode tomorrow. But start planning the move. The ingress2gateway tool can automate most of the conversion, and the Gateway API spec is stable and well-documented.

Next up in this series: cert-manager and automated TLS — because creating secrets by hand is so 2024.

Engineering Managers Should Make Engineers Better

Sat, 07 Mar 2026 00:00:00 +0000

During my regularly scheduled lurking on Hacker News, I came across Don’t Become an Engineering Manager, making the case that senior engineers should stay IC. The ladder is flattening, Staff pays better, tech is moving too fast to step away. Probably right, but it bugs me.

It treats the EM role like a career trade. A thing you optimize for or against. And when you look at it that way, sure, the math is shaky right now. But that skips the question I actually care about: what does a good engineering manager do for the engineers on their team?

The actual job

The best EMs I’ve worked with spend most of their time on the unglamorous stuff. Not “shielding the team” in the vague r/LinkedInLunatics way. The specific stuff. Why has the deploy pipeline been flaky for two weeks and nobody’s fixed it? Two teams are about to build the same thing, so get them in a room. Product is asking for something unreasonable and the engineers are too polite to say so.

None of that shows up on a career ladder. It barely shows up in performance reviews. But it’s the difference between a team that ships and a team that fights its own org.

Making people better

The part of the job I care about most doesn’t come up in any of these “should you become an EM” debates. Are the engineers on your team getting better? Not in the promotion-packet sense. Is the person who joined six months ago actually understanding the system now? Is the senior who’s great at heads-down execution starting to think about what happens after they ship?

That work is mostly invisible. It’s a one-on-one that looks like a casual conversation. It’s asking “what would you do?” instead of giving the answer. It’s putting someone slightly past their comfort zone and making sure they don’t eat it.

So should you become an EM?

If you care about people, and you want to make those people better engineers, then yes. Engineers need you.

On being 'wrong'...

Sat, 10 Jan 2026 00:00:00 +0000

Every bug I’ve ever shipped happened because the system did exactly what I told it to, not what I meant. That gap — between what I intended and what I actually built — is where most of the interesting lessons live.

Last week, on three separate occasions, sharp engineers held back their ideas in meetings because they were afraid of being wrong. I only heard those ideas afterward, in side conversations. Every single one of them would have moved the discussion forward, regardless of whether they turned out to be “right.”

That stuck with me.

~~Engineers~~ People who can say “I was wrong” out loud tend to work better together. They pull in feedback earlier, course-correct faster, and make fewer of the same mistakes twice. The teams I’ve seen do the best work aren’t the ones where everyone is right all the time — they’re the ones where nobody is penalized for being wrong.

We don’t get to correctness by avoiding mistakes. We get there by making them, catching them, and iterating. Most of the progress I’ve seen in my career has really just been a series of well-aimed misses.

So say the wrong thing. Propose the bad idea. You might be surprised how often “wrong” turns out to be the fastest way forward.

Let's Blog Again

Sat, 11 Apr 2020 00:00:00 +0000

Here’s the thing: most developers don’t blog. I haven’t in a long time. And that’s a shame — because it’s one of the easiest ways to level up your skills, stand out, and open doors you didn’t even know were there.

I’m not talking about some corporate PR thing or a perfectly polished tech journal. I mean your own little corner of the internet where you write about what you’ve learned, what’s broken, and how you fixed it.

You learn better when you explain things

The moment you try to write about a tricky bug or a new library, you realize all the gaps in your own understanding. Filling those gaps? That’s where the real learning happens.

It’s proof you know your stuff

Anyone can say “I work with Docker.” But if you’ve got posts about container networking quirks or real-world deployment tips, people can see your expertise instead of just taking your word for it.

You stand out

There are a lot of devs out there. Not many share their thinking publicly. Blogging makes you one of the people who contributes, not just consumes.

Your code gets a backstory

GitHub shows what you built. Your blog can show why you built it that way — the trade-offs, the failed attempts, and the weird edge cases. That’s the stuff that’s interesting.

Opportunities find you

A blog post you wrote in 2025 might get you a job lead in 2026. Seriously. The internet has a long memory.

You get better at communicating

Being able to explain complex ideas clearly is just as valuable as writing clean code. Blogging is practice you’ll feel in code reviews, design docs, and meetings.

You see your own growth

Reading your old posts is like looking at old code — a little cringe, a lot of “wow, I’ve improved.” It’s motivating.

You help the next person

That dumb error message that cost you two hours? Write it up. Someone else will Google it tomorrow or use ChatGPT in 6 months.