oomkilling myself because of kubernetes event exporter — may board

“neko ﾟ･✿ヾ╲(｡◕‿◕｡)╱✿･ﾟ”

— TheoryOfNekomata

“bro what happened to claude code”

— johnislarry

“ah sarado”

— lxsmnsyc

multi-domain routing from a single next.js server

what i learned serving two different sites from one Next.js codebase using middleware hostname detection, rewrites, and a Caddy dev proxy.

wfd · 2026-02-28T02:01:00 · 6 min

replacing tailscale with headscale

tailscale's free tier is generous but i'd rather own the control plane. headscale is the self-hosted replacement and it runs fine on proxmox.

wfd · 2026-02-28T02:01:00 · 5 min

picking an analytics provider

i want to know if anyone's actually reading this. not to optimize anything, just to know.

wfd · 2026-02-11T08:50:00 · 3 min

five certifications from a third world country

CKA, AWS SAA, Terraform Associate, CKAD, RHCSA with no degree. It's expensive as shit but I'm doing it anyway.

wfd · 2026-02-07T09:37:00 · 5 min

oomkilling myself because of kubernetes event exporter

I wanted Discord notifications when pods crashed. What I got was weeks of debugging memory leaks in a tool that loads every event into RAM before deciding to ignore it.

wfd · 2026-02-01T18:42:00 · 4 min

fuwasoft project screenshot

fuwasoft

software consultancy concept site

graveyardnexttypescript

git.eden-emu.dev

3900: sampled descriptor indices as nonuniform

from fixing tomodachi life's black grass in eden

git.eden-emu.dev

3898: dynamic texture descriptor strides

from fixing tomodachi life's black grass in eden

docs.vulkan.org

vulkan descriptor sets

from fixing tomodachi life's black grass in eden

docs.vulkan.org

VkPhysicalDeviceLimits

from fixing tomodachi life's black grass in eden

docs.vulkan.org

VkDescriptorSetLayoutBindingFlagsCreateInfo

from fixing tomodachi life's black grass in eden

github.com

Headscale GitHub

from replacing tailscale with headscale

tailscale.com

DERP

from replacing tailscale with headscale

tailscale.com

CGNAT range

from replacing tailscale with headscale

tailscale.com

subnet routing

from replacing tailscale with headscale

registry.terraform.io

awlsring/headscale Terraform provider

from replacing tailscale with headscale

tailscale.com

preauth key

from replacing tailscale with headscale

xda-developers.com

this XDA article

from replacing tailscale with headscale

headscale.net

Headscale docs

from replacing tailscale with headscale

github.com

Headscale config reference

from replacing tailscale with headscale

fosdem.org

FOSDEM 2026: Headscale & Tailscale

from replacing tailscale with headscale

vercel.com

vercel analytics

from picking an analytics provider

simpleanalytics.com

simple analytics

from picking an analytics provider

matomo.org

matomo

from picking an analytics provider

github.com

35.2k

from picking an analytics provider

plausible.io

plausible

from picking an analytics provider

ackee.electerious.com

ackee

from picking an analytics provider

github.com

4.3k

from picking an analytics provider

cncf.io

CKA

from five certifications from a third world country

aws.amazon.com

$150

from five certifications from a third world country

developer.hashicorp.com

$70.50

from five certifications from a third world country

cncf.io

CKAD

from five certifications from a third world country

redhat.com

$450

from five certifications from a third world country

nodeflair.com

₱65,000/mo

from five certifications from a third world country

nodeflair.com

₱80,000/mo

from five certifications from a third world country

training.linuxfoundation.org

$445

from five certifications from a third world country

en.wikiquote.org

Sun Tzu, The Art of War

from five certifications from a third world country

docs.linuxfoundation.org

performancebased lab

from five certifications from a third world country

kodekloud.com

KodeKloud

from five certifications from a third world country

pkg.go.dev

Kubernetes informers

from oomkilling myself because of kubernetes event exporter

stackoverflow.com

one Stack Overflow user noted

from oomkilling myself because of kubernetes event exporter

github.com

GitHub issue 232

from oomkilling myself because of kubernetes event exporter

docs.fluentbit.io

chunkbased processing and filesystem buffering

from oomkilling myself because of kubernetes event exporter

github.com

original Opsgenie repository

from oomkilling myself because of kubernetes event exporter

github.com

resmoio/kuberneteseventexporter

from oomkilling myself because of kubernetes event exporter

github.com

GitHub issue 237

from oomkilling myself because of kubernetes event exporter

docs.fluentbit.io

kubernetesevents input plugin

from oomkilling myself because of kubernetes event exporter

github.com

github.com/schmayterling/kubesee

from oomkilling myself because of kubernetes event exporter

wfd / Feb 1, 2026 / 4 min

oomkilling myself because of kubernetes event exporter

Legacy This WFD was imported from old blog posts scattered across my files. It may still be relevant or it may not. Kept for the record.

I run a hobby Kubernetes cluster. A few namespaces, some apps, a CI system with ephemeral runners. All I wanted was Discord notifications when pods crashed.

kubernetes-event-exporter seemed like the obvious choice. It watches Kubernetes events and forwards them to various destinations via webhooks. Over 1,000 GitHub stars, actively forked. I deployed the Bitnami Helm chart, pointed it at Discord, and called it a day.

Then I spent weeks figuring out why it kept running out of memory.

how kubernetes-event-exporter works

The tool uses Kubernetes informers to watch the Event resource. When you start the exporter, the informer syncs all existing events from the API server into memory, then watches for new ones via a persistent connection.

Each event goes through a routing tree you define in YAML. You can drop events based on namespace, type, reason, labels. Events that pass the filters get sent to receivers: webhooks, Elasticsearch, files, whatever.

flowchart TD
    A[K8s API Server] -->|"LIST all events (initial sync)"| B[Informer]
    A -->|"WATCH new events"| B
    B -->|"Every event loaded"| C[In-Memory Store]
    C -->|"Fetch involved object metadata"| D[API Enrichment]
    D --> E{Drop Rules}
    E -->|"matched"| F[Discarded]
    E -->|"not matched"| G{Match Rules}
    G --> H[Receivers]
    
    style C fill:#ff6b6b,color:#fff
    style D fill:#ff6b6b,color:#fff
    style F fill:#888,color:#fff

The red boxes are where memory gets consumed. Drop rules come after the event is already loaded and enriched. The informer has no idea your routing config exists. It syncs everything.

drop rules don't reduce memory

This is the part the docs don't mention.

When you write:

route:
  drop:
    - namespace: ci-runners
    - namespace: kube-system
    - type: "Normal"

You might think you're telling the exporter to ignore those events. You're not. You're telling it to load them into memory, enrich them with API calls, and then throw them away.

The informer callback fires for every event. The exporter fetches the involved object's metadata (labels, annotations, owner references) from the API server. Then it evaluates your routing tree. Then it maybe drops the event. But by then, the memory is already allocated and the API call already happened.

This isn't unique to kubernetes-event-exporter. As one Stack Overflow user noted, informers watching 15k Pods can consume over 1GB of memory because they cache all watched objects before any filtering occurs.

My CI system generates thousands of events per hour from ephemeral runner pods. Every pod create, every container start, every scheduling decision gets loaded and enriched, even the ones I explicitly said to drop.

Increasing memory limits to 1Gi delayed the OOM by a few hours. Not a fix.

the namespace field only takes one string

I tried a different approach: only watch namespaces I care about.

The config has a top-level namespace field that restricts the informer's scope, but it only accepts a single string:

# What I wanted:
namespace: "app-*"  # Doesn't work

# What I tried:
namespace: "app-staging|app-prod"  # Doesn't work

# What actually works:
namespace: "app-staging"  # One namespace only

GitHub issue #232 asks for wildcard or multi-namespace support. Still open with no replies.

If you want to watch multiple namespaces while excluding noisy infrastructure namespaces, you have two options: watch everything and deal with memory issues, or deploy separate exporter instances for each namespace you care about.

no backpressure handling

The informer fires callbacks as fast as events arrive. If your receiver is slow (Elasticsearch indexing, webhook timeouts, network issues), events queue in memory.

There's no filesystem buffer, no way to pause the informer, no retry limits that shed load. Events accumulate until the process hits its memory limit and gets killed.

Fluent Bit handles similar problems with chunk-based processing and filesystem buffering. When memory limits are reached, chunks get written to disk instead of accumulating in RAM. kubernetes-event-exporter has none of that. It assumes receivers can always keep up.

the project is effectively unmaintained

The original Opsgenie repository was archived August 31, 2022. A community fork exists at resmoio/kubernetes-event-exporter with the same codebase and architecture. The last release (v1.7) was February 2024.

The Bitnami Helm chart that most people use for deployment is being discontinued. From GitHub issue #237:

"Bitnami helm chart is not going to be supported anymore after August 28th"

workarounds

Increase memory limits. 512Mi to 1Gi depending on cluster size. Buys time but doesn't fix the underlying problem.

Dedicated instances per namespace. Deploy one exporter per namespace you care about, each with namespace: <name> set. Memory stays bounded because the informer only watches one namespace. Downside: N deployments to manage.

Use Fluent Bit instead. Fluent Bit's kubernetes_events input plugin can capture events and forward them with proper backpressure handling. The config is more complex and it's a different mental model (log pipelines vs event routing), but it scales better.

Accept event loss. Set maxEventAgeSeconds low (5-10 seconds) and accept that events will be discarded during spikes. Better than OOM crashes.

kubesee

I started writing a replacement in Elixir: github.com/schmayterling/kubesee

The idea is to use the same YAML config format but fix the architectural problems. Each receiver runs in its own Erlang process with its own message queue. Slow receivers don't block fast ones. The supervision tree restarts failed components without taking down the whole process.

It's early and most receivers are still unimplemented. If you need something production-ready today, the Resmo fork with workarounds is your best bet. But if these architectural issues bother you as much as they bothered me, maybe take a look.