Stop working after no change.

[joaogbcravo]

Hello!

Not sure this is a bug or just misconfiguration, therefore I open a discussion.

I have stunner-gateway-operator deployed using helm chart version 1.1.0.
Everything works, and suddenly, after some days, maybe weeks, livekit streaming doesn't work anymore.

On the stunner gateway I see

"194.230.147.131:4204" to peer 10.200.115.242: no route to endpoint
08:58:29.215436 handlers.go:98: auth INFO: permission denied on listener "my-namespace/livekit-stunner-gateway/stunner-udp" for client "194.230.147.131:48582" to peer 3.125.117.144: no route to endpoint
08:58:29.215473 handlers.go:98: auth INFO: permission denied on listener "my-namespace/livekit-stunner-gateway/stunner-udp" for client "194.230.147.131:4204" to peer 3.125.117.144: no route to endpoint
09:22:08.520517 server.go:170: turn ERROR: Failed to close conn: tls: failed to send closeNotify alert (but connection was closed anyway): write tcp 10.200.218.3:443->10.200.0.35:13291: write: broken pipe (suppressed 7 log events)
09:22:11.572773 server.go:170: turn ERROR: Failed to close conn: tls: failed to send closeNotify alert (but connection was closed anyway): write tcp 10.200.218.3:443->10.200.0.35:15688: write: broken pipe

On the operator

`2026-01-19T13:28:38.065239215Z INFO udproute-controller Validating backend {"udproute": "not found"}`

I manage to understand that, if I restart the operator, then the gateway is restarted as well, and everything starts working.

AI suggests it can be some network blip and suggested me that I should increase the replicas of the operator (before was just 1 - I did that, not sure if it will fix the problem.

Any pointers of the root cause?

[rg0now]

This usually happens under resource stress: the operator gets bogged down (e.g., when it re-renders the whole dataplane config periodically) and fails CDS pings. Increase the CPU limit: this makes sure the operator will not run out of CPU even under load spikes. You can also play with throttling datplane updates, e.g., setting --throttle-timeout=2s on the gateway operator command line usually makes it more robust (at the cost of delaying dataplane updates by at most 2s).

[joaogbcravo]

Thank you for the fast reply.

We do not have CPU limits for the operator :) But i will take a look, maybe it's hitting on the limit of the node CPU itself (but I doubt it)

I will also go with that throttle-timeout. thanks for the advice!
What is the impact of dataplane update?

Does it make any sense, or is there any advantage to have multiple replicas on the operator?

[rg0now]

The operator writes a new TURN server (AKA "dataplane") config every time you add/remove/modify a Gateway or a UDPRoute or add/remove pods to/from backend services, etc. This can take a lot of resources in large deployments, hence the high CPU limit.

You can scale the operator to multiple pods, but at any instance of time only one pod (the "leader") will be active, the rest will just sit there doing nothing. If the leader fails, the remaining pods automatically elect a new leader that takes over.

[joaogbcravo]

@rg0now Thanks for the explanation.

I don't see it being a CPU problem, do you?

Also, having two pods (master/slave) didn't help on the problem.

Next thing to try is --throttle-timeout=2s

[rg0now]

Nothing. We've seen a couple of instabilities with hosted Kubernetes platforms in production so I guess AI was right: this was "some network blip". That being said, the operator should never get stuck even under instabilities so if you have more info don't hold it back

[joaogbcravo]

At the moment, I don't know what more info is relevant for you to help me. Happy to share the info if you tell me what should I look for.

The problem with "some network blip" is that it happens too often, I mean, once a month at least.