Premium features

STUNner's premium features are designed to help medium to large scale enterprises to deploy, operate and scale pools of STUN and TURN servers over Kubernetes. Below is a reference of the premium features currently available in STUNner.

User quota
STUN server mode
Deploying into a DaemonSet
Relay address discovery
TCP routes
Dual-stack TURN
Operator restarts and failover
Post-quantum TLS
TURN offload

User quota

Feature: UserQuota. Availability: member and enterprise tiers.

Once a client has obtained a valid TURN authentication credential, they can open any number of TURN connections by reusing the same credential. Since TURN credentials are available in plain text at clients (this is by WebRTC JavaScript API design), malicious clients can launch a Denial-of-Service (DoS) attack by creating excess of TURN allocations in quick succession. Unfortunately, even an ephemeral credential leaves a time window open for a DoS attack before it expires.

STUNner's UserQuota feature allows to set an upper limit on the number of simultaneous allocations that can be made with the same TURN credential. This feature is available in your tier if the UserQuota feature is enabled in the license status (recall, the status can be obtained using stunnerctl license).

Note that STUNner's quotas are per per-user-id. This means that if you obtain multiple different credentials for the same user-id (e.g., by using stunnerctl auth --username my-user) then the credentials map to the same quota: the TURN allocations authenticated with the same credential add up when imposing the quota. Also note that stale TURN allocations also count towards the quota. If a client fails to close unused TURN allocations (which TURN clients routinely do) then these stale allocations will be active until they time out (usually after 5 mins). This may prevent clients from re-connecting when an overly restrictive user quota is in effect. Quotas are per-dataplane-pod: if you scale STUNner then the quota multiplies by as much as there are dataplane pods.

Configure a user quota for a Gateway by setting the userQuota field in the GatewayConfig:

apiVersion: stunner.l7mp.io/v1
kind: GatewayConfig
metadata:
  name: stunner-gatewayconfig
spec:
  authType: ephemeral
  sharedSecret: my-shared-secret
  userQuota: 10

This will set the quota to 10. Setting the quota to zero means no quota (the default setting).

You can query the configured user quota using stunnerctl. Suppose you deployed a TURN/UDP gateway called udp-gateway in the stunner namespace. The current quota can be obtained as follows:

stunnerctl -n <gateway-namespace> status <gateway-name> -o jsonpath='{.admin.quota}'
10

Alternatively, You can query the gateway's quota from the corresponding dataplane config:

stunnerctl -n stunner config udp-gateway -o jsonpath='{.admin.user_quota}' -->
10

Once the number of allocations created for a user-id reach the configured quota, new allocations will be rejected with an error 486: Allocation Quota Exceeded status.

STUN server mode

Feature: STUNServer. Availability: member and enterprise tiers.

By default STUNner is configured to run as a TURN server. As TURN is an extension of the STUN protocol, this setting lets STUNner to serve plain STUN requests as well. Running a TURN server, however, comes at a potentially high cost, typically needing a high-bandwidth network connection and consuming pricey CPU resources. This is suboptimal for the case when STUNner is deployed as a pure STUN service, since malicious clients can consume excess server resources by creating phony TURN allocations.

In order to prevent the potential DoS attack vector, STUNner's TURN protocol engine can be completely turned off. This prohibits clients from making TURN allocations, but still guarantees that STUNner will serve STUN requests. Pure STUN server mode is available in your tier if the STUNServer feature is enabled in the license status (recall, the status can be obtained using stunnerctl license).

To switch a Gateway into STUN server mode, set STUNServer: true in the GatewayConfig:

apiVersion: stunner.l7mp.io/v1
kind: GatewayConfig
metadata:
  name: stunner-gatewayconfig
spec:
  STUNMode: true

This will disable STUNner's authentication engine, prohibiting clients from creating TURN allocations all together:

bin/stunnerctl -n <gateway-namespace> status <gateway-name> -o jsonpath='{.auth.type}'
none

Set STUNServer: false to re-enable the TURN protocol engine.

Deploying into a DaemonSet

Feature: DaemonSet. Availability: member and enterprise tiers.

By default, the TURN server pods that run the dataplane for STUNner gateways are deployed into a Kubernetes Deployment. This ensures that a configurable number of TURN servers are available per each Gateway. In certain cases, however, it may be desirable to deploy STUNner with a single dataplane pod per each Kubernetes node instead. This is crucial, for instance, when the STUNner dataplane is deployed in the host-network namespace to run a public TURN service, or when a Gateway is exposed with the service.spec.externalTrafficPolicy: Local configuration to implement direct server return to minimize clients' round-trip-time.

To configure STUNner to run a single STUNner dataplane pod per each node in the Kubernetes cluster, you can set spec.dataplaneResource to DaemonSet in the Dataplane resource corresponding to your Gateway. This will instruct STUNner to re-deploy the dataplane into a Kubernetes DaemonSet instead of a Deployment. DaemonSet mode is available in your tier if the DaemonSet feature is enabled in the license status (recall, the status can be obtained using stunnerctl license).

The below will set the dataplane for all gateways using the default Dataplane to use a DaemonSet.

apiVersion: stunner.l7mp.io/v1
kind: Dataplane
metadata:
  name: default
spec:
  dataplaneResource: DaemonSet
  ...

Set dataplaneResource: Deployment to return to the default deployment mode.

Relay address discovery

Feature: RelayAddressDiscovery. Availability: member and enterprise tiers.

STUNner was designed for a specific use case: ingest real-time media into a Kubernetes cluster and forward incoming connections to a pool of WebRTC media servers deployed into the same cluster. This, however, does not prevent you from leveraging STUNner for other purposes, like as a public TURN server, but this may need some tweaking.

Public TURN servers typically run on a public IP address, which makes it possible for both clients and peers to connect via the server. However, STUNner's TURN servers (the stunnerd pods) are by default deployed over private IPs. This is perfectly fine when peers, e.g., WebRTC media servers, are deployed into the same cluster (the pivotal use case for STUNner), but this pretty much makes it impossible to deploy STUNner as a public TURN server since peers will not be able to connect to the private IP of STUNner's TURN servers. (Note that "symmetric ICE mode" would still work but it may increase STUNner's resource consumption.)

Relay address discovery, when enabled, will configure STUNner's TURN servers with a public IP address, if available. The public IP is obtained from the Kubernetes node the TURN server runs at. Relay address discovery is available in your tier if the RelayAddressDiscovery feature is enabled in the license status (recall, the status can be obtained using stunnerctl license).

Below is the set of steps to enable relay address discovery:

Enable host-network mode.

Host-networking will re-deploy the STUNner dataplane to run in the network namespace of Kubernetes nodes, so that it will have access to the node's public IP address (if any). You can enable host-networking by configuring hostNetwork: true in the Dataplane spec:
```
apiVersion: stunner.l7mp.io/v1
kind: Dataplane
metadata:
  name: default
spec:
  dataplaneResource: DaemonSet
  hostNetwork: true
```
Only enable host-networking with the DaemonSet feature also enabled, otherwise you will not be able to scale your TURN server pool.
Enable relay address discovery.

Relay address discovery mode can be switched on by adding the stunner.l7mp.io/enable-relay-address-discovery: "true" annotation on a STUNner Gateway:
```
kubectl annotate --overwrite gateway <your-stunner-gateway> stunner.l7mp.io/enable-relay-address-discovery="true"
```
Check whether STUNner has successfully discovered the node's public IP.

As usual, the stunnerctl tool comes in handy. The below will load the dataplane configuration for the gateway <gateway-namespace>/<gateway-name> with respect to the node <node-name> and print the relay address of each TURN listener:
```
stunnerctl -n <gateway-namespace> config <gateway-name> --node=<node-name> -o jsonpath='{.listeners[*].address}'
<relay-address>
...
```
The <relay-address> above should be the node's public IP for <node-name>. You can also request the status directly from the dataplane pods, but in this case parsing the output requires a bit of getting used to:
```
stunnerctl -n <gateway-namespace> status <gateway-name>
<gateway-namespace>/<gateway-pod>:
admin: ...
auth: ...
listeners: <listener-name>:{turn://<relay-address>:<port>?transport=TURN-UDP...}
clusters: ...
```
Note that depending on your Kubernetes provider's platform your nodes may run without a public IP, or host-networking may not be available at all. Symmetric ICE mode is still be usable as a fallback in such cases.

TCP routes

Feature: TCPRoute. Availability: member and enterprise tiers.

STUNner can route clients to backends over TCP over TURN. The TCPRoute resource is the TCP counterpart of the UDPRoute: it admits the same kinds of backends, but the relay leg towards the backend is a TCP connection.

apiVersion: stunner.l7mp.io/v1
kind: TCPRoute
metadata:
  name: media-plane-route
  namespace: stunner
spec:
  parentRefs:
    - name: tcp-gateway
  rules:
    - backendRefs:
        - name: media-server-pool
          namespace: media-plane

See the TCPRoute reference for the full spec. There is no protocol conversion anywhere: the client's allocation fixes the protocol of the relay leg, so a TCP backend is reachable only through an RFC 6062 TCP allocation. Browsers do not implement RFC 6062, so TCP backends are not reachable from a browser's relay candidates.

TCP routing is available in your tier if the TCPRoute feature is enabled in the license status (recall, the status can be obtained using stunnerctl license). Without the feature the operator still accepts your TCPRoutes and maintains their status, but it renders no dataplane cluster for them and reports the reason in the route's ResolvedRefs status condition:

kubectl get tcproute media-plane-route -o jsonpath='{.status.parents[0].conditions}'

[..., {"type":"ResolvedRefs","status":"False","reason":"UnsupportedValue",
       "message":"route kind not available in the current license tier"}]

Dual-stack TURN

Feature: DualStack. Availability: member and enterprise tiers.

A TURN server that is reachable over both IPv4 and IPv6 must hand each client a relayed transport address in the client's own address family: an IPv6 client that ingresses over the IPv6 VIP needs an IPv6 relay to reach IPv6 peers, and analogously for IPv4. Kubernetes preserves the address family end-to-end (a dual-stack Service gets one VIP and one EndpointSlice per family and Kubernetes never cross-connects families), so the missing piece is STUNner respecting the requested address family.

The DualStack feature wires this up:

the STUNner dataplane pods receive their pod IPs for both families and advertise a relay address matching each client's family;
the public addresses of a Gateway are advertised per family, so the ICE server list your clients receive contains an entry per family;
the LoadBalancer Service is configured with a per-family VIP.

Note that the address family is chosen by the client's ICE agent when it picks which of the advertised ICE server URLs to contact, and the family of that connection then selects the family of the relay address.

Dual-stack TURN is available in your tier if the DualStack feature is enabled in the license status (recall, the status can be obtained using stunnerctl license). Without the feature STUNner falls back to single-stack mode.

High-availability operator

Feature: HAOperator. Availability: member and enterprise tiers.

In a production STUNner deployment the gateway operator, which is responsible for providing the dataplane configuration over the config discovery service (CDS), may unavoidably go offline. Sometimes the node the operator is scheduled at is removed, sometimes it is a planned restart, sometimes it may even be a genuine operator bug.

A solution to this is to run multiple operator pods side by side: enabling leader election (--leader-elect) makes sure only one operator instance is active (i.e., updates Kubernetes and emits dataplane configurations) at a time, the rest are the hot standby. Once the leader operator terminates, a new leader is selected that takes over the responsibilities of the former leader. By default, however, this involves the restarting of the running dataplane pods and the disconnection of all active clients.

When using the high-availability mode, operator failovers become seamless: the new replica takes over the control of the running dataplane pods and makes sure there is no stale config emitted along the way. If only a single operator replica is run, the high-availability mode makes sure the restarted operator will not cause a dataplane churn. Note that the finalizer (--enable-finalizer) mode is incompatible with leader election: a replica losing the lease must not tear down resources the new leader keeps serving.

High-availability operator mode is available in your tier if the HAOperator feature is enabled in the license status (recall, the status can be obtained using stunnerctl license). Without the feature the operator falls back to rolling the dataplane as before.

Post-quantum TLS

Feature: PQC. Availability: member and enterprise tiers.

A TURN session over a TLS transport carries everything between the client and the server inside the TLS connection. Standard TLS uses a weak encryption that a large enough quantum computer could break, exposing anything recorded today to be decrypted later. The PQC feature allows for STUNner to upgrade to a stronger "post-quantum encryption" scheme, which is protected against quantum computers.

The selected PQC TLS mode can be chosen by setting the stunner.l7mp.io/pqc-mode annotation per Gateway. The selected mode setting applies to all TURN-TLS listeners of a Gateway. There are three modes: default (or a missing annotation) serves the standard "weak" TLS settings, preferred prefers a post-quantum encryption but still admits clients that offer only a classical one, and enforced serves only a post-quantum safe encryption, refusing clients that don't support it. The policy is TLS-only. Post-quantum TLS is available in your tier if the PQC feature is enabled in the license status (recall, the status can be obtained using stunnerctl license).

Below is the set of steps to enable post-quantum TLS:

Select the mode.

The mode is selected per Gateway by adding the stunner.l7mp.io/pqc-mode annotation, with the value preferred or enforced:
```
kubectl annotate --overwrite gateway <your-stunner-gateway> stunner.l7mp.io/pqc-mode="enforced"
```
The operator renders the mode into the pqc_mode field of every TURN-TLS listener of the Gateway; listeners of any other protocol are left alone. Removing the annotation, or setting it to default, restarts the listeners with the standard TLS settings.
Check whether the mode reached the dataplane.

The stunnerctl tool will load the dataplane configuration for the gateway <gateway-namespace>/<gateway-name> and print the mode of each listener (an empty value is the default mode):
```
stunnerctl -n <gateway-namespace> config <gateway-name> -o jsonpath='{.listeners[*].pqc_mode}'
enforced
...
```

TURN offload

Feature: TURNOffload. Availability: only the enterprise tier.

User plane TURN message processing may be costly. To cut down CPU usage and latency, STUNner can offload TURN message processing to one of its Linux/eBPF-based kernel packet processing engines. The offload engines support TURN channel processing for UDP, and provide massive bandwidth, delay, and jitter performance boost and can cut down CPU usage by several orders of magnitude. TURN acceleration is available in your tier if the TURNOffload feature is enabled in the license status (recall, the status can be obtained using stunnerctl license).

STUNner's eBPF offload requires Kubernetes nodes running Linux and elevated privilege access to interact with the eBPF/tc (TC) or eBPF/XDP (XDP) kernel framework. Both provide outstanding performance: TC is supported in most Kubernetes environments (e.g., public clouds), while XDP is faster but it is typically limited to bare metal clusters. For more info on the TURN offload implementation, see the pion/turn PR.

To use the TURN offload feature of STUNner, set the spec.offloadEngine in the Dataplane custom resource: TC means eBPF/TC, XDP is eBPF/XDP, None falls back to user-space TURN processing, and Auto will let STUNner to pick the best offload engine for your platform. You can also manually configure the network interfaces on which STUNner will enable TURN offload via the spec.offloadInterfaces in the Dataplane spec. This parameter assumes a list of network interface names and an empty list means to enable offload on all interfaces (this is the default). To use eBPF offload, you must also enable elevated rights in your STUNner pods. To achieve this, edit the spec.containerSecurityContext field and add the necessary NET_ADMIN, SYS_ADMIN, SYS_MODULE capabilities.

Configuration

The below will set the dataplane for all gateways using the default Dataplane to use the TURN offload on all available network interfaces and select the optimal offload mode.

apiVersion: stunner.l7mp.io/v1
kind: Dataplane
metadata:
  name: default
spec:
  containerSecurityContext:
    capabilities:
      add:
      - NET_ADMIN
      - SYS_ADMIN
      - SYS_MODULE
  offloadEngine: Auto

Testing

The simplest way to test whether eBPF offload was successfully enabled is to deploy a simple STUNner tutorial (like the iperf-test example) and watch for the offload statistics in the output or stunnerctl status:

stunnerctl -n <gateway-namespace> status <gateway-name>
<gateway-namespace>/<gateway-name>:
	admin: ...
	static-auth: ...
	listeners: <listener-name>:{...},offload(rx/tx): 1152/345 pkts 168192/19834 bytes
	clusters: ...
	allocs:1/status=READY

You may also check the dataplane logs. The below was logged by stunnerd while running stunnerctl icetest --offload-mode=TC --force-cleanup:

stunnerd INFO: Starting stunnerd id "icetest/icetest-udp-gateway", STUNner version ... 
...
offload INFO: TC init complete on interfaces: [lo eth0]
...
stunner INFO: New dataplane status: ... admin:{...,license-info={tier=enterprise,unlocked-features=[TURNOffload,UserQuota,DaemonSet,STUNServer,RelayAddressDiscovery]...
...
offload INFO: Created TC offload udp:local:[::]:3478-remote:...:...-chan:16384->udp:local:....:...-remote:...:...-chan:0
...

You can also directly check the eBPF maps created by the offload engine on the host using bpftool.

Limitations

Effective TURN protocol offload requires some low-level tweaking of the underlying OS kernel. As such, some Kubernetes deployments may support only a limited TURN offload functionality, or even no TURN acceleration at all:

The offload engine relies on the OS to support eBPF TC/XDP. Currently this works on GNU/Linux hosts only.
Loading the eBPF program requires elevated admin privileges that might be not available on arbitrary Kubernetes clusters. Make sure to add the required capabilities (at least NET_ADMIN, SYS_ADMIN and SYS_MODULE) to the dataplane pod security context (see above).
Offload applies only to a leg between a TURN client and a TURN server, where plaintext ChannelData arrives on one side and raw traffic leaves on the other: a client of a TURN/UDP listener relaying to a UDPRoute backend, and a UDP client of the tunnel mode tunnelling over TURN/UDP. TURN over TCP, TLS or DTLS, TCPRoute backends and TURN relay chaining stay in user space.
Currently TURN offload only supports UDP TURN channels. Implementing TURN/TCP acceleration and offloading TURN send indications are on the TODO list, reach out to us if you need these features.
TURN/XDP offload is disabled for host-local redirects (except the lo interface). Use the TURN/TC engine when host-local redirect is important, like accelerated symmetric ICE mode deployment.

Navigation Menu

Uh oh!

Premium features

Table of Contents

User quota

STUN server mode

Deploying into a DaemonSet

Relay address discovery

TCP routes

Dual-stack TURN

High-availability operator

Post-quantum TLS

TURN offload

Configuration

Testing

Limitations

Uh oh!

FilesExpand file tree

PREMIUM_REFERENCE.md

Latest commit

History

PREMIUM_REFERENCE.md

File metadata and controls

Premium features

Table of Contents

User quota

STUN server mode

Deploying into a DaemonSet

Relay address discovery

TCP routes

Dual-stack TURN

High-availability operator

Post-quantum TLS

TURN offload

Configuration

Testing

Limitations