Kubernetes Watches by Example

Published in

codeburst

3 min readNov 22, 2020

A simple example of using Kubernetes watches.

Recently I ran into a situation where I needed to maintain a list of endpoints — specifically the IP addresses of a Kubernetes service. Because the service was backed by a deployment with a horizontal pod autoscaler, the list of endpoints (pods) was updated regularly and unpredictably.

One approach to maintaining this list of endpoints is to regularly — say every minute — query the endpoints URL. To illustrate this, we start a kubectl proxy on our workstation with:

$ kubectl proxy --port 8080

And then we can use a browser on our workstation to query the URL, here a deployment and service are already running in the my-namespace namespace.

$ curl http://localhost:8080/api/v1/namespaces/my-namespace/endpoints

The output:

A better approach, however, is to use a watch URL.

To enable clients to build a model of the current state of a cluster, all Kubernetes object resource types are required to support consistent lists and an incremental change notification feed called a watch. Every Kubernetes object has a resourceVersion field representing the version of that resource as stored in the underlying database. When retrieving a collection of resources (either namespace or cluster scoped), the response from the server will contain a resourceVersion value that can be used to initiate a watch against the server. The server will return all changes (creates, deletes, and updates) that occur after the supplied resourceVersion. This allows a client to fetch the current state and then watch for changes without missing any updates. If the client watch is disconnected they can restart a new watch from the last returned resourceVersion, or perform a new collection request and begin again. See Resource Version Semantics for more detail.

— Kubernetes — Kubernetes API Concepts

Let’s explore using a Kubernetes watch by monitoring the endpoints of the service described in the previous example. As we did in the earlier example, we first:

Start a kubectl proxy
Query the endpoints URL. Observe the resourceVersion of the EndpointsList

We then execute, supplying the observed resourceVersion:

$ curl http://localhost:8080/api/v1/watch/namespaces/my-namespace/endpoints?resourceVersion=[REPLACE]

Things to observe:

Because we supplied an up-to-date resourceVersion value, the output is initially empty
The curl command does not immediately return; this is an example of a long-poll

Please note: The section on watches in Kubernetes API Concepts uses a different URL for a watch; they simply append the query parameters watch=1&resourceVersion=[REPLACE] to the original endpoints URL. When I tried this, I could not get the watch to respect the resourceVersion value I supplied, i.e., it initially returns an ADD event for each existing endpoint.

In another window, we scale the deployment:

$ kubectl scale --replicas=2 deployment/nginx -n my-namespace

The long-poll curl command outputs the following (and continues to run):

We could write code that uses these URLs to monitor the endpoints. We, however, have to also handle the following behavior.

A given Kubernetes server will only preserve a historical list of changes for a limited time. Clusters using etcd3 preserve changes in the last 5 minutes by default. When the requested watch operations fail because the historical version of that resource is not available, clients must handle the case by recognizing the status code 410 Gone, clearing their local cache, performing a list operation, and starting the watch from the resourceVersion returned by that new list operation.

— Kubernetes — Kubernetes API Concepts

An easier solution, however, is to use a Kubernetes library that implements all this for us. For example, the following code uses the Go client-go library.

Running the aforementioned code when scaling the Nginx deployment to two replicas outputs the following: