Kubernetes, gRPC Services, and Probes by Example

Setting up Kubernetes probes for workloads providing gRPC services.

The working example described in this article is available for download.

You have a workload running on Kubernetes that provides gRPC services and are looking to add both a liveness and readiness probe to it. First, we need to remind ourselves what they are and more importantly what they are used for.

Many applications running for long periods of time eventually transition to broken states, and cannot recover except by being restarted. Kubernetes provides liveness probes to detect and remedy such situations.

— Kubernetes — Configure Liveness, Readiness and Startup Probes

Sometimes, applications are temporarily unable to serve traffic. For example, an application might need to load large data or configuration files during startup, or depend on external services after startup. In such cases, you don’t want to kill the application, but you don’t want to send it requests either. Kubernetes provides readiness probes to detect and mitigate these situations. A pod with containers reporting that they are not ready does not receive traffic through Kubernetes Services.

— Kubernetes — Configure Liveness, Readiness and Startup Probes

Thinking about this, a simple liveness check would be to validate that the workload can respond to any gRPC service method call. A readiness check, presumably exposed as a gRPC service method, would be need to crafted more specifically to reflect the availability of the workload’s external dependencies.

At first glance, we might think that we can use the same gRPC service method for both the liveness and readiness probe. But, this generally is not a good idea, e.g., if a gRPC service method is designed to indicate the workload is loading large configuration files during startup and that method is used as a liveness probe, the container will be continually restarted; essentially preventing the workload (or pod) from ever reaching a ready state.

note: There are many articles that mistakenly suggest that one should use the same gRPC service method for both the liveness and readiness probes.

Our solution will be to create two gRPC service methods; one for the liveness probe and one for the readiness probe.

We, however, have another problem.

Kubernetes does not support gRPC health checks natively. This leaves the gRPC developers with the following three approaches when they deploy to Kubernetes:

httpGet probe: Cannot be natively used with gRPC. You need to refactor your app to serve both gRPC and HTTP/1.1 protocols (on different port numbers).

tcpSocket probe: Opening a socket to gRPC server is not meaningful, since it cannot read the response body.

exec probe: This invokes a program in a container’s ecosystem periodically. In the case of gRPC, this means you implement a health RPC yourself, then write and ship a client tool with your container.

— Kubernetes — Health checking gRPC servers on Kubernetes

Luckily, there is already a such a client tool.

The grpc_health_probe utility allows you to query health of gRPC services that expose service their status through the gRPC Health Checking Protocol.

This command-line utility makes a RPC to /grpc.health.v1.Health/Check. If it responds with a SERVING status, the grpc_health_probe will exit with success, otherwise it will exit with a non-zero exit code (documented below).

— grpc-ecosystem — grpc-health-probe

So, all we need to do is implement the /grpc.health.v1.Health/Check method and use the grpc_health_probe utility as an exec probe for our readiness probe. But what about our liveness probe?

For this, there is another client tool at our disposal.

grpcurl is a command-line tool that lets you interact with gRPC servers. It’s basically curl for gRPC servers.

The main purpose for this tool is to invoke RPC methods on a gRPC server from the command-line. gRPC servers use a binary encoding on the wire (protocol buffers, or “protobufs” for short). So they are basically impossible to interact with using regular curl (and older versions of curl that do not support HTTP/2 are of course non-starters). This program accepts messages using JSON encoding, which is much more friendly for both humans and scripts.

— fullstorydev — grpcurl

For the liveness probe, we then need to implement another gRPC method, e.g, /ping.Pinger/Ping, and use grpcurl as an exec probe for our liveness probe.

To make this all work, we need the container delivering our workload to also have both the grpc_health_probe and grpcurl utilities installed. For that we use a Docker multi-stage build, e.g., with a Dockerfile:

With the implemented gRPC methods and the utilities installed, we can now create the liveness and readiness probes for our workload.

Wrap Up

That is about it; hope you found it useful.