Traffic Director by Example: Part 2

John Tucker

Published in

codeburst

8 min readMar 14, 2021

Using Traffic Director with Google Kubernetes Engine (GKE).

This article is part of the series that starts with Traffic Director by Example: Part 1.

Sidebar into Automatic Envoy Deployments

In the last article, we manually installed the Envoy service proxy for the client on a GCE VM instance. Below, we will manually install it on a GKE pod. At the same time, Traffic Director supports installing the Envoy service proxy automatically for both GCE VM instances and GKE pods.

At the same time, there are clear advantages to the automated installs.

When you use automated Envoy deployment with Compute Engine VMs, the Envoy version installed is one that we have validated to work with Traffic Director. When a new VM is created using the instance template, the VM receives the latest version that we have validated. If you have a long-running VM, you can use a rolling update to replace your existing VMs and pick up the latest version.
When you use the Envoy sidecar injector with GKE, the injector is configured to use a recent version of Envoy that we have validated to work with Traffic Director. When a sidecar is injected alongside your workload Pod, it receives this version of Envoy. If you want to pick up a more recent version of Envoy, update the Envoy sidecar injector.

— Google Cloud — Preparing for Traffic Director setup

So, why did we do the manual installation?

It turns out that the feature of configuring the service proxy to only intercept traffic to the VIP service CIDR block 10.0.0.0/16 is not available (or not documented) for the automated installation. This is a critical problem assuming that the workload needs to communicate service other than the ones that we provide ourselves.

Client on GKE Pod

We first need to create a GKE cluster to create our client GKE pod on.

Please note: These activities loosely mirror the Google Cloud provided documentation, Traffic Director setup for Google Kubernetes Engine Pods with manual Envoy injection.

The GKE cluster does not require any special configuration other than allowing the service proxies to connect to Traffic Director by allowing the Compute Engine default service account (with the role we assigned in the previous article) to have full access to all Cloud APIs.

Please note: This assumes that we are not using GKE Workload Identity. In that case, we would need to add the Compute Network Viewer (compute.networkViewer) role to the GCP service account backing the k8s service account associated with the client GKE pod.

We specifically use the menu, Kubernetes Engine > Clusters, and press the Create button. We choose the Standard option.

From the menu, default pool > Security, we select Access scopes > Allow full access to all Cloud APIs.

Press the Create button to create the cluster.

One subtle problem is that the cluster creation process creates two additional subnets for the cluster’s pods and services; these subnets, however, and not configured in the VPC Firewall configuration. We can observe these subnets from the menu VPC network > VPC networks.

The larger subnet, 10.4.0.0/14 here, is the pod subnet that we need to allow traffic to freely flow in and out of the rest of the subnets. We accomplish this from the menu VPC Firewall > Firewall and pressing the Create Firewall Rule button. We configure the Firewall Rule with:

Name: fw-allow-pods
Targets: All instances in the network
Source IP ranges: 10.4.0.0/14
Protocols and ports: Allow all

and then press the Create button to create the Firewall Rule.

After installing the gcloud and kubectl CLIs onto our workstation, we can authenticate kubectl to our cluster.

We create the following client GKE pod configuration file on our workstation.

Things to observe:

This configuration is based on the example provided in the instructions Traffic Director setup for Google Kubernetes Engine Pods with manual Envoy injection
The client is a simple busybox container; the bulk of the pod configuration is for the service proxy
Similar to the GCE VM client configuration, here we updated the service proxy to only intercept traffic to the VIP service CIDR block 10.0.0.0/16

We now create the client GKE pod.

$ kubectl apply -f td-demo-gke-client-pod.yaml 
pod/td-demo-gke-client created

We then login to the pod.

$ kubectl exec -it td-demo-gke-client -- /bin/sh

We confirm that the client’s traffic to the VIP 10.0.0.1 on port 80 is intercepted and sent to one of the the GCE VMs backing the Traffic Director service.

# wget -O - http://10.0.0.1
Connecting to 10.0.0.1 (10.0.0.1:80)
writing to stdout
<!doctype html><html><body><h1>td-demo-hello-world-mig-10cw</h1></body></html>
-                    100% |********************************|    79  0:00:00 ETA
written to stdout

We can also use the DNS name hello-world-gce.example.private that we setup in the previous article.

# wget -O - http://hello-world-gce.example.private
Connecting to hello-world-gce.example.private (10.0.0.1:80)
writing to stdout
<!doctype html><html><body><h1>td-demo-hello-world-mig-l2c6</h1></body></html>
-                    100% |********************************|    79  0:00:00 ETA
written to stdout

We can also observe that traffic not destined to a VIP in the service CIDR block, 10.0.0.0/16, bypasses the service proxy.

# wget -O - http://www.google.com
Connecting to www.google.com (64.233.191.106:80)
writing to stdout
<!doctype html>
[obmitted]

Service Backed by GKE Pods

Here we will deploy a Traffic Director service backed by GKE pods.

We first create a Kubernetes services by creating and applying (using kubectl) the following configuration.

Things to observe:

The cloud.google.com/neg annotation automatically creates a GCP Network Endpoint Group (NEG); we will examine this shortly

We then create a deployment and its configuration by creating and applying the following configurations.

Here we can observe the key Kubernetes resources we created.

$ kubectl get all
NAME                         READY   STATUS    RESTARTS   AGE
pod/app-1-596849dbc9-hhm65   1/1     Running   0          46m
pod/app-1-596849dbc9-t6bmd   1/1     Running   0          46m
pod/td-demo-gke-client       2/2     Running   0          114mNAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/app-1        ClusterIP   10.8.8.222   <none>        80/TCP    46m
service/kubernetes   ClusterIP   10.8.0.1     <none>        443/TCP   3h51mNAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/app-1   2/2     2            2           46mNAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/app-1-596849dbc9   2         2         2       46m

Things to observe:

While the Kubernetes service, app-1, provides both an VIP address, 10.8.8.222, and DNS name, app-1.default.svc.cluster.local, we will not be using either one of them to access the backing pods

Instead, we will be using the app-1 Network Endpoint Group, available from the menu Compute Engine > Network Endpoint Group.

We can think of a Network Endpoint Group much like a Kubernetes Endpoint, except it exists outside of the GKE cluster.

A network endpoint group (NEG) is a configuration object that specifies a group of backend endpoints or services. A common use case for this configuration is deploying services in containers. You can also distribute traffic in a granular fashion to applications running on your backend instances.

— Google Cloud — Network endpoint groups overview

From this point forward, we essentially follow the steps we followed in the previous article to create a Traffic Director service.

Please note: For some reason, we are forced to create the Health Check as part of the Traffic Director service creation process.

First we create the Traffic Director service backed by GKE pods from the Network Services > Traffic Director menu and pressing the Create Service button. We supply the values:

Name: td-gke-service
Backend Type: Network Endpoint Group
Network Endpoint Group: app-1 (us-central1-c)
Maximim RPS: 5
Health Check: Create health check

For the Health Check we supply the values:

Name: td-gke-health-check
Protocol: HTTP

We then press the Save and Continue button to save the Health Check. Then press the Continue and Done button to create the Traffic Director service.

While we have created the new Traffic Director service, we have not specified the logic, Routing Map Rules, that direct traffic to it. In this case we want to direct any traffic to the 10.0.0.2 VIP on port 80 to the service.

From the Traffic Director page, we select the Routing rule maps tab and press the Create Routing Rule Map button.

We name the Routing Map Rule td-gke-routing-rule-map.

We then add a Forwarding Rule using the Add forwarding rule button; providing Name: td-gke-forwarding-rule, Custom IP: 10.0.0.2, and Port: 80 (default).

For the provided Host and path rules, we select the td-gke-service; the service we created earlier.

Save the Routing Map Rule.

At this point our second Traffic Director service is fully configured.

Verifying the Configuration

As we did previously, we can verify the configuration by logging into both the td-demo-vm-client GCE VM instance as well as the td-demo-gke-client GKE pod.

In both cases, we confirm that the client’s traffic to the VIP 10.0.0.2 on port 80 is intercepted and sent to one of the the GKE pods backing the Traffic Director service.

$ curl http://10.0.0.2
<html>
<body>
Hello World!
</body>
</html>

We can also use the DNS name hello-world-gke.example.private that we setup earlier.

$ curl http://hello-world-gke.example.private
<html>
<body>
Hello World!
</body>
</html>

Next Steps

So far, we have focused on the basic service discovery feature of Traffic Director, in the next article, Traffic Director by Example: Part 3, we will build upon this; exploring some of Traffic Director’s more advanced features.

Traffic Director by Example: Part 2

Sidebar into Automatic Envoy Deployments

Client on GKE Pod

Service Backed by GKE Pods

Verifying the Configuration

Next Steps

Written by John Tucker