Container Orchestration — Technology Choices For Microservices and Other Workloads
Running server processes inside containers is here to stay. If your environment is small with a couple servers running a few dozen containers, you can likely get away with doing everything by hand. Beyond that scale, you need great tooling to deal with the heavy lifting and to provide common, baseline functionality. The alternative is a lot of tedious, error-prone, repetitious, manual work.
If you do not utilize a CI/CD pipeline and an orchestration system, development and operations will have to perform extreme, continuous collaboration and coordination.
When I began investigating the world of microservices earlier this year, I had no idea the extensive support infrastructure I would discover. Kubernetes has been an absolute treasure trove of a find, and Istio appears to be simply amazing for microservices—even though I know I have only scratched the surface of both these technologies.
From its humble beginnings less than three years ago, Kubernetes has quickly grown to be an amazing orchestration engine employed by countless corporations and embedded in many other projects. As a software designer with multiple decades under my belt, I am quite impressed with Kubernetes’ architecture. It is extremely modular and built under the expectation that many pieces can be replaced; in some cases there are already numerous choices for a given component.
All of this newness and multiplicity of choice can make getting started quite daunting. Just as I sit on the precipice of going full bore into Kubernetes, I am struck by a more fundamental decision:
Making the Right Container Orchestration Choice
As I began to dig deeper into the world of container orchestration, it became apparent that there are more than a few choices available. My instincts told me Kubernetes is the thing to use — but I also began to question how I knew I was right. There is nothing quite like uncertainty to make me dig deeper.
First question, what are the alternatives for container orchestration? Here is the list of orchestration systems I could find through reasonable amount of time searching and reading:
- Kubernetes — The apparent big-daddy of them all. The project itself is very active, and the architecture gives me comfort that continued development is going to be swift and safe. This is my instinctual choice.
- Docker Swarm — This is built into Docker by default, and has a lot of core functionality you want in a system. It has a lot of parity with Kubernetes, but a key lacking item in the free, open source version is Role Based Access Controls (RBAC); you can get that in the paid, Enterprise version.
- Marathon on Mesos — Mesos itself is a highly scalable clustering system for running tasks of all kinds. It relies on frameworks to support different kinds of tasks, and Marathon is the plugin which provides the support for container orchestration within the Mesos ecosystem. The list of frameworks is impressive.
- Titus — As I was writing this, Netflix open sourced their internal orchestration system. As an author trying to finish an article: Thanks Netflix… As a technologist looking for great technology: Thanks Netflix! Titus was designed to provide the tightest of integrations with the Amazon AWS infrastructure — where Netflix maintains its operations. One of their intentions is that other project will use their technology so that Netflix can use them in the future.
- Cattle — This is the orchestration engine made for and embedded within the Rancher system. I did not give Cattle a very deep look since its parent project has apparently bought into Kubernetes as its preferred and primary orchestration engine. (The main title on the Rancher website reads: “Enterprise Kubernetes Made Easy.” The page is riddled with how it helps you run Kubernetes clusters. No mention of Cattle exists on the webpage. It is clear the Rancher project has made its choice.)
- Nomad — Okay, this is Hashicorp. As a huge fan of Hashicorp, I would feel unjust if I did not give their product at least a once over. The product looks interesting on the surface with some fairly major paywall concerns: Namespaces are enterprise; service discovery, add on Consul; secret management, add on Vault. By a review of the documentation, it also appears to lack basic CNI configuration — the primary discussion for networking configuration is on mapping ports and static IP mappings.
- Kontena —This is a visually stunning product. You can run in their cloud offering, or you can setup your own platform master on your infrastructure of choice. If you choose to bring your own infrastructure, you can either choose to connect it to the Kontena Cloud for $15/month or not — the pretty web interface is what you give up in that case. Not having delved beyond a few hours of digging around their site, I am not certain the impact that would cause.
There are still others that you find hints of if you look hard enough: Deis, Mantl, Cloud Foundry, and Amazon ECS to name a few. These guys probably deserve more than this simple, honorable mention.
Requirements First
Making the choice here is difficult. Of course it depends on your requirements, and so let me list out a few important ones to me:
- Active development— The container orchestration world is relatively young. Inactive projects will quickly fall behind and signify that bugs are not being addressed. I get the sense that Cattle is on the way out. So scratch it here.
- No cloud vendor lock-in— I am not interested in being tied to any single cloud provider at this time. Titus falls out here — its tight integration with AWS is definitely a down side here.
- Simplicity — The more complex a system, the harder it will be to operate it. This requirement causes me to drop Mesos out of the running because it is not a container orchestration system first. It tries to be many things to many people, and that feels like a wrong fit.
- CNI Networking— The ability to have trivial network connectivity between my services is important. I do not want the developers spending time on special purpose code for finding dependent services. Docker Swarm and Kubernetes, you are both still in the running.
- Namespaces with RBAC — I work in a corporate environment, and one of my goals is to provide development, QA, staging, and production setups that do not collide. I could setup a separate cluster for each, or I could use RBAC and share my compute power. Docker Swarm, I am sorry to see you go, but this is the end of our journey together. Nomad, I love Hashicorp, but you too put this behind your paywall.
There you have it, some pretty high-level requirements that pretty quickly whittle down the playing field. It might not be fair to drop Mesos out on the “simplicity” category, but if you spend half the time I have investigating all of these options you will understand that at some point you must simplify your decision making in order to actually start moving forward.
I am left with the bizarre state of having Kubernetes and Kontena still on the list. Kontena is literally an 11th hour investigation — I almost left it relegated to the list of others. If I had done so, this final hour of authorship would have been less painful. Here it is. A decision has to be made, and while I will eventually circle back around to Kontena… Kubernetes is my current vote.
I feel guilty leaving so many amazing projects on the cutting room floor. This is what happens in today’s world of amazing options coupled with the age-old need to make a decision.
Getting Started With Kubernetes
So I have chosen Kubernetes to be my container orchestration system of choice. How do I get a cluster operational for testing and production use? The answers to this question are quite varied as well.
Kubernetes Deployment Methods
- Minikube — The recommended method to get a single-node Kubernetes running quickly for testing and development purposes. I prefer to see things in full action, so I did not settle for a single node deployment for my tests.
- Kubeadm — This is provided by kubernetes.io as a method to deploy a single-master, multi-node cluster. There are additional instructions for setting up a multi-master configuration too. In a previous article, I wrote more concise information on how I used Kubeadm by hand to setup my Digital Ocean cluster — and it also includes a Terraform configuration for automating this same deployment.
- Docker Enterprise 2.0 — As I was working on this article, Docker announced the upgrade to EE 2.0. This new version now incorporates a full Kubernetes deployment built into the product. From a quick reading they utilize Swarm to bootstrap the cluster and deploy Kubernetes.
- Rancher — “Enterprise Kubernetes Made Easy” is their claim. Indeed, I was able to get a full Kubernetes cluster running on Digital Ocean in under an hour by following their guide. My initial reaction was: “Holy cow! Rancher is AMAZING.” It supports managing the Kubernetes deployments into many environments and trivializes the High Availability deployment. It purports to allow management of multiple clusters along with managing other orchestration alternatives including their own Cattle and Apache Mesos.
- Mesosphere DC/OS — Possibly coming in as an even heavier weight champion as a container orchestration system in its own right, but now able to also administer Kubernetes clusters as well. This product appears quite compelling… Except that the really good stuff is under the Enterprise pay wall. I am also unclear from their website if the DC/OS version is free and the DC/OS Enterprise version is paid — or if they are both paid. Anytime I see a “Contact us for pricing,” I tend to move on. This will keep me from looking too closely — apologies to any offended.
- Kontena’s Pharos — It seems that even companies who have their own complete alternative to Kubernetes cannot keep their hands out of the Kubernetes deployment software initiatives. Their “Usage with Terraform” documentation looks to have a lot of power in making your Kubernetes installation a distinct, composable step: Setup your infrastructure in one step using whatever tool you have for that; then setup Kubernetes on top of that.
setup-infrastructure | install-kubernetes > profit
The list goes on: Pivitol’s Kubo, Apprenda Kismatic, CoreOS Tectonic, RedHat Openshift v3, Openshift Origin, and certainly more.
Hosted Options
- Amazon EKS — Elastic Container Service for Kubernetes — An Amazon hosted Kubernetes cluster. This is currently an “In Preview” technology by Amazon. This speaks towards the viability and future of Kubernetes…
- Google Kubernetes Engine (GKE) — This is Google’s hosted offering. I would like to say more, but for some reason my account is broken with respect to getting access to it.
- OpenShift — Red Hat’s online container service.
My Kubernetes Deployment Choice?
For deployment of Kubernetes, for me, I plan on continuing to work with both kubeadm (possibly replacing that with Pharos) as well as Rancher. Rancher showed great promise the first time I used it. The only downside is that I must first have a control machine onto which I install Rancher, but that is a small price to pay. I am not certain that I will want to use Rancher’s interface for interacting with my Kubernetes cluster, and so long as it does not get in the way of me using kubectl to control the cluster, we can get along just fine.
What is Next?
Now that I have gone through the exercise to understand the world of options more, I am ready to go heads down more with Kubernetes. There is exploration to do with my deployment methods of choice. I also talked before about Istio which lays on top of Kubernetes to provide even more foundation to support microservice communication and monitoring. Expect more of that in upcoming articles. Oh, and now that I tripped over Kontena… I feel pulled to give it a trial run through.
✉️ Subscribe to CodeBurst’s once-weekly Email Blast, 🐦 Follow CodeBurst on Twitter, view 🗺️ The 2018 Web Developer Roadmap, and 🕸️ Learn Full Stack Web Development.
