k8s

Mcrouter Operator - demonstration of K8s Operator SDK usage with Ansible

It wouldn't surprise me if you've never heard of Mcrouter. Described by Facebook as "a memcached protocol router for scaling memcached deployments", it's not the kind of software that everyone needs.

There are many scenarios where a key-value cache is necessary, and in probably 90% of them, running a single Redis or Memcached instance would adequately serve the application's needs. There are more exotic use cases, though, where you need better horizontal scaling and consistency.

Trying out CRC (Code Ready Containers) to run OpenShift 4.x locally

I've been working a bit with Red Hat lately, and one of the products that has intrigued me is their OpenShift Kubernetes platform; it's kind of like Kubernetes, but made to be more palatable and UI-driven... at least that's my initial take after taking it for a spin both using Minishift (which works with OpenShift 3.x), and CRC (which works with OpenShift 4.x).

Because it took me a bit of time to figure out a few details in testing things with OpenShift 4.1 and CRC, I thought I'd write up a blog post detailing my learning process. It might help someone else who wants to get things going locally!

CRC System Requirements

First things first, you need a decent workstation to run OpenShift 4. The minimum requirements are 4 vCPUs, 8 GB RAM, and 35 GB disk space. And even with that, I constantly saw HyperKit (the VM backend CRC uses) consuming 100-200% CPU and 12+ GB of RAM (sheesh!).

A Drupal Operator for Kubernetes with the Ansible Operator SDK

Kubernetes is taking over the world of infrastructure management, at least for larger-scale operations, and best practices have started to solidify. One of those best practices is the cultivation of Custom Resource Definitions (CRDs) to describe your applications in a Kubernetes-native way, and Operators to manage your the Custom Resources running on your Kubernetes clusters.

In the Drupal community, Kubernetes uptake has been somewhat slow, but is on the rise. Just like with Docker adoption for local development, the tooling and documentation has been slowly percolating. For example, Tess Flynn from TEN7 has been boldly going where no one has gone before (oops, wrong scifi series!) using the Force to promote Drupal usage in a Kubernetes environment.

Follow logs from multiple K8s Pods in a Deployment, ReplicaSet, etc.

For production applications running in containerized infrastructure (e.g. Kubernetes, ECS, Docker Swarm, etc.)—and even for more traditional infrastructure with multiple application servers (for horizontal scalability), it is important to have centralized, persistent logging of some sort or another.

Some services like the ELK/EFK stack, SumoLogic, and Splunk offer a robust feature set for full text searching, filtering, and 'log intelligence'. On the other end of the spectrum, you can use a simple aggregator like rsyslogd or CloudWatch Logs without a fancy system on top if you just need basic central log storage.

But when I'm debugging something in a Kubernetes cluster—especially something like an internal service which I may not want to have logging everything to a central logging system (for cost or performance reasons)—it's often helpful to see all the logs from all pods in a Deployment or Replication Controller at the same time.

You can always stream logs from a single Pod with the command:

Monitoring Kubernetes cluster utilization and capacity (the poor man's way)

If you're running Kubernetes clusters at scale, it pays to have good monitoring in place. Typical tools I use in production like Prometheus and Alertmanager are extremely useful in monitoring critical metrics, like "is my cluster almost out of CPU or Memory?"

But I also have a number of smaller clusters—some of them like my Raspberry Pi Dramble have very little in the way of resources available for hosting monitoring internally. But I still want to be able to say, at any given moment, "how much CPU or RAM is available inside the cluster? Can I fit more Pods in the cluster?"

So without further ado, I'm now using the following script, which is slightly adapted from a script found in the Kubernetes issue Need simple kubectl command to see cluster resource usage:

Usage is pretty easy, just make sure you have your kubeconfig configured so kubectl commands are working on the cluster, then run:

Mounting a Kubernetes Secret as a single file inside a Pod

Recently I needed to mount an SSH private key used for one app to connect to another app into a running Pod, but to make sure it was done securely, we put the SSH key into a Kubernetes Secret, and then mounted the Secret into a file inside the Pod spec for a Deployment.

I wanted to document the process here because (a) I know I'm going to have to do it again and this will save me a few minutes' research, and (b) it's very slightly unintuitive (at least to me).

First I defined a secret in a namespace:

Fixing '503 Service Unavailable' and 'Endpoints not available' for Traefik Ingress in Kubernetes

In a Kubernetes cluster I'm building, I was quite puzzled when setting up Ingress for one of my applications—in this case, Jenkins.

I had created a Deployment for Jenkins (in the jenkins namespace), and an associated Service, which exposed port 80 on a ClusterIP. Then I added an Ingress resource which directed the URL jenkins.example.com at the jenkins Service on port 80.

Inspecting both the Service and Ingress resource with kubectl get svc -n jenkins and kubectl get ingress -n jenkins, respectively, showed everything seemed to be configured correctly:

Kubernetes' Complexity

Over the past month, I started rebuilding the Raspberry Pi Dramble project using Kubernetes instead of installing and configuring the LEMP stack directly on nodes via Ansible (track GitHub issues here). Along the way, I've hit tons of minor issues with the installation, and I wanted to document some of the things I think turn people away from Kubernetes early in the learning process. Kubernetes is definitely not the answer to all application hosting problems, but it is a great fit for some, and it would be a shame for someone who could really benefit from Kubernetes to be stumped and turn to some other solution that costs more in time, money, or maintenance!

Raspberry Pi Dramble cluster running Kubernetes with Green LEDs

Subscribe to RSS - k8s