kubernetes

Ignore noisy logs with fluentd in EKS or other Kubernetes clusters

Recently, I decided to use the fluentd-kubernetes-daemonset project to easily ship all logs from an EKS Kubernetes cluster in Amazon to an Elasticsearch cluster operating elsewhere.

The initial configuration worked great out of the box—just fill in details like the FLUENT_ELASTICSEARCH_HOST and any authentication info, and then deploy the RBAC rules and DaemonSet into your cluster, and you're off to the races (assuming your Elasticsearch instance is configured to allow access from the cluster!).

But once I did that, I noticed the brand new EKS cluster was sending over 16,000 log messages per second to Elasticsearch. Doing a tiny bit of analysis (not much was required, honestly), I found that over 98% of the logs were coming from two EKS-specific noisy containers, efs-csi-node and ebs-snapshot-controller.

Kubernetes 101 livestream series starts Nov 18th!

On November 18th, at 11 a.m., the first episode of my upcoming Kubernetes 101 livestream series will start on my YouTube channel.

Kubernetes 101 Series Artwork

The first episode will be available here on YouTube: Kubernetes 101 - Episode 1 - Hello, Kubernetes!.

You can find more details about the series on my Kubernetes 101 site, and there is also an open-source Kubernetes 101 GitHub repository which will contain all the code examples for the series.

In the spring, I presented a similar livestream series, Ansible 101, covering all the basics of Ansible and setting people up for success in infrastructure automation.

Raspberry Pi Cluster Episode 6 - Turing Pi Review

A few months ago, in the 'before times', I noticed this post on Hacker News mentioning the Turing Pi, a 'Plug & Play Raspberry Pi Cluster' that sits on your desk.

It caught my attention because I've been running my own old-fashioned 'Raspberry Pi Dramble' cluster since 2015.

Raspberry Pi Dramble Cluster with Sticker - 2019 PoE Edition

So today, I'm wrapping up my Raspberry Pi Cluster series with my thoughts about the Turing Pi that I used to build a 7-node Kubernetes cluster.

Video version of this post

This blog post has a companion video embedded below:

Raspberry Pi Cluster Episode 4 - Minecraft, Pi-hole, Grafana and More!

This is the fourth video in a series discussing cluster computing with the Raspberry Pi, and I'm posting the video + transcript to my blog so you can follow along even if you don't enjoy sitting through a video :)

In the last episode, I showed you how to install Kubernetes on the Turing Pi cluster, running on seven Raspberry Pi Compute Modules.

In this episode, I'm going to show you some of the things you can do with the cluster.

10,000 Kubernetes Pods for 10,000 Subscribers

It started with a tweet, how did it end up like this?

I've had a YouTube channel since 2006—back when YouTube was a plucky upstart battling against Google Video (not Google Videos) and Vimeo. I started livestreaming a couple months ago on a whim, and since that time I've gained more subscribers than I had gained between 2006-2020!

So it seems fitting that I find some nerdy way to celebrate. After all, if Coline Furze can celebrate his milestones with ridiculous fireworks displays, I can do ... something?

Raspberry Pi Cluster Episode 2 - Setting up the Cluster

This post is based on one of the videos in my series on Raspberry Pi Clustering, and I'm posting the video + transcript to my blog so you can follow along even if you don't enjoy sitting through a video :)

.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }

In the first episode, I talked about how and why I build Raspberry Pi clusters.

I mentioned my Raspberry Pi Dramble cluster, and how it's evolved over the past five years.

Raspberry Pi Cluster Episode 1 - Introduction to Clusters

I will be posting a few videos discussing cluster computing with the Raspberry Pi in the next few weeks, and I'm going to post the video + transcript to my blog so you can follow along even if you don't enjoy sitting through a video :)

.embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; }

This is a Raspberry Pi Compute Module.

7 Raspberry Pi Compute Modules in a stack

And this is a stack of 7 Raspberry Pi Compute Modules.