debugging

Follow logs from multiple K8s Pods in a Deployment, ReplicaSet, etc.

For production applications running in containerized infrastructure (e.g. Kubernetes, ECS, Docker Swarm, etc.)—and even for more traditional infrastructure with multiple application servers (for horizontal scalability), it is important to have centralized, persistent logging of some sort or another.

Some services like the ELK/EFK stack, SumoLogic, and Splunk offer a robust feature set for full text searching, filtering, and 'log intelligence'. On the other end of the spectrum, you can use a simple aggregator like rsyslogd or CloudWatch Logs without a fancy system on top if you just need basic central log storage.

But when I'm debugging something in a Kubernetes cluster—especially something like an internal service which I may not want to have logging everything to a central logging system (for cost or performance reasons)—it's often helpful to see all the logs from all pods in a Deployment or Replication Controller at the same time.

You can always stream logs from a single Pod with the command:

Fixing '503 Service Unavailable' and 'Endpoints not available' for Traefik Ingress in Kubernetes

In a Kubernetes cluster I'm building, I was quite puzzled when setting up Ingress for one of my applications—in this case, Jenkins.

I had created a Deployment for Jenkins (in the jenkins namespace), and an associated Service, which exposed port 80 on a ClusterIP. Then I added an Ingress resource which directed the URL jenkins.example.com at the jenkins Service on port 80.

Inspecting both the Service and Ingress resource with kubectl get svc -n jenkins and kubectl get ingress -n jenkins, respectively, showed everything seemed to be configured correctly:

Reverse-proxying a SOAP API accessed via PHP's SoapClient

I'm documenting this here, just because it's something I imagine I might have to do again someday... and when I do, I want to save myself hours of pain and misdirection.

A client had an old SOAP web service that used IP address whitelisting to authenticate/allow requests. The new PHP infrastructure was built using Docker containers and auto-scaling AWS instances. Because of this, we had a problem: a request could come from one of millions of different IP addresses, since the auto-scaling instances use a pool of millions of AWS IP addresses in a wide array of IP ranges.

Because the client couldn't change their API provider (at least not in any reasonable time-frame), and we didn't want to throw away the ability to auto-scale, and also didn't want to try to build some sort of 'Elastic IP reservation system' so we could draw from a pool of known/reserved IP addresses, we had to find a way to get all our backend API SOAP requests to come from one IP address.

The solution? Reverse-proxy all requests to the backend SOAP API.

Should I disable PHP warnings and notices?

This is a reposting of what I wrote on the Acquia Dev Center blog in 2016, Should I disable PHP warnings and notices?.

Drupal onscreen logged PHP error messages and warnings

Many developers who work on Drupal (or other web/PHP) projects have error reporting disabled in their local or shared dev environments, for a variety of reasons—some don't know how to enable it, some are annoyed by the frequency of notices, warnings, and errors, and some don't like to be reminded of how many errors are logged!

There are a few important reasons you should make sure to show all errors when developing, though:

Viewing email in Linux using postfix's mailq and postcat

When I'm developing using the Drupal Development VM, or checking into email processing on any of my servers, I usually use postfix to handle mail sending. Postfix is simple, preinstalled on most Linux distributions (and easy to set up if not), and is easy enough to use.

Here are the most common commands I use when either developing or troubleshooting email in production:

  • mailq - print a list of all queued mail
  • postcat -vq [message-id] - print a particular message, by ID (you can see the ID along in mailq's output)
  • postqueue -f - process the queued mail immediately
  • postsuper -d ALL - delete ALL queued mail (use with caution—but handy if you have a mail send going awry!)

There are many other helpful commands and scripts to help deal with mail (e.g. deleting all messages to a certain domain, or deleting specific message IDs easily), but these are the main ones I use during day-to-day development and troubleshooting.

Debugging Varnish VCL configuration files

If you're a Drupal or PHP developer used to debugging or troubleshooting some code by adding a print $variable; or dpm($object); to your PHP, and then refreshing the page to see the debug message (or using XDebug, or using watchdog logging...), debugging Varnish's VCL language can be intimidating.

VCL uses C-like syntax, and is compiled when varnish starts, so you can't just modify a .vcl file and refresh to see changes or debug something. And there are only a few places where you can simply stick a debug statement. So, I'll explain four different ways I use to debug VCLs in this post (note: don't do this on a production server!):

Simple Error statements (like print in PHP)

Sometimes, all you need to do is see the output of a variable, like req.http.Cookie, inside vcl_recv(). In these cases, you can just add an error statement to throw an error in Varnish and output the contents of a string, like the Cookie:

Subscribe to RSS - debugging