docker

Fixing MySQL 'The table is full' error using the official MySQL Docker image

Recently I had to test importing some very large databases with lots of giant log tables (e.g. 5+ GB tables), and when I tried doing an import into a local docker MySQL container instance, I got ERROR 1114: The table is full. Here are the commands I used:

# Run a MySQL container locally to test a large file import.
$ docker run --name mysql-import-test -p 3306 -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=database_name -d mysql:latest

# Import a database .sql file and monitor progress with pv.
$ pv ~/database.sql | mysql -u root -proot -h 127.0.0.1 --port 32774 database_name
ERROR 1114 (HY000) at line 93898: The table 'xyz' is full

I found that—likely due to some Docker filesystem defaults—the MySQL import would fail every time when there was a database table containing more than 1GB of data. Now, this could be related to the way the database was exported, and I also found some issues where people were using memory tables that got exported and wouldn't import cleanly.

Use an ARG in Dockerfile FROM for dynamic image specification

Dockerfiles have been able to use ARGs to allow passing in parameters during a docker build using the CLI argument --build-arg for some time. But until recently (Docker's 17.05 release, to be precise), you weren't able to use an ARG to specify all or part of your Dockerfile's mandatory FROM command.

But since the pull request Allow ARG in FROM was merged, you can now specify an image / repository to use at runtime. This is great for flexibility, and as a concrete example, I used this feature to allow me to pull from a private Docker registry when building a Dockerfile in production, or to build from a local Docker image that was created as part of a CI/testing process inside Travis CI.

To use an ARG in your Dockerfile's FROM:

dockrun oneshot — quick local environments for testing infrastructure

Since I work among a ton of different Linux distros and environments in my day-to-day work, I have a lot of tooling set up that's mostly-OS-agnostic. I found myself in need of a quick barebones CentOS 7 VM to play around in or troubleshoot an issue. Or I needed to run Ubuntu 16.04 and Ubuntu 14.04 side by side and run the same command in each, checking for differences. Or I needed to bring up Fedora. Or Debian.

I used to use my Vagrant boxes for VirtualBox to boot a full VM, then vagrant ssh in. But that took at least 15-20 seconds—assuming I already had the box downloaded on my computer!

Drupal VM does Docker

Drupal VM on Docker Hub

Drupal VM has used Vagrant and (usually) VirtualBox to run Drupal infrastructure locally since its inception. But ever since Docker became 'the hot new thing' in infrastructure tooling, I've been asked when Drupal VM will convert to using Docker.

The answer to that question is a bit nuanced; Drupal VM has been using Docker to run its own integration tests for over a year (that's how I run tests on seven different OSes using Travis CI). And technically, Drupal VM's core components have always been able to run inside Docker containers (most of them use Docker-based integration tests as well).

But Docker usage was always an undocumented and unsupported feature of Drupal VM. But no longer—with 4.5.0, Drupal VM now supports Docker as an experimental alternative to Vagrant + VirtualBox, and you can use Drupal VM with Docker in one of two ways:

Bash logic structures and conditionals (if, case, loops, etc.) in Travis CI

Travis CI's documentation often mentions the fact that it can call out to shell scripts in your repository, and recommends anything more complicated than a command or two (maybe including a pipe or something) be placed in a separate shell script.

But there are times when it's a lot more convenient to just keep the Travis CI-specific logic inside my repositories' .travis.yml file.

As it turns out, YAML is well-suited to, basically, inlining shell scripts. YAML's literal scalar indicator (a pipe, or |) allows you to indicate a block of content where newlines should be preserved, though whitespace before and after the line will be trimmed.

So if you have a statement like:

if [ "${variable}" == "something" ]; then
  do_something_here
fi

You can represent that in YAML via:

Fix for Ansible hanging when used with Docker and TTY

For almost all my Ansible roles on Ansible Galaxy, I have a comprehensive suite of tests that run against all supported OSes on Travis CI, and the only way that's possible is using Docker containers (one container for each OS/test combination).

For the past year or so, I've been struggling with some of the test suites having strange issues when I use docker exec --tty (which passes through Ansible's pretty coloration) along with Ansible playbooks running inside Docker containers in Travis CI. It seems that certain services, when restarted on OSes running sysvinit (like Ubuntu 14.04 and CentOS 6), cause ansible-playbook to hang indefinitely, resulting in a build failure:

How I test Ansible configuration on 7 different OSes with Docker

The following post is an excerpt from chapter 11 in my book Ansible for DevOps. The example used is an Ansible role that installs Java—since the role is supposed to work across CentOS 6 and 7, Fedora 24, Ubuntu 12.04, 14.04, and 16.04, and Debian 8, I use Docker to run an end-to-end functional test on each of those Linux distributions. See an example test run in Travis CI, and the Travis file that describes the build.

Note: I do the same thing currently (as of 2019), but now I'm using Molecule to tie everything together; see Testing your Ansible roles with Molecule.

Reintroducing the sanity of CM to container management

Recently, Ansible introduced Ansible Container, a tool that builds and orchestrates Docker containers.

While tools that build and orchestrate Docker containers are a dime a dozen these days (seriously... Kubernetes, Mesos, Rancher, Fleet, Swarm, Deis, Kontena, Flynn, Serf, Clocker, Paz, Docker 1.12+ built-in, not to mention dozens of PaaSes), many are built in the weirdly-isolated world of "I only manage containers, and don't manage other infrastructure tasks."

The cool thing about using Ansible to do your container builds and orchestration is that Ansible can also do your networking configuration. And your infrastructure provisioning. And your legacy infrastructure configuration. And on top of that, Ansible is, IMO, the best-in-class configuration management tool—easy for developers and sysadmins to learn and use effectively, and as efficient/terse as (but much more powerful than) shell scripts.

From Ansible Container's own README: