Docker and Fence

The Helion Stackato DEA role runs Linux containers to isolate user applications during staging and at runtime. Management of these application containers is handled by the fence process, which in turn uses Docker to create and destroy Linux containers on demand.

Admins can allow users to deploy Docker images directly to stackato with the stackato push command (see Docker Apps below).

Typically, admins will not have to work directly with Docker, but it is available if needed to customize or create new container images.

Docker Apps Administration

Docker Apps are web applications that can be pushed to Helion Stackato in the same way as regular, staged web applications. Administrators can control which users have this ability, which images are allowed, and how much space is allocated on the Controller nodes for storing images.

Docker App Deployment Permission

Because arbitrary Docker images can potentially expose a root user and other escalated privileges, you should restrict the ability to push Docker images to:

  • A trusted group of users in an Org with sudo permissions enabled
  • A trusted list of Docker images, namespaces, or registry servers
  • A trusted group deploying from a trusted list

The Helion Stackato Settings view in the web management console exposes these settings, but you can also set them using the kato config command. The following settings are available in the cloud_controller_ng group:

  • docker_apps/require_sudo: Boolean (true by default) Limit Docker image deployment to organizations with 'sudo' permissions enabled in the Quota Plan.
  • docker_apps/allow_any_registry: Boolean (true by default) Let users deploy Docker images from any registry server. Docker image deployment should be limited to either a trusted group of users, trusted image sources, or both. Allowing any user to deploy Docker images without restrictions is not recommended.
  • docker_apps/allowed_registries: List (not set by default) A list of servers, namespaces, or images that users are allowed to deploy. Entries are specified in the standard format accepted by the docker client (user:pass@domain.com:port/namespace/repo:tag). Credentials specified in the Allowed Docker Registries list are used as defaults when the user requests a matching image name without credentials. See the Helion Stackato Settings for further examples.

Note

If the registry credentials contain the @ or : characters, URL encode the username or password strings as needed.

Docker Image Storage Limits

Helion Stackato fetches images from the Docker Hub or the specified repository and keeps copies in a local registry server on the controller so that subsequent deployments using any of the downloaded layers can be done more quickly. A cron job runs periodically to keep the local registry server from growing too large. The size limit is defined in the following setting in kato config:

  • cloud_controller_ng docker_apps/storage_limit_mb: Int (default 10240) Size limit of the cloud controller's local Docker registry server.

This value should be adjusted based on the disk space available on the cloud controller nodes.

Modifying or Updating the Container Image

Application containers are created from a base Docker image (a template used to create Linux containers). Admins can create new images to add specific software required by applications or update operating system packages.

To create a new base image for Helion Stackato to use for application containers, perform the following steps on all nodes running the DEA role:

  1. Start with an empty working directory:

    $ mkdir ~/newimg
$ cd ~/newimg

  2. Check which images Helion Stackato is currently using as an app container template. The first image in the list is the default, while the subsequent images are fallback entries (checked in order):

    $ kato config get fence docker/images
- stackato/stack-alsek
- stackato/stack/alsek

  3. Create a Dockerfile which inherits the current Docker image, then runs an update or installation command. For example:

    FROM stackato/stack-alsek:kato-patched
RUN apt-get -y install libgraphite2-dev
    • FROM: inherits the environment and installed software from the Helion Stackato application image (the latest patched version).
    • RUN: specifies arbitrary commands to run before saving the image.
    • ADD: could be used to copy files into the image.

  4. Build the image, setting the maintainer's name, and an image name:

    $ sudo docker build -rm -t exampleco/newimg .

    If the new image is only an updated version of the default base image, you can tag the image stackato/stack-alsek:latest (as described in the Ubuntu Security Updates) and skip the following step.

  5. Configure Helion Stackato to use the new image:

    $ kato config set fence docker/images '["exampleco/newimg","stackato/stack-alsek","stackato/stack/alsek"]'
WARNING: Assumed type struct
- exampleco/newimg
- stackato/stack-alsek
- stackato/stack/alsek

    This step only needs to be done once, because the configuration change is shared with all nodes.

Note

See also the Upgrade the Docker Image section in the Best Practices section, which shows how to modify the Docker base image without changing kato config.

Privileged Containers

Helion Stackato runs unprivileged Docker containers by default. The containers do not have access to any devices on the (virtual) host.

This is a barrier to certain operations such as mounting disk partitions via NFS, so an option exists to change Docker container creation to privileged:

$ kato config set dea_ng docker/privileged true

This container privilege elevation would normally be used in conjunction with sudo container access in the quota:

$ stackato quota configure --allow-sudo

Warning

These changes should only be made when all users of the system are completely trusted (with admin privileges) because they compromise the isolation between application containers and host. Every application running under this configuration has the potential to become root on the host.

Admin Hooks

If an administrator wants to run arbitrary commands in all application containers, global admin hooks can be set to run immediately after corresponding user-specified deployment hooks (pre-staging, post-staging, pre-running) set in the application's manifest.yml file.

These hooks must be:

  • Plain Bash scripts with the executable bit set (chmod +x)
  • Named pre-staging, post-staging, or pre-running
  • Installed in /etc/stackato/hooks within the Docker image

For example, a pre-running admin hook might look like this:

#!/bin/sh
export PRE_RUN_DATE=`date`
export EXAMPLECO_KEY="3A0fwPwUftDu0FEzmhN8yJkvM1vS6A"
if [ -z "$NEW_RELIC_LICENSE_KEY" ]; then
  echo "setting default New Relic key"
  export NEW_RELIC_LICENSE_KEY="bdb9b44e8n4411d8bf39870f1919927d79cr0f1r"
fi
export STACKATO_HOOK_ENV=PRE_RUN_DATE,EXAMPLECO_KEY
sudo /usr/sbin/nrsysmond-config --set license_key=$NEW_RELIC_LICENSE_KEY
sudo /etc/init.d/newrelic-sysmond start

Note

The STACKATO_HOOK_ENV environment variable is needed to expose the specified variables in stackato ssh sessions, the application container's crontab, and PHP applications using the Legacy buildpack. This requirement may change in subsequent releases.

The Dockerfile for creating the image (see Modifying or Updating the Container Image ) would use the ADD directive to put a local hooks directory in the Docker image's /etc/stackato/ directory:

FROM stackato/stack/alsek
ADD hooks /etc/stackato/hooks

The pre-running hook example above would require the addition of newrelic-sysmond to the Docker image. A Dockerfile enabling that might look like this:

FROM stackato/stack/alsek

RUN echo deb http://apt.newrelic.com/debian/ newrelic non-free >> /etc/apt/sources.list.d/newrelic.list
RUN wget -O- https://download.newrelic.com/548C16BF.gpg | apt-key add -
RUN apt-get update
RUN apt-get install newrelic-sysmond
# The nrsysmond scripts are run with sudo
RUN echo "stackato ALL= NOPASSWD: /etc/init.d/newrelic-sysmond" >> /etc/sudoers
RUN echo "stackato ALL= NOPASSWD: /usr/sbin/nrsysmond-config" >> /etc/sudoers

ADD hooks /etc/stackato/hooks

Creating a Docker Registry

The steps above will work with smaller clusters or micro clouds where the creation of Docker images on each DEA can be done manually. On larger clusters, you should set up a Docker registry as a central repository for the base image. This registry server can also be used for deploying Docker apps.

Consult the documentation for guidelines on making Docker's official registry image production-ready.

If you do not want to set up your own private Docker registry server, you can push your images to the Docker Hub Registry instead. Private repositories are available.

To use a local registry server (for example, registry.example.com:5000) as the source for the exampleco/newimage base image described above:

  1. Push the newly built Docker image to the registry:

    $ sudo docker push registry.example.com:5000/exampleco/newimg

Note

The stackato/stack/alsek and stackato/base images (approximately 1.7GB) are pushed to the registry in addition to the new image. Make sure you have sufficient disk space available on the VM.

  1. On all DEA nodes, pull the new image from the registry:

    $ sudo docker pull registry.example.com:5000/exampleco/newimg

  2. Find the image ID of the image:

    $ sudo docker images | grep exampleco/newimg
registry.example.com:5000/exampleco/newimg   latest     49a17e99acgf    1 hour ago   1.707 GB

  3. Run the docker tag command to remove the registry hostname and port:

    $ sudo docker tag 49a17e99acgf exampleco/newimg

  4. Configure Helion Stackato to use the new image:

    $ kato config set fence docker/images \
> '["exampleco/newimg","stackato/stack-alsek","stackato/stack/alsek"]'
WARNING: Assumed type struct
- exampleco/newimg
- stackato/stack-alsek
- stackato/stack/alsek

    The kato config step only needs to be done once, because the configuration change is shared with all nodes

If you use the Docker Hub directly, exclude the hostname and port from the steps above, and skip the re-tagging step.