Best Practices¶

Upgrading the Helion Stackato VM¶

• On production systems, configure Ubuntu's unattended-upgrades package to apply security updates automatically. To enable this setting, run the following command:

  $ sudo dpkg-reconfigure -plow unattended-upgrades
  

  By default, this setting will upgrade packages only from the security origin. These packages are safe to apply to Helion Stackato VMs. For more information, see Ubuntu's documentation on using the "unattended-upgrades" package.

  Important

  Some security upgrades (for example, kernel patches) require a reboot before taking effect. Reboot cluster nodes manually during scheduled Helion Stackato cluster maintenance. Enabling the Unattended-Upgrade::Automatic-Reboot option is not recommended.

• To apply security upgrades manually, run the following commands on all cluster nodes, one node at a time:

  $ sudo apt-get update
  

  $ sudo unattended-upgrades -d
  

  If you use a proxy, export the http_proxy and https_proxy environment variables. For example:

  $ sudo sh -c "http_proxy=http://myproxy.example.com:3128 \
  https_proxy=http://myproxy.example.com:3128 apt-get update && unattended-upgrades -d"
  

  To ensure that new kernels, modules, and libraries are loaded, you must reboot each node after unattended-upgrades -d completes.

Using Docker Hub and Docker Registries¶

If your cluster has several DEA nodes, it is a good practice to share your base Docker image from Docker Hub instead of generating an updated image on each DEA. For more information, see the Docker documentation on Pushing a Repository Image to Docker Hub.

You can also share your base Docker image from a Docker registry on your network. For more information, see the documentation on Docker Registry.

To Upgrade the Docker Image¶

When the Helion Stackato VM is up-to-date, you must also upgrade the base Docker image. Perform the following steps on each DEA node in the cluster, one node at a time.

1. Create a new working directory:

   $ mkdir ~/upgrade-alsek && cd $_
   

2. In this directory, create a Dockerfile and add the following code to it:

   FROM stackato/stack-alsek:latest
   RUN apt-get update
   RUN unattended-upgrades -d
   RUN apt-get clean && apt-get autoremove
   

   Note

   You can use the kato-patched tag to target the image most recently updated using kato patch. To prevent the accumulation of AUFS filesystem layers, you can use this tag as a starting point

3. Build the docker image using the --no-cache=true option. Give the image a tag relevant to the particular upgrade. For example:

   $ sudo docker build --no-cache=true -rm -t stackato/stack-alsek:upgrade-2017-12-31 .
   

   Important

   The . at the end of the command specifies that the Dockerfile located in the current directory should be used.

4. Tag the Docker image as the latest stack-alsek image:

   $ sudo docker tag -f stackato/stack-alsek:upgrade-2017-12-31 stackato/stack-alsek:latest
   

5. In order for security upgrades to take effect within the application containers, application owners or Helion Stackato administrators must restart all running applications using the management console or the stackato client. To check which image is used by any running apps, run the docker ps command on your DEAs.

   Caution

   Do not run the docker restart command.

6. (Optional) If DEA autoscaling is enabled on your cluster, you must also update the DEA template.

Understanding System Limitations¶

Before choosing a backup, migration, or upgrade strategy, it is important to understand what data Helion Stackato can save, and what data may have to be reset, redeployed, or reconfigured. This is especially important when migrating to a new cluster.

Exporting Data from Helion Stackato¶

You can export data from your Helion Stackato VM using the kato data export command. This command can export internal Helion Stackato data (such as users, groups, quotas, or settings), application droplets, and data services.

• To export data from a single node, ssh into your node from the core node and then run the following command:

  $ kato data export --only-this-node
  

• To export data from an entire cluster, ssh into your core node and run the following command:

  $ kato data export --cluster
  

When the export process is complete, a tgz file is generated. You can use scp or another utility (such as sftp or rsync) to move the tgz file to another system, or to save the file directly to a mounted external filesystem by specifying the full path and filename during export.

Running Scheduled Backups¶

• On production systems, scheduling regular backups of controller data, apps, droplets, and service data is a good practice. Helion Stackato administrators can implement a suitable backup routine using cron/crontab to automate the backup process. The following example is an entry in the root user's crontab on the filesystem node:

  0 3 * * * su - stackato /bin/bash -c '/home/stackato/bin/kato data export --cluster /mnt/nas/stackato-backup.tgz'
  

  This entry runs the kato data export --cluster command every morning at 3 AM as the root user using the stackato user's (required) login environment, and saves a tgz file to a mounted external filesystem.

• Because certain shell operations performed during export require sudo permissions when run interactively, you must use the root user to run scheduled (non-interactive) backups that run the kato export command.

  For clusters, you must also set up passwordless SSH key authentication between the core node and all non-core nodes. Because certain services require shell commands to be run locally, these commands must be run on the node that hosts the filesystem role.

Migrating to Helion Stackato 3.x¶

The kato data import command detects if you are upgrading from Helion Stackato 2.x to 3.x and performs specific processing that accounts for differences between the two versions:

• Each imported user is added to his or her own organization.
• Existing administrators have corresponding (global) administrative privileges in the new system.
• Groups are converted into organizations. All applications and users are placed within a default space within each organization.
• Services are imported.
• Applications are redeployed automatically. Any applications that fail to redeploy are listed and must be redeployed manually by their owners. For more information, see Incompatible Apps.
• AOK (for example, LDAP) configuration is imported.

Other than these processes, migration follows the same export and import steps.

Importing Server Data¶

• To import Helion Stackato data, transfer the exported tgz file to the target VM.

• Alternatively, note the hostname of the old VM or core node.

  1. Log into Helion Stackato and run the kato data import command with the relevant options. In the following example, the command specifies importing all data into a new cluster from a tgz file:

     $ kato data import --cluster stackato-export-data.tgz
     

  2. To import data from a running Helion Stackato system, specify the hostname of the old core node. For example:

     $ kato data import --cluster stackato-host.example.com
     

Upgrading Stackato 3.x¶

In Stackato 3.0, the kato node upgrade command was added, allowing upgrades of Helion Stackato systems in place. For more information and instructions, see Upgrading Helion Stackato.

To Monitor Using New Relic¶

To use New Relic to monitor Helion Stackato, you need need a New Relic account and a license key.

1. Install the newrelic-sysmond package and start the monitoring daemon on each Helion Stackato VM.

   For more information, see the the New Relic Server Monitor installation (Ubuntu) documentation.

2. (Optional) To use nrsysmond to monitor application containers, install the newrelic-sysmond package in the Docker image by scripting the installation steps in the Dockerfile of a new container image.

To Monitor Using Nagios¶

Although Helion Stackato can supervise processes on a server or cluster using Supervisor, it is a good practice to add external monitoring using a tool such as Nagios.

For more information and instructions on installing and configuring Nagios, see the sample Nagios configuration file and the Nagios Core Documentation.

To Relocate Services, Droplets, and Containers¶

To move database services, application droplets, and application containers to larger partitions, perform the following steps:

1. Mount the filesystem or block storage service on the VM, with quotas enabled.

2. Create directories for the items you want to move.

3. Run the following kato relocate commands:

   $ kato stop
   

   $ kato relocate services /mnt/ebs/services
   

   $ kato relocate docker_registry /mnt/ebs/docker_registry
   

   $ kato relocate droplets /mnt/ebs/droplets
   

   $ kato relocate containers /mnt/ebs/containers
   

To Enable Filesystem Quotas¶

Helion Stackato filesystem quotas cannot be enforced unless they are mounted on partitions that support Linux quotas. You may need to specify this requirement explicitly when running the mount command. The kato relocate command will warn you if this step is necessary.

1. Run the mount and quota commands. For example:

   $ sudo mount -o remount,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 /mnt/containers
   

   $ sudo quotacheck -vgumb /mnt/containers
   

   $ sudo quotaon -v /mnt/containers
   

2. To ensure that the quotas are preserved after reboot, edit /etc/init.d/setup_stackato_lxc to include mount commands for each partition. For example:

   # enable quotas for Helion Stackato containers
   if [[ -f "/mnt/containers/aquota.user" ]]; then
     mount -o remount,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 /mnt/containers
     quotaon -v /mnt/containers
   fi
   

To Adjust Filesystem Quotas Retroactively¶

When you change the quota setting, it takes effect only on filesystems created after changing this setting. Thus, after you increase the filesystem quota, the quota may still appear to be too small.

You can use the quota and setquota commands to increase the filesystem quota retroactively.

1. Examine the location of your filesystems. Unless you relocate your filesystems or use external storage, your filesystems are located on your Helion Stackato VM, on a path similar to /var/stackato/services/filesystem/storage and have names such as stackatofs-2623db6af1dc84b.

2. To check the quota of a given filesystem, use the quota command. For example:

   $ sudo quota -s -u stackatofs-2623db6af1dc84b
   

3. To set this filesystem’s quota to 200MB, use the setquota command. In the following example, 204800KB is specified:

   $sudo setquota -u stackatofs-2623db6af1dc84b 0 204800 0 0 -a
   

For more information about the quota and setquota commands, see the Ubuntu documentation.