Cluster Setup¶

MBUS_IP¶

A static IP address is necessary to provide a consistent network interface for other nodes to connect to. This address is called the MBUS IP. If your IaaS or cloud orchestration software provide IP addresses which persist indefinitely and are not reset on reboot you may not have to set this explicitly.

Take note of the internal IP address of the core node. It will be required when configuring additional nodes in the following steps, so that they can attach to the core node.

Make sure that the IP address of its eth0 interface is registering the correct address, which may not be the case if you have set a static IP and not yet rebooted or restarted networking. To check the IP address, run:

$ ifconfig eth0

If necessary, set the static IP address:

$ kato op static_ip

Hostname¶

Next, set the fully qualified hostname of the core node. This is required so that the internal configuration of Helion Stackato matches the DNS record created for this system.

To set the hostname, run:

$ kato node rename hostname.example.com

This hostname will become the basename of the "API endpoint" address used by clients (for example, https://api.hostname.example.com).

Wildcard DNS¶

A wildcard DNS record is necessary to resolve not only the API endpoint, but all applications which will subsequently be deployed on the PaaS. Create a wildcard DNS record for the Core node (or Load Balancer/Router).

Core Node¶

On the core node, execute the following command:

$ kato node setup core api.hostname.example.com

This sets up the core node with just the implicit controller, primary, and router roles. The router role is required on this node even if there are other routers in the cluster and the node is not exposed to the internet.

If you intend to set up the rest of the cluster immediately, you would carry on to enable those roles you ultimately intend to run on the Core node. For example, to set up a core node with the controller, primary, router, and dea roles:

$ kato node setup core api.hostname.example.com
$ kato role add dea

Then proceed to configure the other VMs by attaching them to the Core node and assigning their particular roles. Register Helion Stackato and add the license key before adding nodes to the cluster.

Router Nodes¶

In smaller clusters, the Router enabled with the core node should be sufficient, and the core node can function as the gateway. To attach a node enabling just the router role:

$ kato node attach -e router <CORE_IP>

If a Router-only node is the gateway to Helion Stackato, the public DNS entry for the API endpoint must point to that node. For larger clusters requiring multiple gateway Routers, see the Load Balancer and Multiple Routers section below.

Data Services Nodes¶

Data services can share a single node (small clusters) or run on separate nodes (recommended for production clusters). To set up all available data services on a single node and attach it to the core node, run the following command on the data services node:

$ kato node attach -e data-services <CORE_IP>

DEA Nodes¶

Nodes which stage application code and run application containers are called Droplet Execution Agents (DEAs). Once the controller node is running, you can begin to add some of these nodes with the kato node attach command. To turn a generic Helion Stackato VM into a DEA and connect it to the core node:

$ kato node attach -e dea <CORE_IP>

Continue this process until you have added all the desired DEA nodes. You will be prompted to enter the password for the core node during the attach process. You can avoid this prompt by configuring passwordless authentication from the DEA node to the core node before attaching:

$ ssh-copy-id stackato@<CORE_IP>

Windows DEA Nodes¶

Verification¶

To verify that all the cluster nodes are configured as expected, run the following command on the core node:

$ kato status --all

Removing Nodes¶

Use the kato node remove to remove a node from the cluster. Run the following command on the core node:

$ kato node remove <NODE_IP>

Role Configuration using the Management Console¶

Once cluster nodes are connected to the core node, roles can be enabled or disabled using the Cluster Admin interface in the Management Console.

Single-Node¶

This is a configuration (not actually a cluster) which you would not generally deploy in production, but it helps to illustrate the role architecture in Helion Stackato. A node in this configuration will function much like a micro cloud, but can be used as the starting point for building a cluster later.

All that is required here is to enable all roles except for mdns (not used in a clustered or cloud-hosted environment):

$ kato node setup core api.hostname.example.com
$ kato role add --all-but mdns

Three-Node¶

This is the smallest viable cluster deployment, but it lacks the fault tolerance of larger configurations:

• 1 core node consisting of primary, controller, and router (and supporting processes)
• 1 data-services node running the database, messaging and filesystem services
• 1 DEA (Droplet Execution Agent) node

This configuration can support more users and applications than a single node, but the failure of any single node will impact hosted applications.

Five-Node¶

A typical small Helion Stackato cluster deployment might look like this:

• 1 core node consisting of primary, controller, and router (and supporting processes)
• 1 data-services node running the database, messaging and filesystem services
• 3 DEA (Droplet Execution Agent) nodes

In this configuration, fault tolerance (and limited scalability) is introduced in the pool of DEA nodes. If any single DEA node fails, application instances will be automatically redeployed to the remaining DEA nodes with little or no application down time.

20-Node¶

A larger cluster requires more separation and duplication of roles for scalability and fault tolerance. For example:

• 1 core node running the primary and controller roles (with supporting processes)
• 1 supplemental Controller node (sharing a filesystem and PostgreSQL database with the core node)
• 1 Load Balancer (Helion Stackato VM or hardware)
• 2 Router nodes
• 1 Filesystem service node
• 1 PostgreSQL + MySQL data service node
• 1 MongoDB, Redis, RabbitMQ + other data service node
• 12 DEA (Droplet Execution Agent) nodes

In this configuration:

• application instances span a larger group of DEA nodes so applications can be easily scaled to meet increasing demand
• web requests are evenly distributed between two Router nodes, either of which can fail without any interruption of service
• any data service node failure will be localized, not affecting data services on other nodes
• the auxiliary controller balances the load on the Management Console and system management tasks

Service Nodes¶

In addition to the ports listed above for service nodes and gateways, several service nodes assign a port for each individual user-requested service instance. These ranges should be kept open between DEA nodes and their respective service nodes. The default ranges are:

• harbor: 35000 - 40000
• memcached: 45001 - 50000
• mongodb: 15001 - 25000
• rabbit: 35001 - 40000
• rabbit3: 25001 - 30000
• redis: 5000 - 15000

Container Allowed Hosts and Ports¶

For security reasons, Docker application containers restrict access to hosts on the eth0 subnet. By default, only ports and hosts for built-in services and components (for example, service instances bound to an application) are explicitly allowed.

To configure a cluster for host and port access, you must determine the IP address of each DEA node using the kato node list command and then ssh to each DEA node. For example:

$ ssh -i myClusterPublicKey stackato@198.51.100.0

The following commands display the current configuration:

• fence docker/allowed_subnet_ips: Display a list of all allowed IP addresses. For example:

  $ kato config get fence docker/allowed_subnet_ips
  - 192.51.100.0
  - 192.51.100.1
  - 192.51.100.2
  

• fence docker/allowed_host_ports: Display a list of all allowed ports. For example:

  $ kato config get fence docker/allowed_host_ports
  - 80
  - 443
  - 8123
  - 3306
  - 6379
  

The following commands modify the current configuration:

• fence docker/allowed_subnet_ips <ip-address:port>: Delete an IP address from the allowed_subnet_ips list. For example:

  $ kato config pop fence docker/allowed_subnet_ips 192.0.2.24:6379
  

  Note

  You can open a port for an individual IP address or an IP CIDR block. Port settings apply only to the specified individual IP address.

• fence docker/allowed_host_ports <port>: Delete a port from the allowed_host_ports list. For example:

  $ kato config pop fence docker/allowed_host_ports 6379
  

The following settings allow or restrict access from application containers:

• fence docker/allowed_host_ports: If applications need access to custom services on a specific port, but the IP address changes or is not known ahead of time, add the port to this list. For example:

  $ kato config push fence docker/allowed_host_ports 25
  

  Warning

  Because this action opens the port to all IP addresses, do not perform it on production systems.

• fence docker/allowed_subnet_ips: If the specific IP address for a service is static and known, add the IP address with or without the port specification. For example:

  $ kato config push fence docker/allowed_subnet_ips 198.51.100.0
  $ kato config push fence docker/allowed_subnet_ips 198.51.100.1:9001
  

• fence docker/block_network_ips: To explicitly block access to a specific IP address (internal or external):

  $ kato config push fence docker/block_network_ips 203.0.113.0
  

To apply these changes to new application containers, and to restart the applications that have already been deployed, restart the DEA role.

Harbor (Port Service) Node Configuration¶

To enable port forwarding for user applications, you must configure the optional Harbor TCP/UDP port service on a node with a public network interface. The security group or firewall settings for this node should make the configured port range publicly-accessible. For complete configuration instructions, see Harbor Setup.

To Configure Networking Settings¶

You can configure a single load balancer to work with multiple router nodes in your Helion Stackato cluster. For the load balancer to work correctly, it must be able to communicate with the router nodes in the cluster.

Perform the following steps on each router node:

1. ssh into the node.

2. Rename the node using the kato node rename command.

   Important

   Each router node must have the same API endpoint.

3. Attach the node as a router to the core node:

   $ kato node attach --enable router <core-ip>
   

4. (Optional) Set the HTTP and HTTPS proxies for applications <server-config-application-proxy>.

For more information, see Router Role.

Load Balancers and SSL Termination¶

You can configure load balancers to either terminate SSL connections or to pass them through to the routers. These configurations require different router settings.

• If the load balancer terminates SSL:

  • Use HTTP or HTTPS listeners or ensure that the load balancer adds X-Headers.

  • In the router configuration, set prevent_x_spoofing to false.

  • Backends (connections to router nodes) should forward using HTTP.

    Note

    Using HTTPS is possible, but this requires chaining your certificates on the router nodes.

• If the load balancer does not terminate SSL:

  • Use TCP listeners.
  • Set prevent_x_spoofing to true.
  • Ensure that listeners on port 443 forward to port 443 on the backend.
  • The certificates should be identical on all router nodes.

Rename the Load Balancer¶

The Load Balancer is the primary point of entry to the cluster. It must have a public-facing IP address and take on the primary hostname for the system as configured in DNS. Run the following on Load Balancer node:

$ kato node rename hostname.example.com

Set up the Core Node¶

The core node will need to temporarily take on the API endpoint hostname of the Helion Stackato system (the same name as the Load Balancer above). Run the following on the core node:

$ kato node rename hostname.example.com

If it is not already configured as the core node, do so now:

$ kato node setup core api.hostname.example.com

The kato node rename command above is being used to set internal Helion Stackato parameters, but all hosts on a network should ultimately have unique hostnames. After setup, rename the core node manually by editing /etc/hostname and /etc/hosts, then sudo service hostname restart.

Set up Supplemental Routers¶

As with the core node, you will need to run kato node rename on each router with the same API endpoint hostname. Run the following on each Router:

$ kato node rename hostname.example.com

Then enable the 'router' role and attach the node to the cluster:

$ kato node attach -e router <MBUS_IP>

As above, rename each host manually after configuration to give them unique hostnames. The MBUS_IP is the network interface of the Core node (usually eth0).

Configure the Helion Stackato Load Balancer¶

Attach the Helion Stackato VM to the core node:

$ kato node attach <MBUS_IP>

To set up the node as a Load Balancer automatically:

$ kato node setup load_balancer --force

This command fetches the IP addresses of all configured routers in the cluster.

To set up the Load Balancer manually, specify the IP addresses of the Router nodes. For example:

$ kato node setup load_balancer 10.5.31.140 10.5.31.145

Helion Stackato Load Balancer SSL Certificates¶

The Helion Stackato load balancer terminates SSL connections. SSL certificates must be set up and maintained on this node, and the backend router nodes in order for application SSO and AOK to work correctly.

See the Using your own SSL certificate and CA Certificate Chaining sections for Helion Stackato Load Balancer instructions.