High-availability configuration

View as Markdown

You can configure a self-hosted runZero installation for high-availability: a load balancer directs traffic to multiple console servers, which share a PostgreSQL cluster and a storage backend. The diagram below shows an example architecture on AWS with two availability zones.

Self-Hosted HA Architecture

In the diagram, an application load balancer (ALB) terminates TLS and points to a target group of two runZero servers in separate availability zones. Both servers use a multi-availability-zone PostgreSQL RDS instance configured for the same two zones, and both point to the same set of S3 storage buckets.

Installation steps

Installing and configuring a high-availability instance of runZero takes these steps:

  1. Deploy an application load balancer.
  2. Install and configure the first server node.
  3. Install and configure subsequent server nodes.

Deploy a load balancer with TLS termination

An application load balancer gives the runZero cluster its high-availability interface: it receives TLS connections and forwards the HTTP requests to the available servers. The load balancer should be highly available on its own. Any load balancer that can proxy large HTTP requests and websockets should work. It should add an X-Forwarded-For header to the requests it sends on to the runZero servers, and it should detect unhealthy nodes so that failover happens automatically. You can use the /health endpoint on the runZero servers for its health checks.

Prepare the first runZero server node

To install and configure the first runZero node:

  1. Install the runZero platform.
  2. Configure the console URL and TLS/XFF settings.
  3. Configure the database settings.
  4. Configure the shared storage settings.
  5. Verify your configuration.
  6. Create the initial user account.
  7. Add the runZero server to the load balancer and verify the connection.

Install the runZero platform

Some parts of the application still use the name "Rumble" for backwards compatibility. We'll update the documentation as they change.
  1. Go to the platform download page in the runZero Console.
  2. Copy the URL of the platform installer from the download page. The download path is uniquely keyed to your license.
  3. Download the installer with Wget or cURL:
  • wget -O runzero-platform.bin https://console.runzero.com/download/platform-combined/<UNIQUE KEY>/62e41615/runzero-platform-vX.X.X-linux-amd64.bin
  • curl -o runzero-platform.bin https://console.runzero.com/download/platform-combined/<UNIQUE KEY>/62e41615/runzero-platform-vX.X.X-linux-amd64.bin
  1. Make the installer executable:
# chmod u+x runzero-platform.bin
  1. Run the install command with the manual database option:
# ./runzero-platform.bin install --manual-database

Configure the console URL and TLS/XFF

The high-availability settings live in /etc/runzero/config. Start with the console URL and the TLS/XFF settings.

  • Set the RUNZERO_CONSOLE variable to the hostname or IP address of the load balancer:
RUNZERO_CONSOLE=https://{IP ADDRESS OR HOSTNAME}:443
  • Disable TLS and trust X-Forwarded-For headers with these two lines:
TLS=false
RUNZERO_TRUST_XFF=true

Configure the database

  • Edit the DATABASE_URL value to match this format:
DATABASE_URL=postgres://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}?{DB_OPTIONS}

An example DATABASE_URL for AWS RDS looks like:

DATABASE_URL=postgres://rumble:[password]@rumbledb.rds-id.us-east-1.rds.amazonaws.com:5432/rumble?sslmode=require

To send read-only queries to a replica, set DATABASE_REPLICA_URL with the same syntax. For RDS Aurora, use the read-only endpoint as the replica URL. Without replicas, only DATABASE_URL is required, and on RDS Aurora it should point to the writer endpoint.

With any other type of PostgreSQL clustering, make sure DATABASE_URL points to the highly-available endpoint for the active writer.

Configure shared storage

The shared storage backend holds asset data, raw scan data, and reports. runZero supports both mounted NFS and object storage.

  • To use NFS, mount a read-write NFS share at /opt/runzero/storage and select local storage mode with these options. The bucket names are created under the NFS mount base directory.
RUNZERO_STORAGE_MODE=local
RUNZERO_STORAGE_PATH=/opt/runzero/storage
ASSET_BUCKET=runzero-assets
SCAN_BUCKET=runzero-scans
  • To use AWS S3 as the shared storage, either configure an IAM instance role with read-write access to two S3 buckets, or put the AWS credentials in the configuration file.
    • To set AWS credentials specifically for runZero, use this syntax:
AWS_ACCESS_KEY_ID=AKIA…
AWS_SECRET_ACCESS_KEY=[SECRET]
  • Select S3 storage and name the two buckets:
AWS_REGION=<your-region>
RUNZERO_STORAGE_MODE=s3
ASSET_BUCKET=<company>-runzero-assets
SCAN_BUCKET=<company>-runzero-scans

For a non-AWS backend that is compatible with the S3 API, use the same AWS and bucket variables, override AWS_REGION, and set AWS_ENDPOINT_URL_S3 to the endpoint. Reach out to runZero support if you run into trouble with the endpoint configuration.

Most AWS CLI environment variables are also available alongside these variables for finer tuning.

Verify the configuration

  1. After saving the configuration file, verify that the self-hosted runZero platform can connect to your database:
sudo runzeroctl database verify
  1. Once the settings are configured and verified, restart the self-hosted runZero platform service:
sudo systemctl restart runzero-console

Create the initial user account

  • Run runzeroctl initial <your-email> to create the initial user account.

Add the runZero server to the load balancer and verify the connection

  • Point the load balancer at the new runZero server’s IP on port 80 (unless you changed it).
  • Browse to the load balancer URL over HTTPS using the configured DNS name.
  • Verify that the user account you just created can sign in.

Installing and configuring subsequent nodes

  • Download and install the runZero platform binary on each node as above, with the --manual-database option.
  • Copy the configuration file from the first node to /etc/runzero/config on each subsequent node.
  • After saving the configuration file, verify that the self-hosted runZero platform can connect to your database:
sudo runzeroctl database verify
  • Restart the platform service:
sudo systemctl restart runzero-console

Spread the runZero nodes across multiple availability zones and add them to the load balancer’s target group. To confirm that requests reach each node, review the syslog entries on that node.

AWS advanced configuration

runZero can load configuration settings from AWS Secrets Manager at startup. The Secrets Manager entries should match the key names in the configuration file. Set the secret name with:

AWS_SECRETS_MANAGER_KEY=rumble/production

For runZero to reach Secrets Manager, the configuration file must set AWS_REGION, and the instance needs either an IAM instance role with permission to read the Secrets Manager key or AWS credentials specified directly in the configuration file. With Secrets Manager in use, only the AWS options need to be in the configuration file; runZero loads the rest of the settings from the Secrets Manager key. That means copying the settings from the generated configuration file into the Secrets Manager key-value pairs.

Many settings must be identical across the runZero cluster for web requests and authentication to work, including the session secrets, the DB encryption key, and the console URL.

Updated