High-availability configuration

Self-hosted installations of runZero can be configured for high-availability. For this configuration, a load balancer is used to direct traffic to multiple console servers, which use a shared PostgreSQL cluster and storage backend. The following diagram illustrates an example architecture of a high-availability installation using AWS with two availability zones.

Self-Hosted HA Architecture

In this diagram, an application load balancer (ALB) is terminating TLS and pointing to a target group that consists of two runZero servers, each in separate availability zones. The runZero servers use a multi-availability-zone PostgreSQL RDS instance, also configured for the same two availability zones, and both servers point to the same set of S3 storage buckets.

Installation steps

These are the general steps for installing and configuring a high-availability instance of runZero:

  1. Deploy an application load balancer.
  2. Install and configure the first server node.
  3. Install and configure subsequent server nodes.

Deploy a load balancer with TLS termination

In order to provide a high-availability interface to the runZero cluster, an application load balancer should be used that receives TLS connections and forwards the HTTP requests to the available servers. This load balancer should be highly-available on its own. Any load balancer that can proxy large HTTP requests and websockets should be supported. The load balancer should add an X-Forwarded-For header to the HTTP requests sent onward to the runZero servers. The load balancer should also support detection of unhealthy nodes in order to handle automatic failover. The /health endpoint on the runZero servers can be used to implement health checks on the load balancer.

Prepare the first runZero server node

The general steps for installing and configuring the first runZero node are:

  1. Install the runZero platform.
  2. Configure the console URL and TLS/XFF settings.
  3. Configure the database settings.
  4. Configure the shared storage settings.
  5. Verify your configuration.
  6. Create the initial user account.
  7. Add the runZero server to the load balancer and verify the connection.

Install the runZero platform

Note: Some components of the application still reference the name "Rumble" for backwards compatibility. The documentation will be updated as these are changed.
  1. Go to platform download page in the runZero Console.
  2. Copy the URL to the platform installer from the download page. The download path for the installer is uniquely keyed to your license.
  3. Download the install using Wget or cURL:
  • wget -O runzero-platform.bin https://console.runzero.com/download/platform-combined/<UNIQUE KEY>/62e41615/runzero-platform-vX.X.X-linux-amd64.bin
  • curl -o runzero-platform.bin https://console.runzero.com/download/platform-combined/<UNIQUE KEY>/62e41615/runzero-platform-vX.X.X-linux-amd64.bin
  1. Run the following command to make the installer file executable:
# chmod u+x runzero-platform.bin
  1. Run the install command with the manual database option:
# ./runzero-platform.bin install --manual-database

Configure the console URL and TLS/XFF

To configure your self-hosted installation for high-availability, the settings below should be configured in /etc/runzero/config, starting with the console URL and the TLS/XFF settings.

  • Set RUNZERO_CONSOLE variable to the hostname or IP address of the load balancer:
RUNZERO_CONSOLE=https://{IP ADDRESS OR HOSTNAME}:443
  • Disable TLS and configure the server to trust X-Forwarded-For headers with the following two lines:
TLS=false
RUNZERO_TRUST_XFF=true

Configure the database

The next step is to configure the database to be used by runZero.

  • To do this, edit the DATABASE_URL value to match this format:
DATABASE_URL=postgres://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}?{DB_OPTIONS}

An example DATABASE_URL for AWS RDS will look like:

DATABASE_URL=postgres://rumble:[password]@rumbledb.rds-id.us-east-1.rds.amazonaws.com:5432/rumble?sslmode=require

To use a replica for read-only queries, specify DATABASE_REPLICA_URL with the same syntax. For RDS Aurora, specify the read-only endpoint as the replica URL. If no replicas are in use, only the DATABASE_URL is required, and this should point to the writer endpoint for RDS Aurora instances.

If a different type of PostgreSQL clustering is in use, make sure that the DATABASE_URL points to the highly-available endpoint for the active writer.

Configure shared storage

Next, configure the shared storage backend. This is used for storing asset data, raw scan data, and reports. runZero supports both mounted NFS and object storage.

  • To use NFS, mount a read-write NFS share to /opt/runzero/storage and specify local storage mode with the following configuration options. The bucket names will be created under the NFS mount base directory.
RUNZERO_STORAGE_MODE=local
RUNZERO_STORAGE_PATH=/opt/runzero/storage
ASSET_BUCKET=runzero-assets
SCAN_BUCKET=runzero-scans
  • To use AWS S3 as the shared storage, either configure an IAM instance role with read-write access to two S3 buckets, or specify the AWS credentials in the configuration file.
    • To specify AWS credentials specifically for runZero, use the following syntax:
AWS_ACCESS_KEY_ID=AKIA…
AWS_SECRET_ACCESS_KEY=[SECRET]
  • Specify that the S3 storage mechanism should be used and set the name of the two buckets:
AWS_REGION=<your-region>
RUNZERO_STORAGE_MODE=s3
ASSET_BUCKET=<company>-runzero-assets
SCAN_BUCKET=<company>-runzero-scans

If a non-AWS backend is used that is compatible with the S3 API, use the same AWS and bucket variables above but override AWS_REGION and set the AWS_ENDPOINT_URL_S3 parameter to the endpoint as appropriate. Reach out to runZero support if you run into any issues with the endpoint configuration.

In addition the variables above, most AWS CLI environment variables are also available for specific tuning.

Verify the configuration

  1. After saving your configuration file, use this command to verify the self-hosted runZero platform can connect to your database:
sudo runzeroctl database verify
  1. Once your settings are configured and verified you can restart the self-hosted runZero platform service:
sudo systemctl restart runzero-console

Create the initial user account

  • Use the runzeroctl initial <your-email> command to create the initial user account.

Add the runZero server to the load balancer and verify the connection

  • Point the load balancer to the new runZero server’s IP on port 80 (unless modified).
  • Browse to the load balancer URL over HTTPS using the configured DNS name.
  • Verify that the user account configured previously can authenticate correctly.

Installing and configuring subsequent nodes

  • Download and install the runZero platform binary on each node as above with the --manual-database option specified.
  • Copy the configuration file from the first node to /etc/runzero/config on each subsequent node.
  • After saving your configuration file, use this command to verify that the self-hosted runZero platform can connect to your database:
sudo runzeroctl database verify
  • Restart the platform service by running:
sudo systemctl restart runzero-console

runZero nodes should be spread across multiple availability zones and added to the load balancer’s target group. Verify that requests are reaching each individual node by reviewing the syslog entries on that node.

AWS advanced configuration

AWS Secrets Manager can be used to retrieve configuration settings at startup. The Secrets Manager entries should match the key names of the configuration file. The secret name can be defined with:

AWS_SECRETS_MANAGER_KEY=rumble/production

To ensure that runZero can access the Secrets Manager, ensure that an AWS_REGION is set in the configuration file, and that the instance has an IAM instance role with permission to read the Secrets Manager key or the AWS credentials have been specified directly in the configuration file. With a Secrets Manager configuration, only the AWS options need to be present in the configuration file and the rest of the settings will be loaded from the Secrets Manager key. This will require copying and defining various settings from the generated configuration file to the Secrets Manager key-value pairs.

Note that many settings need to be identical across the runZero cluster for web requests and authentication to work correctly (including session secrets, DB encryption key, and the console URL).

Updated
Previous: Offline mode configuration Next: Self-hosted troubleshooting