Explorer security model

runZero Explorers are responsible for running network scans, sampling traffic for passive discovery, and implementing integrations within their deployed environment. This document describes the security model of runZero Explorers.

Development

runZero Explorers are built using the Go programming language. This language provides cross-platform compatibility and memory safety by default. The runZero engineering team updates the Explorer code and it’s dependencies on a continuous basis to ensure that any dependency-level vulnerabilities are resolved quickly. All engineers are required to use MFA to access our Git repository. All code changes go through manual review before being merged to our development branch, deployed to a test environment, reviewed again before being merged to our staging branch, and deployed to a staging environment. During the build process, Explorer binaries are signed with a private Ed25519 key and the signature is stamped into the end of the binary. For Windows binaries, an Authenticode signature is also applied, using an extended-validation certificate stored in a cloud-based HSM. Once all automated tests pass and any changes to behavior are confirmed, the staging environment binaries are tagged for release, and moved to the production deployment location in AWS S3.

Deployment

runZero Explorers are initially provisioned by hand through direct downloads from the runZero Console (whether cloud or self-hosted). During this download process the link token and organization ID is stamped into the binary while still preserving the code signatures. The binary will run on the deployed system, connect back to its stamped Console URL, validate TLS, and register. The registration process starts with the Explorer generating a unique host ID using the operating system’s random source. The ID is stored with limited permissions on the Explorer’s file system (non-Windows) or registry (Windows) and will be used to confirm that Explorer’s identity for future connections.

During the first connection from a new Explorer, the Console records the Explorer-generated Host ID, and establishes a database record for that instance. Other Explorers, even those running on the same system, are not able to claim to be this unique Explorer record without also possessing and verifying the Host ID. New Explorers, by default, do not receive any data from the Console and are not sent any tasks (including those with credentials) unless manually selected by the user. As a result, its relatively safe to have untrusted Explorers registered, as long as they aren’t used for tasks that include sensitive data (typically integrations with credentials).

Note that once an Explorer has the Organization ID, the link token (Download Token) is not needed for the Explorer to connect and register. runZero plans to enforce Explorer use of the current Download Token in the near future, but this is not yet a requirement, as there are few if any security risks exposed by unauthorized Explorer registrations. Once the current Download Token becomes a requirement for registration, older Explorers will not be able to reconnect after the Download Token is rotated. Active Explorers should always receive the latest Download Token as part of the upgrade request.

Commands

The runZero Console provides the ability to send new tasks and specific requests to Explorers. Tasks are limited to active scans, passive traffic sampling, or integrations. Integrations can use custom code, written in the Starlark language, and while they have access to the network, they do not have access the Explorer host machine or file system. Outside of tasks, Explorers accept requests to uninstall, update, and post diagnostics (limited to stack traces and Explorer attributes like operating system and hardware specifications). Explorers do not implement any commands that allow network tunneling or host access to the Explorer’s machine.

Credentials are only sent to an Explorer if they are associated with an integration task and if the integration task scope matches the CIDR or other configuration for those credentials. Credentials are never sent to runZero-hosted External Explorers regardless of the task configuration. Credential parameters are sent across in the same TLS connection as the request, but are also encrypted to a unique per-process AGE encryption key. Even if the runZero communication is logged through a TLS proxy and pinned CA, credentials cannot be recovered without also extracting the private, in-process-only, and temporary AGE key.

Updates

The runZero Console verifies that Explorers are running the latest version before scheduling a task (scan, passive, or integration). If the Explorer is out of date, it will be updated to the latest version first, and then the task will be scheduled. If the update takes longer than an internal threshold, or fails due to things like a read-only filesystem on the Explorer side, the task will be scheduled anyways after this threshold is reached.

The update process receives a URL to the new binary. The Explorer connects to this URL, verifies TLS parameters, and downloads the binary to a temporary directory. The Explorer then verifies that the binary has a valid code signature embedded and rejects the update if this verification fails. If the code signature is valid, the Explorer launches the new binary in update mode, which typically replaces the running service with the new executable.

It is possible and supported for the update to change the Organization or even Console for the Explorer service. This feature is used to move Explorers between installations and organizations. In either case, the TLS connection must be valid, and the binary must have a valid code signature. This verification process also rejects version downgrades by comparing the signed version metadata in the binary against the running version.

Uninstallation

runZero Explorers are automatically uninstalled when their associated runZero Organization is deleted. Explorers can also be uninstalled from the runZero Console or by using the “uninstall” CLI option of the Explorer binary. Explorers try to clean up as many artifacts as possible during the uninstallation process. Any artifacts remaining post-uninstall can be manually removed as needed (common on Windows systems where the OS locks the program files and directory while they run).

Defense in Depth

runZero strives to protect customer data by layering multiple defenses into the runZero architecture:

• An attacker that is able to push code to our internal repository would still need to pass review before it can be merged into our protected staging branch.

• An attacker that is able to compromise our binary distribution point (AWS S3) would still need a valid signing key before any customer updates could be affected. Downgrade attacks are prevented by Explorer-time version comparisons of authenticated and signed metadata.

• An attacker that obtains the download link for a customer can register new Explorers, but is not able to obtain any data from the Console unless those Explorers are specifically chosen for a task.

• An attacker that is able to send commands to an Explorer (by compromising a Console) is limited to a few relatively-safe commands and is not able to force downgrades or arbitrary code execution on the Explorer host.

• An attacker that is able to MiTM the TLS communication with the Console is not able to decrypt credentials used in integration tasks.

• An attacker that is able to schedule a custom integration task with a malicious Starlark script can interact with the network from the perspective of the Explorer, but is not able to interface with the Explorer host.

In all of these situations, runZero also operates extensive logging and alerting, so that any malicious action can be investigated, and the impact of any issue readily assessed.