The task
In IO's Calling Mode, IO works as an application proxy that makes requests to remote services on behalf of an application. When IO trusts an application, it can apply secrets to requests that the application makes through IO. This secures secrets by keeping them out of application code and out of possession of application executables. Only IO needs to see these secrets, and as it applies them, IO can track and govern their use.
These secrets must be managed securely. To support that, IO now has a built-in repository that we call a keyring. The IO keyring stores secrets encrypted and, when IO is configured in a cluster, it protects them from loss by replicating them using the Raft algorithm.
Secret types
The IO keyring manages a variety of secret types:
- Tokens are single strings, usually representing API keys.
- Logins are username/password pairs, usually associated with websites.
- Certificates are X.509 digital certificates with associated private keys that can be used to secure websites with TLS.
- Keys are private keys that can be used for SSH and digital signatures.
- OAuth Clients contain the client ids and client secrets that are issued to applications by OAuth identity providers.
- ATProto Clients contain the private keys and identifying information used to identify AT Protocol OAuth clients.
Each secret type is represented with a Protocol Buffer encoding. Encodings are encrypted using a cluster-specific key, and only these encrypted secrets are stored in the keyring and passed around a cluster using Raft.
How IO encrypts secrets
Keychain secrets are encrypted using AES-256-CBC with HMAC-SHA256. This combines symmetric encryption with message integrity verification and is widely used by security frameworks and applications including the Bitwarden password manager.
The keys for this scheme are derived from a user-supplied passphrase that is run through the Argon2id key derivation function as implemented in golang.org/x/crypto/argon2. This produces the AES encryption key and the HMAC key. For FIPS compliance, a PBKDF2 generator is also available and is used when the IO_PBKDF2 environment variable is set to any value.
The user-supplied passphrase is ephemeral: it is not stored anywhere by IO and is used only on startup to generate keys. It is read from the IO_PASSPHRASE environment variable and other specification mechanisms may be added as needed.
When a secret is stored, its protobuf-encoded representation is first encrypted with AES and then this ciphertext is hashed. The stored encrypted values include a 16-byte initialization vector that's unique for each encrypted value, the encrypted ciphertext, and the HMAC-SHA256 signature of the initialization vector+ciphertext.
How IO distributes secrets with Raft
IO instances can be configured in clusters that maintain consistency using the Raft algorithm. Nodes are identified by a name (IO_NAME) and the addresses of peer nodes are configured using the peer configuration element. A cluster of three nodes can be mapped with the following configuration, which should be applied to each node.
peer "alfa" {
address = "https://alfa.agentio.dev"
}
peer "bravo" {
address = "https://bravo.agentio.dev"
}
peer "charlie" {
address = "https://charlie.agentio.dev"
}
Addresses can be any hostname served by the associated IO. IO listens for Raft messages using a gRPC API that is served on all ingresses.
Raft messages sent between cluster members are secured with an HMAC of the message body that uses the passphrase-derived MAC key. With this, only IOs that share a common passphrase can participate in a cluster. To resist replays, the HMAC includes a timestamp; requests with timestamps that are older than a specified lifetime (currently 60s) are rejected. Repeated failures can trigger automatic blocking of senders using an Envoy filter.
Why this is called a "keyring"
IO's keyring is designed to provide durable safe storage of secrets used by applications. It is intended for working storage and not archiving; only the secrets needed by IO-supported applications are expected to be included. But a properly-operated cluster is expected to provide durable encrypted storage, and arbitrary secrets can be copied in and out of a cluster using IO's SSH control interface.