skip to content

Federation

xmsg federates across hosts, virtual machines, and Kubernetes pods over mutual TLS (mTLS) on the /fed/v2 wire protocol.

A node’s name is derived from its cryptographic or OS identity — never assigned by a free-form flag (--host-label and XMSG_HOST_LABEL are refused at startup):

  1. CA-signed identity (--fed-cert with URI SAN): Derived from the single SPIFFE URI Subject Alternative Name (SAN) on the node’s leaf certificate:
    • spiffe://<trust-domain>/host/<host>/user/<user> →\rightarrow <user>.<host> (for example, alice.workstation).
    • spiffe://<trust-domain>/ns/<ns>/sa/<sa> →\rightarrow <sa>.<ns> (for example, genie-guard.genie).
    • Dots (.) inside <user>, <host>, <ns>, or <sa> are refused.
  2. Pinned key (--fed-cert without URI SAN): Derived from the public key itself: the first 16 characters of the lowercase RFC 4648 base32 encoding of the SHA-256 digest of the key’s SubjectPublicKeyInfo DER (for example, k5x3q7mzv2ndp4ha).
  3. No federation identity (local-only): When --fed-cert is not set, the node name is <user>.<host> from the effective UID’s OS username and the short hostname (cut at the first .). This name never travels on the wire.

Envelopes on /fed/v2 carry canonical node names, and the receiving node attributes every inbound message to the canonical name of the peer entry that authenticated the TLS connection (unless relayed by an authorized forwarding hub). When typing @node locally, you may use either the peer’s canonical name or your local nickname for a pinned peer (the key in your peers.json). Exact canonical names and nicknames take precedence over a bare-host selector such as workstation. With that selector, list(node="workstation") queries all matching nodes in the current view. A send(to="guard@workstation", text="...") selects a single match; multiple matches return ambiguous. With no match, the explicit destination follows a configured default route or is unknown. Use a canonical name to select one user on a shared host. See node selectors.

Each entry in peers.json uses exactly one trust mechanism:

  • Pinned certificates (pin): Authenticates a peer by matching its SHA-256 SPKI certificate fingerprint (sha256:...). Ideal for direct 1:1 peerings, laptops, and edge nodes without PKI.
  • CA-signed certificates (ca + identities): Validates the peer’s X.509 certificate chain against a PEM trust anchor bundle (ca) and requires the leaf certificate’s single URI SAN to match the entry’s single concrete URI (identities). Common Name (CN) and DNS SANs are never used for identity. Ideal for SPIFFE/SPIRE, cert-manager, step-ca, or fleet CA deployments where certificates rotate automatically.
{
"laptop": {
"address": "192.168.1.50:7788",
"pin": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"allow": ["send", "reply", "list"],
"from": ["192.168.1.0/24", "fd00::/8"]
},
"worker.prod": {
"address": "10.0.0.12:7788",
"ca": "/etc/xmsg/spire-bundle.pem",
"identities": [
"spiffe://example.org/ns/prod/sa/worker"
],
"allow": ["send", "reply"]
}
}

Naming and non-overlap invariants (xmsg check-peers)

Section titled “Naming and non-overlap invariants (xmsg check-peers)”

Both xmsg serve and the standalone validator xmsg check-peers <peers.json> [--fed-cert <cert.pem>] enforce:

  • A CA peer entry’s key must equal the node name derived from its identities URI (worker.prod above).
  • A pinned peer entry’s key is either its 16-character key-derived name or a local nickname (which may not contain . and may not be 16 characters of [a-z2-7]).
  • All entry keys, derived names, and this node’s own name share a single namespace with no collisions or duplicate JSON keys.
  • At handshake time, a certificate must match exactly one peer entry; overlapping CA entries or a certificate matching both a pinned entry and a CA entry fail closed.

Peer configuration reference (--peers-file)

Section titled “Peer configuration reference (--peers-file)”
FieldTypeDescription
addressstringTarget host:port endpoint. Required.
pinstring (optional)Expected SHA-256 certificate fingerprint (sha256:...). Mutually exclusive with ca.
castring (optional)Path to PEM trust anchor bundle. Mutually exclusive with pin; requires identities.
identitieslist[string] (optional)Single concrete SPIFFE URI SAN (["spiffe://..."]). Required with ca.
allowlist[string]Permitted capabilities ("send", "reply", "list", "inbox", "forward", "relayed").
fromlist[string] (optional)Permitted source IP addresses or CIDR blocks for inbound connections. Empty [] is refused.
leafbool (optional)When true, messages and list queries for this peer queue in the hub’s durable peer_mailbox for long-polling instead of dialing out. Default false.
dial"outbound" (optional)Spawns a background pull loop on this node that dials out to address to pull messages, answer list queries, and poll replies. Mutually exclusive with leaf: true.
principalslist[string] (optional)Inbound sender filter: service name, instance ID, or anon:<from>. Absent or [] permits every principal; a nonempty list restricts senders to its entries.
targetslist[string] (optional)Local recipient filter for messages and list: service name or instance ID, never an instance label. Absent permits all; [] permits none.
default_routebool (optional)Use this peer for an explicit ref@node destination that is not a direct peer. At most one entry may set it to true; direct routes take precedence. See hub forwarding.

When a caller runs list without a node filter, xmsg queries this node and every peer in peers.json concurrently (5-second timeout per peer):

  • Direct peers are queried over GET /fed/v2/sessions.
  • Dial-out clients ("leaf": true) receive a {"kind":"list","query":"..."} item in their hub mailbox, waking their long-poll, and post their filtered recipient list back to POST /fed/v2/list/{query}.
  • If a peer’s allow list for the asker omits "list", it refuses with 403 Forbidden and the asker reports that node in nodes as status: "unreachable", detail: "withholds list".
  • Host-internal details (pid, cwd, harness, local paths) never cross /fed/v2; the asker drops any entry whose reported user does not match the peer certificate’s user.

Clearing principals to [] removes the sender restriction; it does not revoke every sender. To stop an operation, remove it from allow. To expose no local recipients, use targets: []. A network connection still needs to pass the peer’s TLS identity, operation rights and any from filter. allow, principals and from gate traffic this node accepts from that peer. A hub also applies a destination peer’s targets filter when forwarding or queueing for it, and carries that bound into the forwarded message.

The empty-list rules are implemented in PeerConfig parsing and inbound checks.

palette
dark
light
↑↓ select↵ apply esc close

Palettes adapted from Catppuccin (Macchiato) (MIT), Tokyo Night (Apache-2.0), gruvbox (MIT), Catppuccin (Latte) (MIT), Rosé Pine (Dawn) (MIT).