X3DH and the Triple Envelope: the full mathematics of an Enchant session

This is the story of a single message. When you press send in Enchant, a chain of cryptographic machinery runs before anything touches the network. No shortcuts, no "trust us" — every byte on the wire is the output of equations that anyone can read and anyone can verify. This post walks the full mathematics of an Enchant session, exactly as it is implemented in the LibEnchant cryptographic library: the handshake that turns public keys into a secret, and the Envelope that turns that one secret into a fresh key for every single message.

What this post is for. If you read nothing else, read the bottom-line table: seven security properties, each one produced by a specific mechanism, each one verifiable. Everything above it is the derivation of that table.

The cast of keys: what each device holds

Every Enchant installation generates a small keyring. Each key has one job, and no key does anyone else's job:

The signed pre-key is signed with Ed25519 over the exact bytes $\mathrm{prekey_id}(4) \parallel \mathrm{SKP}(32)$ — and the client verifies that signature on-device before trusting a fetched bundle. An attacker who controls the directory can substitute their own pre-key, but they cannot forge the signature without the victim's identity private key. This is the first and most important thing the handshake does: it makes the directory useless to an attacker who is not the directory's owner.

The LibEnchant API surface for this is enchant_prekey_generate_signed, enchant_prekey_generate_batch, and enchant_prekey_generate_kyber_batch — the C functions that produce the bundles, signed with the identity key on-device, never on the server.

The handshake: X3DH

What it is. The Extended Triple Diffie-Hellman handshake — a construction for deriving a shared secret between two parties who have never met, using only public keys fetched from a directory.

Why it exists. The server is the middleman. It sees both parties' public keys. Any protocol where the shared secret is derived from a single DH exchange (Alice's ephemeral $\times$ Bob's identity) is also derivable by the server, because the server saw both halves. X3DH mixes four DH outputs so that the secret is not derivable from any one pair, and the one-time pre-key makes it un-derivable even by the server.

How it works. Alice wants to talk to Bob. She fetches Bob's bundle: his identity key $\mathit{IK}_B$, his signed pre-key $\mathit{SKP}_B$, and (usually) one one-time pre-key $\mathit{OPK}_B$. She generates her own ephemeral $\mathit{EK}_A$ and computes four X25519 scalar multiplications — each one collapses two private/public keys into a 32-byte shared point:

   Alice's side                         Bob's side (mirror)
 
   IK_A (private) ──×── SKP_B (public)  SKP_B (priv) ──×── IK_A (pub)   → DH1
   EK_A (private) ──×── IK_B (public)   IK_B (priv)  ──×── EK_A (pub)   → DH2
   EK_A (private) ──×── SKP_B (public)  SKP_B (priv) ──×── EK_A (pub)   → DH3
   EK_A (private) ──×── OPK_B (public)  OPK_B (priv) ──×── EK_A (pub)   → DH4 (one-time)
        │                                       │
        └───────────  both compute the same IKM  ───────────┘
                  IKM = DH1 ‖ DH2 ‖ DH3 [‖ DH4]

Written out:

$$\mathrm{DH}_1 = \mathrm{X25519}(\mathit{IK}_A, \mathit{SKP}_B) \qquad \text{Alice identity} \times \text{Bob signed pre-key}$$

$$\mathrm{DH}_2 = \mathrm{X25519}(\mathit{EK}_A, \mathit{IK}_B) \qquad \text{Alice ephemeral} \times \text{Bob identity}$$

$$\mathrm{DH}_3 = \mathrm{X25519}(\mathit{EK}_A, \mathit{SKP}_B) \qquad \text{Alice ephemeral} \times \text{Bob signed pre-key}$$

$$\mathrm{DH}_4 = \mathrm{X25519}(\mathit{EK}_A, \mathit{OPK}_B) \qquad \text{Alice ephemeral} \times \text{Bob one-time key}$$

The input keying material is the concatenation of the 32-byte outputs:

$$\mathrm{IKM} = \mathrm{DH}_1 \parallel \mathrm{DH}_2 \parallel \mathrm{DH}_3 , [,\parallel \mathrm{DH}_4,]$$

Why all four? Each DH closes a different hole:

• DH1 authenticates Alice. Only Alice owns $\mathit{IK}_A$'s private half. If Bob were to receive a forged bundle claiming to be Alice, DH1 would be computed with the attacker's identity — and the resulting session key would differ from the one honest Alice derives.
• DH2 + DH3 bind the handshake to Bob's server-published keys and mix in fresh ephemeral randomness, so no two sessions ever share a key, and no static key alone can predict a session.
• DH4 is the anti-server seal. The server watched DH1–DH3 happen — it can compute them all itself. But the one-time pre-key is consumed from the directory after one use and deleted. The server cannot compute $\mathrm{DH}_4$ because the private half no longer exists.

Raw DH outputs are not keys. They are mixed through HKDF-SHA256, salted with 32 zero bytes, with a domain-separation string so that the same material can never be confused across protocols:

$$SK = \mathrm{HKDF\text{-}SHA256}(\mathrm{IKM},\ \mathrm{salt}=0^{32},\ \mathrm{info}=\text{``X3DH"},\ L=32)$$

Then the session Envelope is seeded — the root key and the first chain key, again with a domain-separated label:

$$(RK \parallel CK) = \mathrm{HKDF\text{-}SHA256}(SK,\ \mathrm{salt}=0^{32},\ \mathrm{info}=\text{``EnvelopeRatchet"},\ L=64)$$

Bob performs the mirror operations — swapping the roles in every DH — and lands on the same $\mathrm{IKM}$, the same $SK$, the same root. That uniformity is the handshake: two parties computing the same scalar from opposite sides. The LibEnchant C implementation of this exact sequence is enchant_session_manager_establish (X3DH) and enchant_session_manager_establish_pqxdh (post-quantum), with the full bundle validation in enchant_session_builder_process_bundle.

PQXDH: the same skeleton, one extra seal

The handshake above is classical. A future quantum computer running Shor's algorithm would dismantle every X25519 step. So the modern Enchant session setup — protocol version 4, the current envelope version — is PQXDH (protocol version 3 is the legacy X3DH-only mode). The classical DH terms are never dropped; the lattice KEM secret is appended, and everything is mixed in one HKDF call:

$$\mathrm{IKM}_{\mathrm{PQXDH}} = \mathrm{FF}^{32} \parallel \mathrm{DH}_1 \parallel \mathrm{DH}_2 \parallel \mathrm{DH}_3 , [,\parallel \mathrm{DH}$$4,] \parallel \mathit{SS}{\mathrm{KEM}}

The $\mathrm{FF}^{32}$ prefix is thirty-two bytes of 0xFF — a discontinuity marker. It guarantees that a classical X3DH session and a PQXDH session can never derive the same keys, even if they somehow reused identical DH material. The KEM secret $\mathit{SS}_{\mathrm{KEM}}$ is the 32-byte shared secret from ML-KEM (1024-bit in the handshake; see our post-quantum post):

$$K = \mathrm{HKDF\text{-}SHA256}(\mathrm{IKM}_{\mathrm{PQXDH}},\ \mathrm{salt}=0^{32},\ \mathrm{info}=\text{``EnvelopeText_X25519_SHA-256_ML-KEM-1024"},\ L=96)$$

$$\rightarrow RK \parallel CK_{\mathrm{send}} \parallel pqk$$

An attacker must now break both the elliptic-curve DH and the lattice KEM to recover the session. One seal more on the envelope. A multi-device variant, X4DH, additionally makes the one-time pre-key mandatory and uses the label "EnvelopeText_X25519_SHA-256_ML-KEM-1024_X4DH".

The Triple Envelope

The handshake gives a shared secret once. A messenger sends thousands of messages. The Envelope converts that one secret into a fresh key for every message, so a key leak at time $T$ reveals nothing about messages before or after $T$. This is where the name comes from: the session is an envelope (the handshake secret), and every message rides in a new, smaller envelope nested inside it.

What it is. A symmetric turn: from a chain key $CK$, each message derives two things — the next chain key (so the chain moves forward) and a message key (so the message is encrypted). The key turn on top re-negotiates fresh ephemerals periodically, so the whole state machine stays forward-secret.

Why it works. Two HMAC labels. The chain key is never directly used as a message key; instead, the same key produces two labeled outputs, so the two kinds of key can never be confused even if one is leaked:

$$CK' = \mathrm{HMAC\text{-}SHA256}(CK, 0\mathrm{x01}) \qquad \text{next chain key}$$

$$MSk = \mathrm{HMAC\text{-}SHA256}(CK, 0\mathrm{x02}) \qquad \text{message seed}$$

The message seed is expanded into a message key and a nonce through HKDF, with the library's own domain-separation label ("enchant_TripleRatchet_MessageKey_20240101"):

$$(MK \parallel \mathit{nonce}_{24}) = \mathrm{HKDF\text{-}SHA256}(MSk,\ \mathrm{salt}=0^{32},\ \mathrm{info}=\text{``enchant_TripleRatchet_MessageKey_20240101"},\ L=56)$$

That gives a 32-byte message key and a 24-byte nonce — the exact inputs XChaCha20-Poly1305 needs. Encryption binds the 4-byte counter into the AEAD as associated data:

$$C = \mathrm{XChaCha20\text{-}Poly1305\text{-}Enc}(MK, \mathrm{plaintext}, \mathit{nonce}, \mathrm{aad}=\mathrm{counter} \parallel \mathrm{metadata})$$

$$\text{message on the wire:} \quad \parallel \underbrace{\mathrm{counter}}$${4\ \mathrm{bytes,\ clear}} \parallel C \parallel \underbrace{\mathrm{tag}}{16\ \mathrm{bytes}} \parallel

The counter-in-AAD trick is subtle and important: a captured ciphertext cannot be remapped to a different chain position, because the AEAD tag would no longer verify. The 16-byte tag gives a $2^{-128}$ forgery probability — the chance of a random tampered message passing verification is smaller than the chance of a cosmic ray flipping the right bit at the right time.

The key turn: rotating the walls

Why it exists. Chains give forward secrecy for past messages against a stolen chain key. But if the root key itself is stolen, an attacker could ride the chain forward. The key turn prevents that: it periodically runs a fresh ephemeral DH exchange and re-derives the root, so compromise at time $T$ cannot decrypt anything after $T$.

At each turn, both sides generate brand-new ephemeral keys and compute two shared secrets — the classical half and the post-quantum half:

$$ec_{ss} = \mathrm{X25519}(\text{our new ephemeral},\ \text{their ephemeral})$$

$$pq_{ss} = \mathrm{ML\text{-}KEM\text{-}768}_{\mathrm{Encap/Decap}}(\text{peer ciphertext or our key})$$

If PQ keys are unavailable, the fallback derives the quantum half from the classical one through HKDF with its own label ("enchant_TripleRatchet_PQ_Derive_20240101"), so the Envelope never stalls. Both halves are combined in a single HKDF call:

$$(RK' \parallel CK'$${\mathrm{send}} \parallel CK'{\mathrm{recv}}) = \mathrm{HKDF\text{-}SHA256}(ec_{ss} \parallel pq_{ss},\ \mathrm{salt}=0^{32},\ \mathrm{info}=\text{``enchant_TripleRatchet_Combine_20240101"},\ L=96)

The old receive chain is archived (up to a bounded number of previous chains) so out-of-order messages that crossed a turn can still be decrypted. Because both sides generate new keys at every step, a compromise at point $T$ cannot decrypt anything after $T$. That is forward secrecy: the stolen phone reveals only its current epoch. The message "Envelope #418" stays sealed forever.

Skipped keys and out-of-order delivery

Messaging is not a perfectly ordered channel. If message 7 arrives before message 5, the receiver derives keys for messages 5 and 6 from the chain, stores them, and decrypts 7 when its predecessors arrive. LibEnchant bounds this bookkeeping — these are protocol constants, not implementation details:

Once a counter is decrypted, it is marked seen forever in the current chain and across turns — a replayed ciphertext is rejected instead of re-accepted. Replay protection is not a bolt-on; it is enforced in the decrypt path before any key derivation begins.

What the whole affair gives you

None of this requires trusting Enchant's servers. The server only ever holds sealed envelopes it cannot open — which is exactly the story of our zero-access server post. The math is the product. The product is the math. And the LibEnchant library, in constant-time C with no branching on secret data, is where that math lives.