Skip to content

Fenced publication

For publishers whose ownership moves between processes (WebSocketManager connections). Spec: openspec change 2026-10-15-fenced-publication.

It runs on Dapr building blocks only: any Dapr state store with the ETAG and TRANSACTIONAL capabilities (the same ones an actor state store needs), and the configured pub/sub. The gateway checks the store's capabilities through the sidecar's metadata and answers FAILED_PRECONDITION if they are missing. PublicationStateStore names the store (default statestore).

Fence

ChangePublicationFence manages a per-connection fence: token, active flag and lease (lease_seconds).

  • A fresh new_token claims the fence when expected_token is current, the fence is inactive or absent, or its lease expired; otherwise ABORTED.
  • new_token == expected_token renews (active) or revokes (!active), for the current holder only, and never reactivates an inactive fence.
  • Every admission renews the lease.

A holder that crashes after claiming but before recording its token delays a successor by at most one lease. Lease times use the gateway's UTC clock; leases last minutes, so ordinary clock skew does not matter.

Admission

PublishFenced writes the CloudEvent under a fresh key, then appends a reference to it to the fence with an ETag-guarded write of the fence record. That single-key guarded write is what admits the frame, so a replaced or inactive token is never admitted (FAILED_PRECONDITION).

Nothing relies on multi-key transactions being all-or-nothing: Dapr's Redis store, for one, applies the other writes of a transaction whose guard failed. A failed admission therefore leaves at most an unreferenced item, which the gateway deletes on the spot.

At most 1,000 frames per connection may be pending; beyond that only that connection gets RESOURCE_EXHAUSTED.

Delivery

Each connection has one sequence. Frames are published through Dapr with rawPayload in admission order, and every frame an old owner admitted is delivered before any frame of its successor.

  • The gateway delivers inline, before PublishFenced returns. The replica delivering a connection's frames holds its drain lock (3 s, renewed with every delivery) while frames keep coming. Frames sent to another replica meanwhile wait until that lock expires.
  • A recovery sweep (every second, FencedPublicationDispatcher) walks the fence index. It delivers frames whose drainer died or whose publication failed, and removes fences that are inactive or expired, fully delivered and older than two leases.
  • Delivery is at-least-once. A failed publication keeps that frame ahead of later ones and does not repeat the frames before it.

Steady state per frame: one bulk read, two writes and the publish. Measured with the opt-in benchmark (FENCE_BENCH=true FENCE_DAPR_TESTS=true, one connection publishing sequentially, local daprd), fencing added about 1 ms per frame on Redis and 2 ms on PostgreSQL. Expect more where the store is across a network.

Keys

From the virtufin.service.api naming fragment: sys.api.publication.fence.<connection_id> (with the index sys.api.publication.fence.index), sys.api.publication.outbox.<connection_id>.<item_id> and sys.api.publication.drain.<connection_id>.

Limits

Loss or rollback of the state store loses fences. This is publication safety, not physical WebSocket exclusivity, nor caller authorization beyond the gateway key model. Unfenced PublishEvent remains for other callers.