Commonplace

Commonplace Protocol Introduction and Overview

Overview

The Commonplace Protocol is a custom binary messaging protocol designed for efficient communication between Commonplace clients and servers. The protocol prioritizes performance, extensibility, and developer ergonomics through careful design choices.

Key Design Principles

Binary Efficiency: The protocol uses a compact binary format optimized for network transmission and minimal overhead. Message envelopes include only essential headers (protocol version, message type, and payload length) to reduce wire overhead.

Zero-Copy Architecture: Wire format structures closely mirror in-memory representations, minimizing serialization overhead and memory copying. The protocol uses bincode for serialization, maintaining structural similarity between network and memory formats.

Forward Compatibility: Built-in versioning support allows protocol evolution without breaking existing clients. Message types are explicitly versioned, and the envelope format supports protocol version negotiation.

Protocol Structure

The protocol consists of three main layers:

  1. Transport Layer: Handles connection management and raw data transmission over TCP
  2. Message Layer: Provides typed message envelopes with routing and versioning information
  3. Application Layer: Implements domain-specific operations for cards, collections, and schemas

Message Categories

  • Connection Management: Server discovery, client authentication, and connection lifecycle
  • Card Operations: CRUD operations, Card Type changes, and metadata updates for individual cards
  • Collection Operations: Workspace management and bulk card operations
  • CardType Operations: Type definitions and field constraints management
  • Bulk Operations: Transaction-like operations for multiple entities

The protocol is implemented in Rust using serde for serialization, with extensive use of tokio for async I/O.

This document provides a complete reference for the Commonplace protocol, organized by functional categories. All commands listed are currently implemented unless otherwise noted.

Protocol Overview

  • Protocol Version: 2
  • Max Message Size: 16MB
  • Serialization: Bincode
  • Transport: TCP with message envelopes
  • Connection Management: TCP keepalive (30 second interval)

Message Envelope Structure

All messages are wrapped in a common envelope:

  • Protocol version (1 byte)
  • Message type (1 byte)
  • Payload length (4 bytes, little-endian)
  • Payload data (variable length)

The current protocol version is 2. Messages with unsupported protocol versions are rejected.

Protocol Versioning: The envelope protocol version is the single source of truth for message format compatibility. Server capabilities (rather than a separate protocol version field) indicate supported features and future protocol versions.

Endianness: Both envelope and payload use little-endian byte order for optimal performance on modern platforms (amd64, arm64).

Message Type Reference

The following message types are currently implemented:

Request Messages

  • GetServerInfo - Retrieve server information and capabilities
  • Connect - Establish authenticated session
  • Disconnect - Close session gracefully
  • Ping - Test connection latency

Card Operations

Note: all card operations are scoped to a given Collection, which must always be provided.

  • CreateCard - Create new card of a Card Type in a collection. Adds new logical card, new card instance and new card reference in the identified collection.

  • GetCard - Retrieve card by collection reference (contextual access). Includes both card instance data+metadata AND Card Reference metadata.

  • 'GetMultipleCards' - Retrieve many cards in a single operations, including card instance data+metadata and card reference metadata. Requires a query specification for cards to be returned. This is also where we would handle querying by location in a spatial collection.

  • UpdateCard - Update card (contextual access). Updates card data+metadata AND Card Reference metadata. Results in new card instance using append-only, logical card points to new card instance, and potentially new reference metadata pointing to logical card.

  • CopyCard - Create additional card reference, in the same or different target collection. Increments logical card reference count by 1.

  • CopyAsNewCard - Create new logical card pointing to the same card instance id as source card. Create new card reference pointing to this new logical card. Reference count for new logical card = 1.

  • RemoveCard - remove a card reference from a collection, using lock-free approach. Reduces logical card reference count by 1.

  • ChangeCardType - Migrate card to different card type

Link Operations

  • CreateLink - Create relationship between cards or between cards and collections
  • 'RemoveLink' - Remove relationship between cards or between cards and collections

Collection Operations

  • CreateCollection - Create new collection
  • GetCollectionContents - Retrieve collection contents (options for card references only, child collections only, all contents, filtering/querying for content items.) Note: this never returns actual card data, but only card reference metadata and/or child collection metadata. Requires a query specification for cards to be returned. This is also where we would handle querying by location in a spatial collection.
  • GetCollectionSummary - Get collection metadata and summary data only
  • 'RemoveCollection' - Deletes collection and all cards within it.
  • 'CopyCollection' - create a copy of a collection in the target collection. All Cards in the source collection are created in the target collection, effectively increasing each of the underlying logical card's reference counts by 1.

Card Type Operations

  • CreateCardType - Create new card type definition
  • GetCardType- Retrieve specific card type definition
  • ListCardTypes - List available card types
  • 'UpdateCardType' - Change a card type or card type metadata

Response Messages

  • ServerInfoResponse - Server information response
  • ConnectResponse - Connection establishment response
  • Pong - Ping response with timing

Wire Format

Message Envelope (6 byte header + payload):

Byte(s) Field Description
0 Protocol Version Protocol version (1 byte)
1 Message Type Message type identifier (1 byte)
2-5 Payload Length Length of payload data (4 bytes, little endian)
6+ Payload Data Variable length payload (little endian) 

Example - Ping with "test" payload:
01 0D 04 00 00 00 74 65 73 74
│  │  └─────────┘ └─────────┘
│  │      │           │
│  │      │           └─ "test" (0x74657374)
│  │      └─ Length: 4 (0x00000004)
│  └─ Message Type: Ping (0x0D = 13)
└─ Protocol Version: 1