Commonplace

Collection Management and Organisation

This document describes how collections are managed, organized, and manipulated within Commonplace, covering both the foundational operations needed for a proof of concept and advanced organizational features for future development.

Collection Lifecycle Management

Collection Creation

Basic collection creation involves establishing a new container for card references:

User-Initiated Creation: User creates a new collection through the interface, providing a name and optional description.

Automatic Creation: System creates collections automatically in certain contexts (e.g., "Recently Deleted" collection).

Template-Based Creation: Collections can be created from predefined templates with specific schemas and initial structure.

Collection Identity: Each collection receives a unique identifier that remains constant throughout its lifetime.

Collection Deletion and Cleanup

Soft Deletion: Collections are typically moved to a "Recently Deleted" area rather than immediately destroyed.

Reference Management: When a collection is deleted, card references within it are handled according to reference counting rules.

Cascade Behavior: System handles cleanup of nested collections and their contents appropriately.

Recovery Options: Users can restore accidentally deleted collections within a specified time window.

Collection Hierarchy and Nesting

Hierarchical Organization

Collections can contain other collections, creating navigable hierarchies:

Parent-Child Relationships: Collections can be nested within other collections to create organizational structures.

Navigation Patterns: Users can drill down through collection hierarchies to find specific content.

Permission Inheritance: Nested collections can inherit permissions and settings from their parent collections.

Cycle Prevention: System prevents circular references where a collection contains itself directly or indirectly.

Organizational Patterns

Project-Based Hierarchies: Organize collections around projects, with sub-collections for different aspects or phases.

Topic-Based Hierarchies: Group related collections under broader topic areas.

Temporal Hierarchies: Organize collections by time periods (years, quarters, months).

Access-Based Hierarchies: Structure collections based on who needs access to different types of information.

Collection Display Modes and Views

Mode Management

Collections support multiple display modes that affect how cards are presented:

List Mode: Traditional table-based view with sorting and filtering capabilities.

Spatial Mode: Canvas-based view where cards can be positioned freely in 2D space.

Summary Mode: Compact overview showing aggregate information and key metrics.

Mode Persistence: Collection display mode preferences are saved and restored when users return.

View Customization (Future Enhancement)

Saved Views: Users can save specific combinations of filters, sorts, and display settings as named views.

View Sharing: Saved views can be shared with other users who have access to the collection.

Default Views: Collections can specify default views for different user roles or contexts.

Dynamic Views: Views that automatically update based on changing criteria or data.

Collection Metadata and Properties

Basic Properties

Identification: Name, description, unique identifier, and creation information.

Display Settings: Default mode, column configurations, and visual preferences.

Organization: Tags, categories, and hierarchical position information.

Statistics: Card count, last modification date, activity metrics.

Extended Metadata (Future Enhancement)

Access Control: Permissions, sharing settings, and visibility controls.

Workflow Integration: Status tracking, approval chains, and process management.

Analytics: Usage patterns, access frequency, and performance metrics.

Custom Properties: User-defined metadata fields specific to organizational needs.

Reference Management in Collections

Card Reference Tracking

Reference Counting Implementation:

  • Server maintains an index mapping card_id -> reference_count
  • Each collection stores an array of card references with metadata
  • Reference count incremented when card added to any collection
  • Reference count decremented when card removed from any collection
  • Card instance deleted when reference count reaches zero (moved to "Recently Deleted")

Reference Metadata Storage: Each reference includes:

  • reference_id: Unique identifier for this specific reference
  • card_id: The card instance being referenced
  • schema_id: Schema identifier for efficient client-side caching and loading
  • position: 2D/3D coordinates within the collection
  • local_display_overrides: Collection-specific display settings
  • local_annotations: Notes or tags specific to this reference context
  • link_endpoints: Inlet/outlet connections for this reference
  • schema_context: Information about expected schema compatibility within collection

Cross-Collection References:

  • Same card instance can be referenced by multiple collections
  • Each reference is independent with its own position and metadata
  • Card instance changes visible through all references immediately
  • Reference-specific changes (position, annotations) affect only that reference
  • Schema changes to card instances affect all references but may have collection-specific display impacts

Reference Integrity Mechanisms:

  • Atomic operations for reference creation/deletion with count updates
  • Orphan detection for references pointing to non-existent card instances
  • Reference validation during collection loading
  • Automatic cleanup of invalid references during maintenance operations

Reference Operations Implementation

Adding References:

1. Validate card instance exists and schema is accessible
2. Load card instance to get schema_id for the reference
3. Check schema compatibility with collection constraints (if any)
4. Create new reference record with unique reference_id, card_id, and schema_id
5. Add reference to collection's reference array
6. Increment card's reference count
7. Update collection metadata (card count, schema diversity, modification time)
8. Update schema usage statistics for the collection
9. Notify clients of collection change with schema information

Removing References:

1. Locate reference in collection by reference_id
2. Remove reference from collection's reference array  
3. Decrement card's reference count
4. If reference count reaches zero:
   - Move card instance to "Recently Deleted" collection
   - Create reference in "Recently Deleted" with deletion metadata
5. Update collection metadata
6. Notify clients of collection change and card lifecycle events

Moving References Between Collections:

1. Remove reference from source collection (without decrementing count)
2. Add reference to destination collection (without incrementing count)
3. Update reference metadata for new collection context
4. Reference count remains unchanged (same card, different location)
5. Notify clients of changes in both collections

Copying References (creating additional references to same card):

1. Load source reference to get card_id and schema_id
2. Validate target collection accepts card's schema type
3. Create new reference record with new reference_id, same card_id, and schema_id
4. Add reference to target collection
5. Increment card's reference count
6. Update schema usage statistics for target collection
7. Reference now exists in both source and target collections with schema information

Detaching Instance (creating independent copy of card for this reference):

1. Load current card instance data and schema from reference's card_id and schema_id
2. Generate new unique card_id for detached instance
3. Create new card instance with copied data, same schema_id, and new card ID
4. Update reference record to point to new card_id (schema_id remains same initially)
5. Decrement reference count for original card_id
6. Set reference count for new card_id to 1
7. Update schema usage statistics for the collection
8. If original count reaches 0, move original to "Recently Deleted"
9. Update collection metadata and notify clients with schema information
10. Reference now points to independent instance that can be modified and re-schemed separately

Duplicate References in Same Collection:

  • Same card can be referenced multiple times within one collection
  • Each reference has unique reference_id and position
  • Useful for spatial mode (same card in multiple areas) or workflow views
  • Reference count incremented for each reference, even within same collection

Instance Detachment Patterns:

  • Users can detach shared instances to create independent copies
  • Critical for collaborative workflows where users need to fork shared content
  • Enables "edit locally without affecting others" use cases
  • Supports progressive ownership transfer (shared -> personal copies)
  • Preserves original shared instance for other users while creating independent fork
  • Enables independent schema evolution for detached instances
  • Allows customization of card structure without affecting original shared content

Collection-Based Operations

Bulk Operations with Reference Awareness

Multi-Selection: Select multiple card references across different schemas for batch operations.

Reference-Aware Bulk Editing:

  • Instance Editing: Apply changes to card instances (affects all references everywhere)
  • Reference Editing: Apply changes to reference metadata (affects only selected references)
  • Instance Detachment: Create independent copies before editing (affects only detached references)
  • Mixed Operations: Some fields update instances, others update reference-specific data
  • Selective Detachment: Detach only selected references before bulk editing operations

Batch Reference Management:

  • Move References: Transfer multiple references between collections atomically
  • Copy References: Create new references to same card instances in target collections
  • Remove References: Delete multiple references with proper reference count management
  • Bulk Reference Creation: Add multiple existing cards to a collection simultaneously
  • Bulk Instance Detachment: Detach multiple references from shared instances before modification
  • Selective Detachment: Choose which references to detach while leaving others connected to shared instances

Mass Operations with Reference Counting:

  • Mass Reference Removal: Remove multiple references, handle card lifecycle for zero-count cards
  • Collection Cleanup: Remove all references from a collection before deletion
  • Bulk Card Creation: Create multiple cards with their first references in a collection
  • Reference Validation: Batch verify and repair reference integrity

Atomic Bulk Operations: All reference count changes and collection updates happen atomically to prevent inconsistent states during bulk operations.

Collection-Wide Operations (Future Enhancement)

Schema Analysis: Analyze schema distribution and usage patterns within collections.

  • Schema Distribution Reports: Show which schemas are used and how frequently
  • Schema Compatibility Analysis: Identify potential conflicts in heterogeneous collections
  • Schema Evolution Tracking: Track how schemas change over time within collections
  • Schema Consolidation: Suggest opportunities to merge similar schemas

Content Migration: Move or transform content between collections with different organizational schemes.

  • Schema-Aware Migration: Ensure migrated cards are compatible with target collection constraints
  • Bulk Schema Changes: Apply schema transformations to multiple cards simultaneously
  • Migration Validation: Verify all cards remain valid after collection-to-collection moves

Duplicate Detection: Identify and manage duplicate cards within or across collections.

  • Schema-Based Deduplication: Find cards with same content but different schemas
  • Cross-Schema Matching: Identify equivalent content across different schema types

Archival Operations: Archive entire collections or portions based on age, usage, or other criteria.

  • Schema-Preserving Archives: Maintain schema definitions when archiving collections
  • Schema Usage Cleanup: Remove unused schemas when collections are archived

Search and Discovery

Collection-Scoped Search

Within-Collection Search: Search for cards within a specific collection.

Cross-Schema Search: Find cards across different schemas within heterogeneous collections.

Metadata Search: Search collection properties and organizational information.

Quick Filters: Rapid filtering by common attributes like schema type or modification date.

Advanced Discovery (Future Enhancement)

Cross-Collection Search: Search across multiple collections simultaneously.

Relationship Discovery: Find collections related through shared cards or similar content.

Pattern Recognition: Identify organizational patterns and suggest improvements.

Content Recommendations: Suggest relevant content based on current collection context.

Performance and Scalability

Collection Size Management

Large Collection Handling: Strategies for managing collections with thousands or millions of cards.

Pagination Support: Efficient loading of large collections in manageable chunks.

Index Optimization: Maintain indexes optimized for common collection operations.

Memory Management: Balance between performance and memory usage for large collections.

Concurrent Access

Multi-User Support: Handle multiple users accessing and modifying the same collection simultaneously.

Conflict Resolution: Manage conflicts when multiple users modify collection organization.

Real-Time Updates: Propagate collection changes to all connected clients efficiently.

Locking Strategies: Prevent data corruption during concurrent collection modifications.

Integration with Other Systems

Import and Export

Collection Import: Import collections from external systems with appropriate schema mapping.

Bulk Export: Export collection contents in various formats for external use.

Selective Export: Export filtered subsets of collections based on criteria.

Format Conversion: Convert between different organizational schemes and metadata formats.

API Integration

Collection APIs: Programmatic access to collection management operations.

Webhook Support: Notify external systems of collection changes and events.

Synchronization: Keep collections synchronized with external data sources.

Federation Support: Share and synchronize collections across different Commonplace instances.

This collection management architecture provides both the fundamental capabilities needed for basic Commonplace operations and clear extension points for sophisticated organizational features as the system evolves.