Storage

How does Fjall LSM store grains?

Fjall is an embedded LSM-tree storage engine written in Rust that provides the durable persistence layer for all AI memory grains in the Areev context database.

LSM-trees (Log-Structured Merge-trees) buffer writes in an in-memory memtable, then flush sorted runs to disk as immutable SST (Sorted String Table) files. This architecture delivers high write throughput because writes never require random disk I/O — they append sequentially. Fjall compacts SST files in the background, merging sorted runs to reclaim space and maintain read performance. The autonomous memory engine writes encrypted .mg blobs to Fjall, keyed by the grain’s SHA-256 content hash for content-addressed storage.

Fjall organizes data into partitions. The primary partition stores grain blobs. Separate partitions hold the wrapped DEKs (data encryption keys), the audit trail, and the entity_latest index (a mapping from entity key to the most recent grain version). Each partition has independent compaction settings tuned to its access pattern — the grain partition uses leveled compaction for read-heavy workloads, while the superseded partition uses size-tiered compaction for write-heavy access patterns. AI agent memory operations (write, recall, forget) are atomic within a single partition and use Fjall’s batch write API for cross-partition consistency.

# Inspect storage directory structure
ls -la ./data/
# grains/       — Fjall LSM partition for encrypted grain blobs
# keys/         — Fjall partition for wrapped DEKs
# audit/        — Fjall partition for audit trail events
# fts/          — Tantivy full-text index segments
# vectors/      — HNSW graph files
# hexastore/    — Entity-relation triple index

How does full-text search work?

Tantivy, a Rust-native full-text search engine, indexes grain content using BM25 ranking and supports phrase queries, boolean operators, and field-scoped searches.

When the autonomous memory engine writes a grain, it extracts the text content and inserts it into the Tantivy index along with metadata fields (subject, namespace, grain type). Tantivy tokenizes the text, builds an inverted index, and stores term frequencies and positions for BM25 scoring. The context database queries Tantivy during recall operations that include a text query component.

Tantivy manages its own segment files independently of Fjall. Segment merges happen in the background and are triggered by the number of uncommitted segments. The fts feature flag controls whether Tantivy is compiled into the binary — without it, text queries return an error. AI memory grains indexed by Tantivy remain searchable until they are explicitly forgotten (crypto-erasure deletes both the Fjall blob and the Tantivy index entry atomically).

# Search via HTTP API
curl -s -X POST http://localhost:4009/api/memories/default/recall \
  -H "Content-Type: application/json" \
  -d '{"query": "quarterly revenue", "limit": 10}'
# Python full-text search
import requests
results = requests.post("http://localhost:4009/api/memories/default/recall", json={
    "query": "quarterly revenue",
    "limit": 10
}).json()
for grain in results["items"]:
    print(grain["subject"], grain["score"])

How does vector similarity search work?

The HNSW (Hierarchical Navigable Small World) graph indexes grain embedding vectors for approximate nearest-neighbor search, powered by USearch or FAISS depending on the build configuration.

When a grain includes an embedding vector (either provided by the caller or computed by an LLM enrichment hook), the AI agent memory engine inserts the vector into the HNSW graph. The graph structure allows approximate nearest-neighbor queries in logarithmic time relative to the dataset size. The context database uses cosine similarity as the distance metric for vector search. Below 10,000 vectors, the engine uses brute-force cosine search; above that threshold, it automatically switches to the HNSW graph index.

The vector feature flag controls HNSW compilation, with vector-usearch and vector-faiss selecting the backend. USearch is the default backend — it provides a lock-free C++ HNSW implementation suitable for most CPU-only deployments. FAISS can be selected as an alternative for workloads that benefit from its GPU acceleration or quantization features. Both backends use cosine similarity as the distance metric. Vector search results include a similarity score that the recall engine fuses with BM25 scores (via RRF) when both text and vector queries are present in a hybrid recall request.

# Vector similarity search via HTTP
curl -s -X POST http://localhost:4009/api/memories/default/recall \
  -H "Content-Type: application/json" \
  -d '{"vector": [0.1, 0.2, 0.3, ...], "top_k": 5}'

How does the hexastore index entity relations?

The hexastore stores entity-relation triples (subject, predicate, object) in six permuted indexes, enabling fast lookups from any starting point in the triple.

Entity-type AI memory grains automatically generate triples that the context database inserts into the hexastore. For example, a grain with subject user-123 and a works_at relation to acme-corp produces six index entries: SPO, SOP, PSO, POS, OSP, OPS. This allows queries like “find all entities that work at Acme Corp” (starting from object), “find all relations for user-123” (starting from subject), or “find all works_at relations” (starting from predicate) — all with O(1) lookup time.

The entity_latest partition complements the hexastore by maintaining a pointer from each entity key to its most recent grain version. This supports use cases where agents need the current state of an entity rather than its full history. The autonomous memory engine updates entity_latest atomically with each grain write, so it always reflects the most recent version.