Beaver2.5.0 syalia-srl/beaver

Inside beaver

This chapter is for people who change beaver rather than use it: how the pieces of the earlier chapters are built, and where to look in the source. Every block runs against the package like the rest of the guide, and some of them reach into private attributes to show what is there. Code that uses beaver should not.

One file, a fixed set of tables#

Every collection of every kind lives in a small, fixed set of tables, created with CREATE TABLE IF NOT EXISTS each time a database is opened. A new kind of data adds a table, and an old file simply gains it on its next open: there is no migration step for additive changes. Every table name is dunder-quoted, which is how 2.x tells its files from 1.x ones (chapter 16).

import sqlite3

from beaver import BeaverDB

with BeaverDB("inside.db") as db:
    db.dict("settings")["shop"] = "Spoke & Chain"

conn = sqlite3.connect("inside.db")
tables = [r[0] for r in conn.execute("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE '__beaver%' ORDER BY name")]
print(len(tables), "tables:", ", ".join(tables))
print("user_version:", conn.execute("PRAGMA user_version").fetchone()[0])
print(conn.execute("SELECT dict_name, key, value FROM __beaver_dicts__").fetchall())
conn.close()
23 tables: __beaver_blobs__, __beaver_dicts__, __beaver_documents__, __beaver_edges__, __beaver_field_index__, __beaver_field_index_manifest__, __beaver_fts_index__, __beaver_fts_index___config, __beaver_fts_index___content, __beaver_fts_index___data, __beaver_fts_index___docsize, __beaver_fts_index___idx, __beaver_lists__, __beaver_lock_waiters__, __beaver_logs__, __beaver_lsh_config__, __beaver_lsh_index__, __beaver_manager_versions__, __beaver_priority_queues__, __beaver_pubsub_log__, __beaver_sketches__, __beaver_trigrams__, __beaver_vectors__
user_version: 1
[('settings', 'shop', '"Spoke & Chain"')]

user_version is the schema version, BEAVER_DB_VERSION in core.py. A file whose version is higher than the running beaver's fails to open with BeaverIncompatibleSchemaError, so raising it is a deliberate break of file compatibility. Prefer a change that fits as a new table.

An async engine and a sync portal#

AsyncBeaverDB owns one aiosqlite connection in WAL mode, and every collection of that database shares it. BeaverDB starts a thread running an event loop, builds the AsyncBeaverDB on it, and hands each collection back wrapped in a BeaverBridge. The bridge forwards every attribute access to the async object: a coroutine is run on the loop and its result returned, an async generator becomes a sync iterator, and an async context manager is wrapped so that with works. len(), [], in and iteration map to count, get and set, contains and __aiter__.

import threading

db = BeaverDB("inside.db")
settings = db.dict("settings")
print(type(settings).__name__, "->", type(settings._async_obj).__name__)
print([t.name for t in threading.enumerate() if t.name.startswith("BeaverDB")])
BeaverBridge -> AsyncBeaverDict
['BeaverDB-Reactor']

So the logic is written once, in the async classes, and the sync interface follows. A new method on a manager needs nothing in the bridge.

Every factory method returns the same object for the same name, from a cache on the engine (singleton()). That is why the model and the secret of the first call win (chapter 2), and why locks and listeners are shared within a database.

print(db.dict("settings")._async_obj is db.dict("settings")._async_obj)
db.close()
True

Managers#

Each collection class derives from AsyncBeaverBase. The base keeps the name, the optional model, and serialization: a pydantic model is dumped with model_dump_json, anything else with json.dumps, and reads validate with the model when there is one. It also gives every collection two locks of the kind chapter 5 described: a public one, which with collection: takes, and an internal one with a five-second TTL, held for milliseconds around each write.

Three decorators in manager.py carry the concurrency rules. @atomic takes the internal lock and a transaction, for writes. @read_atomic takes only a transaction, so a read never sees another task's uncommitted write on the shared connection, and never writes a lock row. @emits publishes an event after a write, but only once something has attached to the collection's bus (chapter 11), so collections nobody listens to pay nothing.

AsyncBeaverDict adds encryption on top of the base serialization: with a secret, values pass through a Cipher, which derives a Fernet key from the secret with PBKDF2 and a per-dictionary salt, and whose encrypt and decrypt work on bytes. The salt and a known encrypted value, to verify the secret, are kept in the internal __security__ dictionary, which is_system marks as never encrypted.

from beaver.security import Cipher

cipher = Cipher("shop master key")
token = cipher.encrypt(b"ck_live_51H8")
print(len(cipher.salt), "byte salt;", Cipher("shop master key", salt=cipher.salt).decrypt(token))
16 byte salt; b'ck_live_51H8'

Transactions on one connection#

All writers of a process share one connection, so a transaction is an asyncio.Lock held by one task: Transaction takes it, issues BEGIN IMMEDIATE, and commits at the end of the outermost block of that task. Inner blocks of the same task pass through, which is what lets @atomic methods run inside a caller's transaction. SQLite's busy timeout only arbitrates between connections, so the wait for that lock has its own bound, connection_timeout, and TransactionTimeoutError when it runs out. If the task that held it finished without releasing it, the next writer sees the owner is done, rolls its transaction back, logs a warning on the beaver logger and proceeds.

One consequence for operations: in WAL mode, readers keep working while writers queue. A process stuck holding the write lock still answers reads, so a health check of a beaver-backed service has to write to mean anything.

Lists without renumbering#

A list's order is a text key per item, not an index. Inserting between two items asks key_between(a, b) for a key that sorts between theirs, so an insert writes one row and moves none.

from beaver._fracdex import key_between

first = key_between(None, None)
last = key_between(first, None)
middle = key_between(first, last)
print(first, middle, last, sorted([last, first, middle]) == [first, middle, last])
V c k True

Channels and events#

All channels of a database share one PubSubEngine: a single task that polls the __beaver_pubsub_log__ table for rows newer than the last it saw, and pushes each one onto the asyncio.Queue of every listener of that channel. listen() registers a queue, and leaving the iterator unregisters it (unsubscribe). An events bus is a channel named after it, plus a dispatch task that reads the channel and calls the attached callbacks. close() stops both kinds of task.

docs.index() stores the document as JSON and flattens every string field, nested ones included, into the __beaver_fts_index__ FTS5 table, one row per field path, which is what lets search(on=[...]) restrict to fields. With fuzzy=True it also writes the trigrams of each field to __beaver_trigrams__. A fuzzy search counts the query's trigrams that each document shares, and returns the highest counts. where= filters compile to json_extract predicates over the document JSON.

Field indexes (chapter 13) live in indexing.py. extract_path walks a dotted path through models and dictionaries, normalize turns a scalar into its text form and, for a number, a numeric form, so that range filters compare numbers as numbers and the string "10" stays a string, and build_rows produces an item's index rows. At query time, plan_filters splits the filters into those an index can answer, through compile_indexed_filter, and those it cannot, through compile_scan_filters. A field's index is trusted only once it is marked complete, so a half-built index makes queries slower, never wrong.

from beaver import q
from beaver.indexing import extract_path, normalize

print(extract_path({"agent": {"name": "ana"}}, "agent.name"), normalize("10"), normalize(10))
print(q().agent.name == "ana")
(True, 'ana') ('10', None) ('10', 10.0)
Filter(path='agent.name', operator='==', value='ana')

The second line is what a q expression builds: a Query collects the dotted path through attribute access, and a comparison returns a Filter with the path, the operator and the value.

Vectors#

Vectors are stored as raw float32 bytes and searched with numpy over the whole matrix. Past 10,000 vectors, each vector is also hashed with 16 random hyperplanes into a bucket of __beaver_lsh_index__, and a search scores only the query's bucket and its 16 one-bit neighbours (chapter 7). Everything is pure numpy: a compiled ANN library would break the rule that beaver installs anywhere Python does.

Sketches#

An ApproximateSet packs a HyperLogLog and a Bloom filter into one byte array, sized from the capacity and error rate, and to_bytes() is what is stored. Each add reloads the bytes, sets the bits and saves them inside a transaction. contains reads the copy in memory (chapter 12).

from beaver.sketches import ApproximateSet

s = ApproximateSet(capacity=1000, error_rate=0.01)
for n in range(100):
    s.add(f"id-{n}".encode())
print(b"id-7" in s, b"id-x" in s, len(s), len(s.to_bytes()), "bytes")
True False 100 17583 bytes

One decorator, three ways in#

A manager method marked @expose(path=..., method=..., cli_name=..., cli_help=...) is reachable from the command line, the HTTP server and the remote client at once. The decorator only attaches an EndpointMeta to the function. build_typer_for walks a manager class for those and registers a CLI command per method, create_app mounts an HTTP route per method on a FastAPI app, and the client builds a dispatcher per method from the same metadata. @local_only("reason") marks what cannot cross the wire, and the remote classes raise LocalOnlyError with that reason. A method with neither decorator is local by accident rather than by decision.

from beaver.dicts import AsyncBeaverDict

print(AsyncBeaverDict.set.__beaver_endpoint__)
print(AsyncBeaverDict.keys.__beaver_local_only__)
EndpointMeta(path='/{key}', method='PUT', cli_name='set', cli_help='Set a value for a key.', body_param='value')
dict.keys() is only available on local databases (no streaming on the wire yet)

The CLI uses those commands directly, on a file or on a server (--url):

import subprocess
import sys
from pathlib import Path

beaver_cli = str(Path(sys.executable).parent / "beaver")
print(subprocess.run([beaver_cli, "--db", "inside.db", "--raw", "dict", "settings", "get", "shop"], capture_output=True, text=True).stdout)
"Spoke & Chain"

On the server side, an exception becomes an ErrorEnvelope with envelope_from_exception, and http_code_for picks its status code (404 for KeyError, 409 for a timeout, 422 for validation, 400 for ValueError, 500 otherwise). The client turns the envelope back into the exception with raise_from_envelope, or into BeaverRemoteError when the class is not one it knows. BearerAuthMiddleware checks the API key before any route runs.

from beaver.errors import envelope_from_exception, http_code_for

error = KeyError("T-404")
print(envelope_from_exception(error), http_code_for(error))
error='KeyError' message="'T-404'" detail=None 404

The cache that is not wired in#

cache.py holds a per-manager read cache: the ICache protocol, a LocalCache that invalidates itself when a version counter in the file changes (touch bumps it), CacheStats with hit_rate, invalidation_rate and operations from its stats, and the @cached and @invalidates_cache decorators. None of it is in use in 2.4. AsyncBeaverDB.cache() always returns the DummyCache, whose get misses and whose invalidate does nothing, and no manager method is decorated. The cache_timeout argument of AsyncBeaverDB is accepted and ignored.

from beaver import AsyncBeaverDB

cache = AsyncBeaverDB("inside.db").cache("dict:settings")
print(type(cache).__name__, cache.get("shop"), cache.stats())
DummyCache None CacheStats(hits=0, misses=0, invalidations=0, sets=0, pops=0)

Migration internals#

beaver.migrate opens the 1.x source read-only, so that not even a WAL checkpoint touches it, and refuses a destination that exists. Tables whose schema did not change are copied as they are. Lists get new fractional keys in their old order, and the full-text and trigram indexes are rebuilt from the documents. total_rows on a MigrationReport adds up the rows it reports.

Working on beaver#

make sync           # uv sync with the security and remote extras
uv run black .      # format; CI runs black --check
make test-unit      # the default target, about half a minute
make test-all       # every suite, including concurrency
make docs-check     # this guide: render, tangle, run the examples, rift

pytest.ini sets asyncio_mode = auto and a 30-second timeout per test, so a test that deadlocks on a lock fails instead of hanging the run. Dependencies stay few on purpose: aiosqlite, numpy, pydantic, rich and typer in the core, the HTTP stack and cryptography behind extras.

When you add a public class or method, name it in the chapter where a reader first needs it. rift check fails the docs gate on a public class that no chapter names.

The program#

A small inspector for any beaver file: its tables with their row counts, and its schema version.

examples/inspect_file.py · L1–33
import sqlite3
import sys

def inspect(path: str) -> list[str]:
    conn = sqlite3.connect(f"file:{path}?mode=ro", uri=True)
    try:
        version = conn.execute("PRAGMA user_version").fetchone()[0]
        lines = [f"{path}: schema version {version}"]
        names = conn.execute(
            "SELECT name FROM sqlite_master WHERE type='table'"
            " AND name LIKE '__beaver%' ORDER BY name"
        ).fetchall()
        for (name,) in names:
            rows = conn.execute(f'SELECT COUNT(*) FROM "{name}"').fetchone()[0]
            if rows:
                lines.append(f"  {name:<32} {rows} rows")
        return lines
    finally:
        conn.close()

def main():
    if len(sys.argv) > 1:
        path = sys.argv[1]
    else:
        from beaver import BeaverDB

        path = "inspect-demo.db"
        with BeaverDB(path) as db:
            db.dict("settings")["shop"] = "Spoke & Chain"
    print("\n".join(inspect(path)))

if __name__ == "__main__":
    main()
inspect-demo.db: schema version 1
  __beaver_dicts__                 1 rows
  __beaver_fts_index___config      1 rows
  __beaver_fts_index___data        2 rows
Last updated 2026-11-10 · Edit this pageBeaver is MIT-licensed. Made by Syalia.