Several machines
The desk outgrows one machine. The web page runs on one server, and the
mailbox reader moves to another. Both need the ticket queue and the
settings. An SQLite file is opened by processes on the machine that has it:
SQLite's locking is not reliable over a network filesystem, so putting
helpdesk.db on a shared drive and opening it from both servers invites a
corrupted file. Until now, everything that touched the desk ran where the
file is.
A server for the file#
beaver serve runs an HTTP server on top of a database file. It needs the
remote extra. --api-key (or the BEAVER_API_KEY variable) makes every
request carry that key as a bearer token.
beaver serve --db helpdesk.db --host 0.0.0.0 --port 8000 --api-key "$BEAVER_API_KEY"
This chapter starts it the same way, as a background process on a free port:
import atexit import socket import subprocess import sys import time from pathlib import Path import httpx sock = socket.socket() sock.bind(("127.0.0.1", 0)) port = sock.getsockname()[1] sock.close() beaver_cli = Path(sys.executable).parent / "beaver" server = subprocess.Popen( [str(beaver_cli), "serve", "--db", "helpdesk.db", "--port", str(port), "--api-key", "desk-key"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, ) atexit.register(server.terminate) # stop it when this program ends URL = f"http://127.0.0.1:{port}" for _ in range(100): try: httpx.get(f"{URL}/docs") break except httpx.ConnectError: time.sleep(0.1) print("without the key:", httpx.get(f"{URL}/docs").status_code) print("with it:", httpx.get(f"{URL}/docs", headers={"Authorization": "Bearer desk-key"}).status_code)
without the key: 401 with it: 200
The server is a FastAPI app. /docs is its interactive API page, which lists
every route, and like every route it needs the key.
The client#
beaver.connect(url, api_key=...) returns a BeaverClient, which has the
same methods as BeaverDB for the collections the server exposes:
dictionaries, lists, queues and logs. The mailbox reader on the other machine
changes one line.
from beaver import connect desk = connect(URL, api_key="desk-key") print(type(desk).__name__) desk.dict("settings")["shop"] = "Spoke & Chain" desk.queue("tickets").put({"id": "T-1701", "subject": "Brakes"}, priority=1) desk.log("audit").log({"agent": "mailbox", "action": "import", "ticket": "T-1701"}) print(desk.dict("settings")["shop"], len(desk.dict("settings")), "shop" in desk.dict("settings"))
BeaverClient Spoke & Chain 1 True
Behind it, BeaverClient runs an AsyncBeaverClient on a background thread,
as BeaverDB runs AsyncBeaverDB. connect(url, sync=False) returns the
AsyncBeaverClient itself for async code. Its collections are RemoteDict,
RemoteList, RemoteQueue and RemoteLog, which send each call as an HTTP
request.
What the server wrote is in the file, where local code reads it as before:
from beaver import BeaverDB with BeaverDB("helpdesk.db") as local: print(local.queue("tickets").peek().data, local.log("audit").count())
{'id': 'T-1701', 'subject': 'Brakes'} 1
What does not cross the wire#
Some methods cannot work over plain requests: iterating over a dictionary's
keys (a stream), live() on a log (an endless one), batched(), dump() and
load(). The remote collections raise LocalOnlyError for them, with the
reason:
from beaver import LocalOnlyError for attempt in (lambda: list(desk.dict("settings").keys()), lambda: next(iter(desk.log("audit").live()))): try: attempt() except LocalOnlyError as e: print("LocalOnlyError:", e)
LocalOnlyError: dict.keys() is only available on local databases (no streaming on the wire yet) LocalOnlyError: log.live() is only available on local databases (infinite stream, no SSE yet)
Documents, vectors, graphs, blobs, locks, channels, sketches and events are
not served in 2.4: the client has no docs(), lock() and so on. A program
that needs them runs where the file is, and the other machines reach it
through the queue and the dictionaries, or through an API of your own.
Some results also come back in another shape than locally. In 2.4, a remote
queue's get() returns a plain dictionary rather than a QueueItem, and a
remote log's range() a list of dictionaries rather than LogEntry tuples
(issue #50):
print(desk.queue("tickets").get(timeout=1)) print(desk.log("audit").range())
{'priority': 1.0, 'timestamp': 1794360872.8177652, 'data': {'id': 'T-1701', 'subject': 'Brakes'}}
[{'timestamp': 1794360872.821416, 'data': {'agent': 'mailbox', 'action': 'import', 'ticket': 'T-1701'}}]
Errors over the wire#
An exception on the server is sent back as an ErrorEnvelope, a small JSON
object with the exception's class name and message, and raised again in the
client. KeyError, ValueError, TimeoutError, pydantic's
ValidationError, TransactionTimeoutError and the schema errors of chapter
16 come back as themselves, so the same except clauses work locally and
remotely. Any other exception arrives as BeaverRemoteError, and so does a
missing or wrong API key, which the server's BearerAuthMiddleware rejects
before any collection is touched. In 2.4 a remote KeyError carries its
message quoted twice (issue #50).
from beaver import BeaverRemoteError try: desk.dict("settings")["holiday"] except KeyError as e: print("KeyError:", e) try: connect(URL, api_key="a guess").dict("settings")["shop"] except BeaverRemoteError as e: print("BeaverRemoteError:", e) desk.close() server.terminate() server.wait()
KeyError: '"Key \'holiday\' not found in dictionary \'settings\'"' BeaverRemoteError: invalid or missing bearer token
A sealed database#
Once the desk's data is baked into an image, the file arrives sealed: mode
0444, in a directory the runtime mounts 0555. A normal open fails there twice
over. beaver asks for journal_mode = WAL, creates its tables and stamps
user_version, and SQLite takes the file O_RDWR, which on an overlay
filesystem copies the whole thing into the writable layer on the first read.
read_only=True opens it the other way:
import os from beaver import BeaverReadOnlyError with BeaverDB("sealed-desk.db") as db: db.dict("settings")["shop"] = "Spoke & Chain" db.docs("tickets").index(id="T-1", body={"title": "Broken bell"}) db.list("thread").push("My bell is broken") os.chmod("sealed-desk.db", 0o444) try: with BeaverDB("sealed-desk.db", read_only=True) as sealed: print(sealed.dict("settings")["shop"]) print(sealed.docs("tickets").search("bell")[0].document.id) print(list(sealed.list("thread"))) try: sealed.dict("settings")["shop"] = "somewhere else" except BeaverReadOnlyError as e: print("BeaverReadOnlyError:", str(e).splitlines()[0][:64]) finally: os.chmod("sealed-desk.db", 0o644)
Spoke & Chain T-1 ['My bell is broken'] BeaverReadOnlyError: Database 'sealed-desk.db' was opened read-only (read_only=True),
It opens the path as file:<path>?immutable=1, so SQLite never takes a write
lock and creates no -wal, -shm or -journal beside it. beaver skips the
journal, synchronous, table-creation and user_version steps on that open,
keeps the version check, and turns every write into BeaverReadOnlyError
instead of SQLite's attempt to write a readonly database. A read that meets
an expired key raises KeyError and leaves the row alone, since a sealed file
cannot delete it. AsyncBeaverDB(path, read_only=True) is the same open for
async code, and read_only=True on a missing path raises FileNotFoundError
rather than creating a file.
The other edge is older files: a 1.x file is refused as always, and a pre-rc4
file cannot be upgraded without writing, so a read-only open says so instead of
trying. Reopening a current file read-write writes user_version only when it
differs, so an ordinary restart no longer touches the header at all.
The program#
The mailbox reader, now on its own machine. It reads the server's address and key from the environment, and falls back to the local file when there is no server, so the same program runs in both places.
import os from beaver import connect def open_desk(): url = os.environ.get("BEAVER_URL", "helpdesk.db") return connect(url, api_key=os.environ.get("BEAVER_API_KEY")) def import_email(desk, ticket: str, subject: str, urgent: bool) -> None: desk.queue("tickets").put( {"id": ticket, "subject": subject}, priority=1 if urgent else 5 ) desk.log("audit").log({"agent": "mailbox", "action": "import", "ticket": ticket}) def main(): desk = open_desk() try: desk.queue("tickets").clear() import_email(desk, "T-1801", "Brakes failed", urgent=True) import_email(desk, "T-1802", "Invoice copy", urgent=False) print(type(desk).__name__, "queue:", len(desk.queue("tickets"))) finally: desk.close() if __name__ == "__main__": main()
BeaverDB queue: 2
