Similar tickets
When a new ticket comes in, the agent wants the old tickets that were about the same problem, and how they were solved. Full-text search needs shared words. "Brakes squeal when stopping" and "noise from the back wheel when I slow down" share none, and they are the same problem.
The usual answer is embeddings: a model turns each text into a vector so that texts with similar meaning get nearby vectors. Real ones come from a model such as sentence-transformers or an embeddings API. To keep this chapter self-contained it uses a stand-in: the first five letters of each word that is not a stop word, hashed into 256 numbers. It only knows shared word stems, but the vectors behave like real ones: you store them, and you look for the nearest.
import hashlib import re import numpy as np STOP = {"my", "the", "a", "in", "for", "when", "i", "lot", "from", "last"} def embed(text, dim=256): v = np.zeros(dim, dtype=np.float32) for word in re.findall(r"\w+", text.lower()): if word not in STOP: # The first five letters stand in for meaning: brake, brakes, braking. v[int(hashlib.md5(word[:5].encode()).hexdigest(), 16) % dim] += 1 norm = np.linalg.norm(v) return (v / norm if norm else v).tolist() old_tickets = { "T-9001": "Rear brake squeaks when stopping", "T-9002": "Chain skips in seventh gear", "T-9003": "Front brake squeaks loudly in the rain", "T-9004": "Invoice copy for my last order", "T-9005": "Gear cable snapped", }
With what we have, the vectors go in a dictionary and the search compares the query with every one of them:
from beaver import BeaverDB db = BeaverDB("helpdesk.db") by_id = db.dict("embeddings") for ticket_id, text in old_tickets.items(): by_id[ticket_id] = embed(text) def nearest(text, k=2): query = np.array(embed(text)) scored = [(1 - float(np.dot(query, v)), tid) for tid, v in by_id.items()] return sorted(scored)[:k] print(nearest("my brakes squeak"))
[(0.2928932309150696, 'T-9001'), (0.36754448636323644, 'T-9003')]
It works, and every search reads every vector out of the file as JSON and compares them one at a time in Python. At a few thousand tickets that is already the slow part of the request.
A vector collection#
db.vectors(name) stores vectors as packed float32 arrays, each with an id
and optional metadata. near(vector, k=...) returns the k closest ones,
computed with numpy over the whole collection at once.
vectors = db.vectors("tickets") for ticket_id, text in old_tickets.items(): customer = "maria" if ticket_id in ("T-9001", "T-9004") else "omar" vectors.set(ticket_id, embed(text), {"customer": customer, "text": text}) for item in vectors.near(embed("my brakes squeak"), k=2): print(item.id, round(item.score, 3), item.metadata["text"])
T-9001 0.146 Rear brake squeaks when stopping T-9003 0.184 Front brake squeaks loudly in the rain
Each result is a VectorItem with the id, the vector, the metadata and
a score. The default score is the cosine similarity mapped to a distance
between 0 and 1, (1 - cos) / 2: 0 is the same direction, 0.5 unrelated, and
lower means closer. It is half the 1 - cos the loop above computed. get(id),
delete(id), len() and iteration work as in the other collections.
Narrowing the search#
filters= takes the same q conditions as chapter 6, applied to the
metadata. candidate_ids= restricts the search to a list of ids, for example
the result of a full-text search, which is how you combine the two kinds of
search.
from beaver import q print([i.id for i in vectors.near(embed("squeaky brakes"), k=3, filters=[q().customer == "omar"])]) print([i.id for i in vectors.near(embed("squeaky brakes"), k=3, candidate_ids=["T-9002", "T-9005"])])
['T-9003', 'T-9002', 'T-9005'] ['T-9002', 'T-9005']
metric= replaces the cosine distance with any function that takes the
matrix of stored vectors and the query and returns one score per row, lower
being better. beaver.vectors has three: cosine, the default, euclid for
Euclidean distance and manhattan for the sum of absolute differences. A
function of your own has the same shape:
from beaver.vectors import euclid, manhattan def chebyshev(matrix, query): return np.max(np.abs(matrix - query), axis=1) for metric in (euclid, manhattan, chebyshev): hits = vectors.near(embed("squeaky brakes"), k=2, metric=metric) print(metric.__name__, [(i.id, round(i.score, 3)) for i in hits])
euclid [('T-9001', 0.765), ('T-9003', 0.857)]
manhattan [('T-9001', 1.414), ('T-9003', 1.861)]
chebyshev [('T-9003', 0.447), ('T-9001', 0.5)]
Many vectors#
Comparing with every vector is exact, and fine up to some thousands. Past
10,000 vectors in a collection, near() switches to an index: each vector is
hashed into a 16-bit bucket by random hyperplanes (SimHash), and the search
only scores the vectors in the query's bucket and the 16 buckets one bit away.
It is faster and approximate: a close vector that hashed far away is missed.
method="exact" or method="lsh" picks one regardless of size.
print([i.id for i in vectors.near(embed("squeaky brakes"), k=2, method="lsh")]) db.close()
['T-9001', 'T-9003']
The index is pure numpy, like the rest of beaver, so there is nothing compiled to install.
The program#
Suggesting similar solved tickets for a new one. Swap embed for a real
model and nothing else changes.
import hashlib import re import numpy as np from beaver import BeaverDB, q STOP = {"my", "the", "a", "in", "for", "when", "i", "lot", "from", "last"} def embed(text: str, dim: int = 256) -> list[float]: """A stand-in for a real embedding model: hashed bag of word stems.""" v = np.zeros(dim, dtype=np.float32) for word in re.findall(r"\w+", text.lower()): if word not in STOP: v[int(hashlib.md5(word[:5].encode()).hexdigest(), 16) % dim] += 1 norm = np.linalg.norm(v) return (v / norm if norm else v).tolist() def remember(db: BeaverDB, ticket_id: str, text: str, solved: bool) -> None: db.vectors("tickets").set(ticket_id, embed(text), {"text": text, "solved": solved}) def similar_solved(db: BeaverDB, text: str, k: int = 3) -> list[str]: hits = db.vectors("tickets").near(embed(text), k=k, filters=[q().solved == True]) return [h.metadata["text"] for h in hits if h.score < 0.4] def main(): with BeaverDB("helpdesk.db") as db: db.vectors("tickets").clear() remember(db, "T-9101", "Rear brake squeaks when stopping", solved=True) remember(db, "T-9102", "Chain skips in seventh gear", solved=True) remember(db, "T-9103", "Front brake squeaks in the rain", solved=False) for text in similar_solved(db, "brakes squeak a lot"): print("see:", text) if __name__ == "__main__": main()
see: Rear brake squeaks when stopping
