ego-OS

Every AI agent I use starts cold. It doesn't know which project I'm deep in this week, who the people on my calendar are, what I decided last month, or which of my efforts are still alive. I built ego-OS to fix that for my own agents. It reads everything I write into a Markdown vault, builds a graph of memories, entities and concepts in Postgres, and serves that graph to Claude Code, claude.ai or any other MCP client through 32 tools. I have run it daily since June 2026, and v0.1.0 is open source under Apache-2.0.

The problem

The usual fix for agent context is to index your notes and let the agent search them. That handles "find the note where I wrote X." It does nothing for "bring me up to speed on X." Flat ranking scores each note on its own against the wording of the question. A planning doc a coding agent wrote and a meeting summary about the same project use different vocabulary, so they sit far apart in embedding space, and a search that finds one often misses the other. What I want to know is what belongs to the project, and similarity to my phrasing doesn't answer that.

There is also an ownership problem. Agent memory today belongs to the agent vendor, so what one assistant learns about me doesn't carry over to another. I wanted the memory on my own machine, in files I control, behind a protocol any agent can speak.

Companies have the same problem at a larger scale: giving agents what the CEO knows. ego-OS is my attempt at that for one person.

The design

Three layers, each replaceable

ego-OS has three layers, and the boundaries between them are the main design decision.

The first layer is writers. Each is a small deterministic program that mirrors one source into a folder of the vault: meeting summaries, email threads I replied to, todos, and a hook that copies every Markdown file my coding agents write in any git repo. Notes I write by hand in Obsidian land in the same place.

The second layer is the vault itself, plain Markdown in git. It is the source of truth. Everything in the graph is derived from it, nothing in the graph is hand-edited, and the engine can be rebuilt from the files alone.

The third layer is the engine, a monorepo of six packages over one Postgres database. A core package owns the schema, ingest, enrichment and every write. Retrieval strategies sit behind one contract, the MCP server is a thin transport, local models run through Ollama, and a React dashboard browses the graph and traces every retrieval. One kb-run command supervises it all. A second, internet-facing MCP listener sits behind Cloudflare Access so claude.ai can reach it, and it refuses to start without auth.

The engine's only input contract is "a Markdown file in the vault." Adding a source means writing a new mirror, not changing the engine. When I moved email ingest from Microsoft Graph to reading the local Apple Mail store, only the email writer changed. Ingest and enrichment didn't, because the writer produces one note per replied thread and the ordinary vault adapter picks it up.

Identity is deterministic. Each note's id is a uuid5 of its source and external id, stamped into its frontmatter, so moving a note never changes its id or triggers costly re-enrichment.

Memories, entities and concepts

The graph itself has three layers.

Memories are one per note or email thread, each with a summary, embedding, importance score, event time and lifecycle. Each has one canonical home in a tree (a materialized Postgres ltree), and typed edges carry relationships across it.

Entities are the people, organizations, tools, projects and events those notes mention, with an open vocabulary of classes. An LLM pass extracts candidates, a gate drops noise such as code symbols, and an entity is promoted only when it recurs in at least two distinct memories. Near-duplicates like "gh" and "GitHub" become merge suggestions that a human approves, and the judge treats co-mention in one note as evidence of two different things. When unsure I reject, because a wrong reject is recoverable and a wrong merge is not. A reject is stored as a durable block so the pair isn't suggested again. In one 27-pair batch that meant 9 merges and 14 rejects; the other four dropped out because an earlier merge or cleanup had already removed one side.

Concepts are the projects, topics and areas I'm working on. They come from Leiden community detection over the bipartite memory-entity graph plus nearest-neighbor edges between memories, labeled by Claude Sonnet. Each concept tracks whether it is current and whether it is rising or fading, and a consolidation daemon recomputes them. This is the layer that puts the planning doc and the meeting summary in the same group when their embeddings disagree. My vault currently has 76 concepts, 28 of them active in the current window.

Enrichment and inference routing

Enrichment runs as eleven tasks, from entity extraction to embedding. Each one is routed in a single file, KB_Core/inference.toml, to one of three providers: local Ollama, the Claude Code CLI under the user's own login, or the metered Anthropic API. An excerpt from the shipped file, trimmed:

[provider.claude-cli]
model = "claude-haiku-4-5-20251001"  # the bulk per-memory passes run on Haiku: they are
                              # structured extraction, and Haiku keeps them off Sonnet quota
                              # (a Sonnet session limit pauses the whole enrichment lane).
                              # concept_label pins Sonnet explicitly below.
# ...
 
[task.doc_kind]
provider = "local"            # cheap one-label pass. Local KB_Inference is single-flight
                              # (30-77s/call on Gemma), fine at steady state; to drain a
                              # large backlog, route it to claude-cli temporarily.
 
[task.concept_label]
provider = "claude-cli"
model = "claude-sonnet-5"     # per-task override beats the [provider.*] default
 
[task.embed]
provider = "local"            # MUST stay local (enforced): changing the embedding
                              # model invalidates every stored vector ...

A few rules keep this file trustworthy. A missing or unknown task block stops the daemons at startup, so routing can't drift silently. The embedder is enforced local, because every stored vector assumes its width. And every model call site in the code is listed in an LLM call map doc that has to change in the same commit as the code.

An outage taught me the next rule. A Claude session limit once pushed 141 pending notes into a terminal "failed" state in three minutes. Now a provider-unavailable error pauses that provider's lane until its reset, leaves the notes pending, and enrichment resumes by itself.

The shipped default sends 7 of the 11 tasks to Claude, so note text leaves the machine under the user's own login. Going all-local means switching seven routing blocks to local, and enrichment gets much slower. Even the default setup runs three tasks on the local chat model, which wants 32 GB or more of unified memory. The README says so up front.

Search versus crawl

Two retrieval tools answer two different questions.

search_memories answers questions about a note. It is hybrid: Postgres full-text search fused with pgvector nearest-neighbor search over whole notes and chunks, then weighted by importance. It is nearly free, and it is what an agent should try first.

search_graph answers questions about a topic. It routes the query to concept clusters and entity anchors, crawls their membership, unions that set with the flat ranker's results, and fuses the two. It is deterministic, takes about 0.2 to 0.4 seconds, and makes no LLM calls. Every row is tagged as coming from the flat ranker, a concept, or a bridge, so you can see which notes only the crawl found. The union is deliberate, so structure adds candidates instead of replacing flat results. On the flat set the crawl pushed none of hybrid's gold out of the top 20; on the package set it pushed out two.

find_entity resolves names and misspellings, and expand returns a one-hop neighborhood filtered by entity kind, which answers "who is involved in X?"

A confidence contract

Ranking is relative. A search always returns the nearest things in the corpus, even when the corpus knows nothing about the question, so an agent that summarizes whatever comes back will confidently summarize the wrong note.

So every retrieval also returns an absolute confidence level, computed apart from ranking from the top-1 cosine similarity. strong means a semantic match. weak means hedge. none means the KB doesn't know, and the agent should say so instead of summarizing the rows. (Two more levels cover literal keyword hits and results chosen by a filter.) The thresholds come from data: WEAK sits 0.003 above the highest top-1 score that any of a fixed set of off-corpus junk probes reaches, and STRONG is 0.904 times the median top-1 score of the gold queries.

Importance and decay

Each node carries an importance amplifier built from registered modifiers: manual boosts, access, inheritance from its parent, and project dormancy. Retrieval ranks by relevance times effective importance, and the boost decays toward a floor. Reading a concept counts as using it, so the concept ranking reflects attention as well as activity.

I kept decay narrow on purpose. Content-age decay (a 17-day half-life on the event date, with a floor of 0.2) de-emphasizes old dated material such as email, but only for concept clustering and entity promotion. Retrieval is left alone, so an old note is still findable when I ask for it. Project dormancy fades a project's inherited boost with a 182.5-day half-life once the project stops changing.

The skill and agent library

The same server hosts a library of reusable skills and Claude Code subagent definitions. Agent sessions read and write them over MCP, and a small client pulls the approved library onto each of my machines at session start. Every write creates a pending revision that no device receives until a session has read it in full and approved it. The approval is bound to the package's hash, so what was reviewed is exactly what gets served, and approvals can't be given over the public listener.

How I measure it

I built three golden sets (retrieval, entity extraction, and the MCP tool surface) and harnesses that score the live server against them. Two rules keep the gold honest. Gold labels are built by inspecting the database directly with SQL, never by running the retrieval under test, so the system can't grade its own work. And results are compared only like-for-like: each results file records the git head, a hash of the gold set, the policy and the embedding model.

The retrieval gold comes in two sets that ask different questions and are never pooled into one number. The flat set asks what matches a phrasing: a question and the note or two that answer it. The package set asks what belongs together: every live note that is part of a topic, labeled exhaustively, with at least 30% of the gold deliberately outside any active concept so a concept router can't win by construction. The headline metric is must_recall@20, the share of the notes a correct answer must include that show up in the top 20 results, averaged over items.

A pre-registered benchmark that deleted a feature

By July, search_graph had two crawl modes. The deterministic one followed concept and entity membership mechanically. The other added an LLM "hop judge" that decided at each step which clusters and neighbors were worth crawling. I expected the LLM tier to match on recall and win on precision.

Before running anything, I wrote down four predictions and what each outcome would mean. Then I ran a 2x2 benchmark: the flat and package sets, against hybrid search and the cluster crawl, with the crawl run both deterministic and LLM-judged. The flat-set results:

Two predictions held: hybrid search reproduced its own history, and the deterministic crawl matched it exactly while displacing zero gold results, for 40 ms more at the median. One was falsified. The LLM tier lost 0.062 recall on the flat set, five times its own run-to-run spread (one of its 159 hop calls fell back to the deterministic judge, too few to explain the gap), gained nothing in precision on either set, and cost 251 LLM calls and about 772,000 provider tokens. Its median latency was about 16 times the deterministic crawl's. The fourth prediction, that the crawl would beat hybrid on the package set, was not confirmed: 0.478, 0.479 and 0.482 is a three-way tie. The pre-registered reading of a tie was that the concept layer is not selective enough to route on, and that is what I reported. Coverage was part of it: only 92 of 136 package gold notes sat in an active concept.

The pre-registered consequence of the LLM result was "ship the deterministic crawl and delete the LLM tier." I merged that the next day. search_graph is deterministic only, and asking it for the LLM judge now returns an error.

The recall audit

In October I audited recall against real use. Every retrieval-shaped MCP call is logged to a trace table with its query, results, scores, confidence and latency, and a follow-up read of a returned note counts as an implicit success signal. A read-only replica of production search, matching production result for result on 79 of 79 queries, let me test changes without touching the live database.

The audit found two problems. First, recall had fallen and nothing had alerted. The corpus had grown about four times since July, to 1,210 live notes and 7,526 chunks, and flat golden recall had drifted from 0.829 to 0.727. Both figures come from July's reference harness, which runs each golden item's original query plan. The benchmark cells above substituted plans and scored 0.895 that month, so 0.829 is the like-for-like baseline. Second, real agent traffic was much harder than the golden set. Real queries averaged 6.8 words against 11.5, and the documents that answered them sat at a median nearest-neighbor rank of about 139, against about 16 for golden items. On a small real-traffic set (12 traces, with ground truth written blind before seeing results, then adjudicated), recall at the caller's own k was between 0.155 and 0.21. I had written the golden questions as long paraphrases in each document's own vocabulary, which made them easy for vector search.

The audit also ruled out the knobs I would have tried first: pool size, the vector index's search depth, the fusion constant and arm weights all measured as non-levers.

One lever, measured

The top lever was small. Qwen3-Embedding expects an instructed query ("Instruct: ... Query: ...") against raw documents, and I had been embedding queries bare, so keyword-bag queries landed near short keyword-list notes that crowded unrelated results.

I shipped a pinned instruction prefix on October 7. Measured on the replica:

MRR is how high the first right answer ranks. The golden rows measure the curated set with each item's question run as a plain search, a different convention from the 0.727 above, so the two baselines aren't comparable. The real-traffic row measures what my agents asked, scored against adjudicated gold. It more than doubled and is still low in absolute terms. The junk-probe row is the confidence contract: cosine scores dropped with the new vectors, so I re-derived both thresholds in the same commit (WEAK 0.512, STRONG 0.556), and probes that had been labeled weak or strong now read none.

The change had costs, and I recorded them. Curated distractors in the golden top 20 rose from 6 to 11, and a few bare person-name queries now fall below the junk floor and read none. On the audit's blind gold set (a stricter cut of the same traces), the prefix took recall from 0.211 to 0.447 on the replica. The next two levers, a BM25-style lexical arm and no longer counting search impressions as access, are projected to bring it to about 0.58. Those two haven't shipped yet.

Publishing it

The private repo holds personal data, so v0.1.0 is a filtered mirror with fresh history. An allowlist manifest, a human-approved path lockfile and a scanner for personal identifiers (down to base64 runs and homoglyphs) gate every export, and only one CI-gated tool can push. I gave the public repo a new name instead of renaming the private one, because GitHub's rename redirects would let a stale remote push private history into it.

What I learned

Golden sets drift away from real use. Mine scored above 0.7 while real queries scored about 0.2, and nothing told me. Logging real traffic and building gold from it blind exposed the gap, and a regression alarm is on the list for the same reason.

I wrote the specs, the benchmark protocols and every adjudication and publication ruling, and coding agents did much of the implementation against them. The benchmarks are how I check code I didn't type.

The code is on GitHub. The README's five-minute demo runs on an invented vault with no models and no login.