engram

--- name: engram version: 1.2.3 description: > Local associative memory graph for an OpenClaw workspace, shipped as a set of CLI scripts: engram.py (store/recall/msgctx/task ledger over SQLite+FTS+embeddings), engram-migrate.py (markdown to nodes), engram-serve.py and bq.py (optional warm daemon over a Unix socket), embeddings.py (ONNX e5 vectors), engram-backup.py (nightly SQLite backup with retention), engram-sleep.py (nightly consolidation, optional local-Ollama distillation), engram-maintenance.py and engram-hygiene.py (dedupe/orphan/archive tiering), engram-aliases.py (alias curation), engram-rewire.py (opt-in: slims MEMORY.md, adds marked cron lines, asks first), engram-uninstall.py (full revert). Task evidence strings run through the shell. No network calls except to localhost Ollama, and only if you enable it. --- # Engram — associative memory as a skill ## Powers, up front Reads and writes one SQLite file under your workspace. Migration and rewire ask before touching `MEMORY.md`, `AGENTS.md` or your crontab, and uninstall reverts them from a manifest. Task evidence strings run through the shell. The only network call is to `localhost:11434` (Ollama) when you turn it on. No email, no credential lookup, no clipboard, nothing outside the workspace. ## What this does Engram gives an agent a memory that answers questions instead of a file it has to re-read. It stores typed nodes in SQLite, links them, and retrieves them with a hybrid of keyword search, embeddings, tags and graph hops. It migrates the `MEMORY.md`, `memory/*.md` and `AGENTS.md` you already have, and it uninstalls back to exactly what it found. Everything runs locally. No cloud call is made at runtime. ## Install ```bash ./install.sh [workspace_dir] ``` The workspace defaults to the parent of this skill directory. The installer stops after a migration dry-run. It does not change your markdown, and it refuses to overwrite an existing `engram/` directory. ## Flow 1. **Preflight** — Python 3.10 or newer is required. numpy, onnxruntime, flashrank and ollama are optional; each one that is missing degrades one step, loudly, and never stops the install. With none of them Engram runs keyword-only. 2. **Stage** — copies twelve engine files into `<workspace>/engram/`, the three cron-driven tools into `<workspace>/scripts/`, and writes `engram/engram.json` with every path. 3. **Init** — creates a fresh `engram.db` with the full schema, and starts the warm daemon if ONNX is present. 4. **Smoke** — stores one node and recalls it. This is the first recall on a brand-new database, the exact path that crashed in v1.1.x. A failure here stops the install before it touches anything else. 5. **Migrate (dry-run)** — parses your markdown into typed nodes, each tagged `source:<file>#<line>`, and writes `migration-report.md`. Nothing is stored until you run `--apply`. The original files are never modified. 6. **Verify** — `engram-seed-battery.py` mines 20 recall cases from your own migrated content and adds 3 negative controls. It passes at 90% of the positives and 3/3 negatives. A non-zero exit or a traceback from the engine fails the whole run, whatever the score says. Steps 5 and 6, and the optional rewire, are yours to run: ```bash python3 engram/engram-migrate.py --apply python3 engram/engram-seed-battery.py python3 engram/engram-rewire.py # optional, asks before touching anything (--yes to skip) ``` `engram-rewire.py` backs up `MEMORY.md` and `AGENTS.md` to `.engram-backup/<timestamp>/` with a manifest, slims `MEMORY.md` to a pointer, adds an Engram section to `AGENTS.md`, and appends three cron lines marked `# ENGRAM-MANAGED`: backup at 03:15, sleep consolidation dry-run at 03:45, and a weekly battery on Sunday at 09:00. ## Daily commands ```bash engram recall "hosting setup" # ask a question, get the answer engram msgctx "<incoming message>" # short context block for one message engram store --name N --type fact --content "..." engram startup # WIP plus session briefing engram task add <name> --brief "..." --evidence "<shell command>" engram task probe # what is open, what has gone stale engram intend "when X happens" "do Y" # fires when the trigger appears engram asof "office address" --date 2024-06-01 ``` (`engram` here is `python3 engram/engram.py`, or `python3 engram/bq.py` to go through the warm daemon when it is running.) ### The per-message hook `msgctx` is the command an agent should call on every incoming message, not `recall`. It takes the raw message, decides whether the message is a question worth answering from memory, and returns a short block — usually around 200 tokens — or nothing at all. `recall` is the deeper, more expensive lookup for when the agent knows it needs to go digging. ```bash python3 engram/bq.py msgctx "$USER_MESSAGE" ``` Inject the output ahead of the message. When it prints nothing, inject nothing. ### The task ledger `engram task` is a durable ledger for work that outlives one session. A task carries a brief that can respawn it, an evidence command whose exit code decides whether it is really done, and a status. **Evidence commands run through the shell with your privileges, so treat them as code you wrote, not data.** Only `task add` writes that field; never store evidence from an untrusted source. `engram task probe` reports tasks that are open, stale, or finished but unclosed. `task done` refuses to close a task whose evidence command fails, unless you pass `--force`. ## Uninstall ```bash python3 engram/engram-uninstall.py # restore and archive the db python3 engram/engram-uninstall.py --purge # also delete the archived db ``` Restores every file from the manifest, removes the marked cron lines, stops the daemon, archives `engram.db`. Safe to run twice. ## Configuration Every path and model setting lives in `engram/engram.json`: `workspace`, `db_path`, `socket_path`, `embed_model`, `ollama_url`, `ollama_embed_model`, and `semantic_mode` (`onnx`, `ollama`, or `off`). The engine reads all of them from there and hardcodes none. Point `ENGRAM_CONFIG` at another file to run a second workspace from the same install. ## Degradation | Missing | What happens | |---|---| | flashrank | Lexical rerank (RRF only, no cross-encoder) | | onnxruntime | No ONNX embeddings; ollama or keyword-only | | ollama | No ollama embeddings; ONNX or keyword-only | | numpy | Keyword-only mode | Every step down prints a warning. None of them is fatal. ## Engine files | File | Purpose | |---|---| | `engram.py` | The engine. store, recall, msgctx, task, link, tags, wip, startup. **Generated — see below.** | | `bq.py` | Thin client: daemon socket, CLI fallback | | `engram-serve.py` | Warm daemon, keeps the ONNX model resident | | `embeddings.py` | Semantic engine (ONNX multilingual-e5-small) | | `engram-migrate.py` | Markdown to nodes | | `engram-rewire.py` | Backup, slim `MEMORY.md`, add crons | | `engram-uninstall.py` | Full revert from the manifest | | `engram-seed-battery.py` | Install verification battery | | `engram-battery.py` | Larger adversarial recall battery | | `engram-sleep.py` | Nightly consolidation | | `engram-backup.py` | Nightly backup via the SQLite backup API | | `engram-maintenance.py` | Weekly health job: junk, casing, stale vectors, orphans, duplicates. **Generated.** | | `engram-hygiene.py` | `tier` / `merge` / `link` migration, dry-run by default. **Generated.** | | `engram-aliases.py` | Writes curated aliases into `props.aliases` and re-embeds. **Generated.** | | `aliases.example.json` | Template for your own alias spec. Copy to `aliases.json` and edit. | | `bpconfig.py` | Config loader, single source of truth | ## Aliases An alias is new vocabulary for a node whose real name uses your jargon while the question uses the asker's words. "owner contact details" and a node called "Alice Johnson" share no word, in any channel, so no amount of ranking finds it. One alias does. ```bash cp engram/aliases.example.json engram/aliases.json # then edit it python3 engram/engram-aliases.py engram/engram.db --spec engram/aliases.json python3 engram/engram-aliases.py engram/engram.db --spec engram/aliases.json --apply ``` Keys are node ids and each entry names the node as a safety net, so a spec pointed at the wrong database is refused rather than applied. Aliases are free to carry: `props` is already in the full-text index, `embeddings.py` embeds them with the name, and the name channel treats each one as an alternative name — which makes an aliased node pin-eligible whatever its type. One rule: no alias repeats a word the node's name already has. That is a ranking hack, not a vocabulary fix. ## Hygiene ```bash python3 engram/engram-hygiene.py engram/engram.db tier,merge,link # counts only python3 engram/engram-hygiene.py engram/engram.db merge --apply # writes ``` Without `--apply` the database is opened read-only. Merging is **not** idempotent — a second pass merges the survivors of the first — so every applied step leaves a marker in `config` and refuses to repeat itself without `--force`. Duplicates need an identical normalised name and type, or cosine ≥ 0.97 *and* name-token Jaccard ≥ 0.6 *and* content lengths within 20%; pairs that clear the first two bars but not the third go to a review file, because merging them would drop text. ## Tiering is experimental and ships OFF `nodes.tier`, the archive rule and `recall --all` are all in 1.2.0, and **nothing marks a node `archive`**, so the filter does nothing until you switch it on. That is deliberate. Switching it on measured **16 points off the recall headline**, because the ranking constants (`NAME_MATCH_MAX_DF`, `GATE_EVIDENCE_DF_MAX`, the IDF floors) count document frequency over `nodes_fts`, which indexes archived nodes too — so hiding half the graph changes what every threshold means without changing any threshold. **Make document frequency tier-aware before turning tiering on**: a tier-filtered FTS index, or df computed over the visible corpus. On the workspace this was measured against, 65% of the graph is transport exhaust, so this is the largest remaining token win — which is why the mechanism ships rather than being held back. ## The engine files are generated, not edited Five files under `engine/` are built by `build.py` from upstream sources, with a rename map, replaced regions, and a leak gate that refuses to write if a name, path, id or internal document reference from the source workspace survives. A banner at the top of each generated file says so. For contributors: ```bash python3 build.py # regenerate every target python3 build.py --list # which file comes from where python3 build.py --check # exit 1 if a checked-in file is stale python3 build.py --diff # what would change bash ci/run-battery.sh # build check, fixture + tools, smoke recall, battery ``` `build.py` and `engine/engram_extras.py` are build-time only. `install.sh` does not copy them into a workspace. Before 1.2.0 the engine was maintained by hand and drifted from its upstream. The generated build is what ended that. The engine files that are *not* in `build.py --list` — `engram-sleep.py`, `engram-serve.py`, `bpconfig.py` and the rest — have no upstream twin and are still maintained by hand; that is the remaining drift surface, and it shrinks each time a file moves into the table. ## Constraints - Never overwrites an existing `engram/` directory. - The migration dry-run is mandatory. `--apply` is a separate, human step. - Markdown is never rewritten without a backup manifest. - Uninstall restores from the manifest and is safe to repeat.

存档时间线

版本索引永久保留；内容副本只保留最新 1 份，更早版本仅留索引与哈希（存档时间线的证据链不会因此断裂）。

下载存档内容副本