The problem
I learn a lot from short videos: a new tool, a prompt technique, a business idea. I saved hundreds of them and remembered almost none. Saving isn't learning, and a note nobody links to is a dead note.
I wanted my knowledge to compound on its own, and to be usable by my AI agents, not only by me.
The system
- Capture. Three sources: a personal saves export, TikTok favourites and 20 YouTube channels across 7 categories.
- Transcribe. Locally with Whisper; YouTube uses native subtitles.
- Extract and score. An LLM pulls out tools, prompts, techniques and ideas, then gives each note a 0–100 signal score. Business ideas are also scored against a founder-fit filter.
- Self-improving catalogue. New tool names found in notes are added to the catalogue automatically.
- Write into the vault. Obsidian, PARA structure, a strict data contract, related notes linked.
- Serve the agents. The vault is indexed (full-text, BM25) so Claude Code searches it before answering.
The night shift
At 00:30 a scheduled Claude Code agent reads the day's journal and changes, consolidates decisions and lessons into the right notes, and fixes links. It runs inside hard limits: an explicit time window, an 18-minute timeout, one retry and a lock file. If it fails, it leaves a failure trace in the next day's journal. The runner also checks for an explicit NIGHTLY_OK line, so in the morning “nothing to do” is never confused with “didn't run”.
Decisions worth telling
- Free first, paid as plan B. Extraction tries the Claude CLI already covered by my subscription, then the API, then a regex fallback, and stops cleanly after 5 failures in a row.
- A data contract for the vault. YAML tag lists, title equals filename,
created/updatedon every note. After the clean-up, broken links went to zero; the auditor counted 131 orphans on 2026-09-26. - Failure is designed in. Atomic writes and per-item error handling. An audit found a silent data-loss risk and a path-traversal bug in the vault writer; both are fixed.
- Resilient to upstream changes. When a source API changed, the pipeline switched to a fallback mode and kept running.
Numbers
Vault counters dated 2026-09-26 are the night agent's own, published as last-run.json. The undated figures are point-in-time counts from the build and are not refreshed automatically.
It runs twice a day (09:00 and 21:00), the nightly agent at 00:30, the profile rebuild at 07:17 and the backup at 23:00 — five runs, four systems. The vault itself stays private. The reusable core is open source as nightshift: pipeline, contract-aware vault tools and the night-shift runner, with an offline demo that runs in under a minute.