How To Build A Second Brain That Actually Talks Back

Most second brain systems are storage. You clip articles, save YouTube transcripts, dump PDFs into a folder — and then you never look at them again. The information goes in to die.

Matt Wolf published a step-by-step build for a second brain that actually pulls its weight. The whole thing is free apart from your AI usage, takes about an afternoon to stand up, and runs on tools you probably already use. Here's the architecture, what's clever about it, and where it gets interesting.

The three pillars

Matt structures his system around three inputs feeding one center:

1. A wiki / knowledge base — every YouTube transcript, article, podcast, and tweet you find worth saving.
2. A CRM — names, where you met people, conversations you had, contact details.
3. A journal — daily entries that the AI reads back against your wiki and CRM before responding.

The wiki sits at the center. The CRM and journal both connect to it. When you journal about a problem, the AI answers grounded in everything you've ever clipped — not generic ChatGPT bromides.

The trick isn't the storage. It's that the journal layer turns the wiki from a passive archive into a thinking partner that actually remembers what you've been reading.

The stack

Matt's setup uses three tools:

• Obsidian — free markdown editor and graph viewer. This is your visibility layer; everything is human-readable markdown files on your disk.
• Codex (or Claude Code, or any agent runtime that supports AGENTS.md instructions) — does the actual processing.
• Obsidian Web Clipper — Chrome extension that snapshots any page or YouTube transcript directly into your vault.

Nothing proprietary. No vendor lock-in. If you decide later you'd rather use Claude Code than Codex, you swap the runtime — your data is just markdown.

The architecture (cribbed from Karpathy)

Matt gives credit where it's due: Andrej Karpathy published the original LLM-wiki design as a public GitHub gist. Matt's contribution is the journal-and-CRM layer on top.

The Karpathy core has four files in your vault:

/raw          ← original clips go here
/wiki         ← AI-generated synthesis pages live here
agents.md     ← the instructions the LLM reads
index.md      ← auto-generated catalog
log.md        ← audit trail of every change

When the agent processes /raw, it does five things in order: read the source, create or update wiki pages, extract entities (people, companies, tools, ideas, themes), update index.md, and append to log.md. Everything is markdown. Everything is interlinked.

The journal layer is what makes it sing

Matt's addition: tell the agent that any chat starting with the word journal should be treated as a journal entry. The agent then:

1. Reads the indexes for the wiki, prior journals, and CRM
2. Pulls in anything relevant
3. Writes a new journal file with a smart title and date
4. Responds to your entry grounded in your saved knowledge plus its own reasoning
5. Logs the entry to index.md and log.md

Matt's example: he journals about whether to write more clickbait-y video titles versus literal ones. The agent responds not with generic advice, but with "the creator notes in your vault push back on the second fear — your saved videos on YouTube valley of death and creator persistence both frame visible metrics as lagging indicators." It quotes back to you the things you saved that are relevant to the thing you're working through right now.

That's the moment the system stops being a notebook and starts being a thinking partner.

CRM, same trick

Same mechanism: tell the agent that any message tagged for the CRM should create or update a person's record. Matt's demo: "add to CRM: Matthew Berman. Met at Qualcomm event 2024. Hung out at CES 2025. Got lunch at TechCrunch Disrupt 2025."

Weeks later he can ask "where did I meet Matthew Berman?" and get a real answer. For anyone who networks at conferences, this alone is worth standing up the system.

The automation that makes it sustainable

The system works manually, but Matt closes the loop with Codex automations. He sets one to run hourly:

"If there are any unprocessed files in the raw directory, please process them. Then commit and push the current version to the main branch on GitHub."

Now the loop is fully passive. Clip into Obsidian during the day. Every hour, the agent processes new clips into the wiki, updates indexes, and pushes a backup to a private GitHub repo. You don't have to remember to run anything.

For the price of one Codex automation slot, you've got a self-organizing, self-backing, self-grounding knowledge base.

Where this gets interesting if you push further

Matt keeps his version simple. But the design is open-ended in productive ways:

• Add a meeting-notes folder wired to your meeting recorder of choice (Granola, Otter, etc.). The agent ingests transcripts the same way it ingests YouTube videos.
• Add custom entity folders — books, papers, places. The agent will extract and link them automatically once you describe them in agents.md.
• Swap the runtime per task. Use a cheap model for routine ingestion, a frontier model for journal responses. The agents.md design lets you specify per-operation routing.
• Layer in a vector index if your wiki gets big enough that pure markdown lookup gets slow. Most people won't need this for years, but the architecture allows it.

What to actually do this week

If this sounds like more value than your current note system is delivering:

1. Install Obsidian + the web clipper (10 minutes).
2. Clone Karpathy's LLM-wiki structure into a new vault — Codex or Claude Code can do this from a single prompt.
3. Add Matt's journal and CRM extensions to agents.md (the prompts are walked through in his video).
4. Clip three things you read this week and let the agent process them. See if the output is useful enough to make the system worth feeding.

If it works for you, set up the hourly automation and forget about it. If it doesn't, you've spent an afternoon and you still have your clips in plain markdown.

The Bottom Line

Knowledge management apps fail because they treat ingestion and recall as separate problems. Matt's setup — really Karpathy's, with Matt's journal and CRM bolted on — collapses them. You write into the system; the system writes back; the boundary between thinking and consulting your notes disappears. Stand it up once, feed it, and a few weeks later your graph view starts looking like a brain instead of a folder.