What RAG Actually Is#

RAG isn’t magic - it’s a straightforward pipeline:

Figure 2 - RAG pipeline diagram showing four steps: Embed the question, Retrieve similar chunks from Qdrant, Augment prompt with context, Generate answer with Claude Haiku

The key insight: the LLM doesn’t hallucinate because it’s grounded in your actual notes. No making things up - only synthesizing what’s already in your vault.

Architecture: From Notes to Chat#

Building on the semantic note network we already had, we added three new components:

1
class NoteChunker:
2
    """Splits notes into overlapping chunks for embedding."""
3

4
    DEFAULT_CHUNK_SIZE = 2000   # chars (~500 tokens)
5
    DEFAULT_OVERLAP = 400      # chars (~100 tokens)
6

7
    def chunk_note(self, content: str) -> list[str]:
8
        frontmatter, body = self._extract_frontmatter(content)
9
        context_prefix = self._build_context_prefix(frontmatter)
10
        paragraphs = self._split_into_paragraphs(body)
11
        return self._combine_into_chunks(paragraphs, context_prefix)

1
# Collection: obsidian_chat_chunks
2
# Vectors: 1536 dimensions (text-embedding-3-small)
3
# Payload: file_path, title, chunk_index, chunk_text

1
class ObsidianChatService:
2
    """RAG-based chat service for Obsidian vault."""
3

4
    DEFAULT_SEARCH_LIMIT = 5
5
    DEFAULT_SCORE_THRESHOLD = 0.50
6
    MAX_CONTEXT_CHARS = 8000
7

8
    async def chat(self, question: str) -> ChatResponse:
9
        # Step 1: Search for relevant chunks
10
        sources = self.search(
11
            query=question,
12
            limit=self.DEFAULT_SEARCH_LIMIT,
13
            threshold=self.DEFAULT_SCORE_THRESHOLD,
14
        )
15

16
        # Step 2: Build context from sources
17
        context = self._build_context(sources)
18

19
        # Step 3: Generate response with Haiku
20
        answer = await self._generate_response(question, context)
21

22
        return ChatResponse(
23
            answer=answer,
24
            sources=sources,
25
            success=True,
26
        )

The Chunking Strategy#

Chunking is where most RAG implementations go wrong. Chunk too large, and you miss specific passages. Chunk too small, and you lose context.

We chose 2,000 characters (~500 tokens) with 400-character overlap. Here’s why:

The overlap ensures that if important information spans a chunk boundary, at least one chunk will contain it completely.

1
def _split_into_paragraphs(self, text: str) -> list[str]:
2
    """Split on double newlines while preserving structure."""
3
    text = text.replace("\r\n", "\n")
4
    paragraphs = re.split(r"\n\n+", text)
5
    return [p.strip() for p in paragraphs if p.strip()]

1
Title: Building RAG Systems | Description: A guide to retrieval-augmented
2
generation | Tags: ai/rag, ai/llm, coding/python
3

4
RAG combines the power of large language models with external knowledge
5
retrieval. Instead of relying solely on the model's training data...

Retrieval: Finding What Matters#

Search is straightforward - embed the question and find similar chunks:

1
def search(
2
    self,
3
    query: str,
4
    limit: int = 5,
5
    threshold: float = 0.50,
6
) -> list[ChatSource]:
7
    """Search for relevant chunks."""
8

9
    sources = self.embeddings.search_by_text(
10
        query_text=query,
11
        limit=limit,
12
        score_threshold=threshold,
13
    )
14

15
    # Deduplicate by file_path, keeping highest score per note
16
    unique_sources = {}
17
    for source in sources:
18
        if source.file_path not in unique_sources:
19
            unique_sources[source.file_path] = source
20
        elif source.score > unique_sources[source.file_path].score:
21
            unique_sources[source.file_path] = source
22

23
    return sorted(unique_sources.values(), key=lambda x: x.score, reverse=True)

Generation: The Prompt That Works#

The system prompt sets the tone:

1
system_prompt = """You are a helpful assistant that answers questions
2
about the user's Obsidian vault notes.
3

4
Use the provided context from their notes to answer questions accurately.
5
If the context doesn't contain relevant information, say so honestly.
6
Keep answers concise but informative.
7
When referencing information, mention which source note it came from."""

The key instructions:

1. Use the context - Don’t make things up
2. Be honest - Say when you don’t know
3. Cite sources - Attribution builds trust

The user prompt includes the retrieved context:

1
user_message = f"""Context from your notes:
2

3
[Source 1: Building RAG Systems]
4
RAG combines the power of large language models with...
5

6
[Source 2: Vector Database Comparison]
7
For semantic search, Qdrant provides...
8

9
Question: {question}
10

11
Please answer based on the context above."""

The Frontend Experience#

The chat interface has three main areas:

Figure 3 - Chat dashboard layout mockup showing chat panel on left with conversation history, note preview panel on right showing selected source, and sources list with similarity scores

1
function SourceList({ sources, onSourceClick }) {
2
  return (
3
    <Collapsible>
4
      <CollapsibleTrigger>
5
        Sources ({sources.length})
6
      </CollapsibleTrigger>
7
      <CollapsibleContent>
8
        {sources.map((source) => (
9
          <button
10
            key={source.file_path}
11
            onClick={() => onSourceClick(source)}
12
            className="text-left hover:bg-muted"
13
          >
14
            {source.title || source.file_name}
15
            <span className="text-muted">
16
              ({Math.round(source.score * 100)}%)
17
            </span>
18
          </button>
19
        ))}
20
      </CollapsibleContent>
21
    </Collapsible>
22
  );
23
}

1
function getObsidianUri(filePath: string, vaultName: string): string {
2
  const pathParts = filePath.replace(/\\/g, '/').split('/');
3
  const fileName = pathParts[pathParts.length - 1].replace(/\.md$/, '');
4
  const subFolder = pathParts[pathParts.length - 2] || '';
5
  const fileRef = subFolder ? `${subFolder}/${fileName}` : fileName;
6

7
  return `obsidian://open?vault=${encodeURIComponent(vaultName)}&file=${encodeURIComponent(fileRef)}`;
8
}

Auto-Indexing: Keeping the Index Fresh#

New notes should be searchable immediately. We hook into the save process:

1
@app.post("/api/save-note")
2
async def save_note(note_data: dict):
3
    # Save the file
4
    file_path = save_note_to_disk(note_data)
5

6
    # Index for similarity search
7
    index_note_for_similarity(file_path)
8

9
    # Index for chat
10
    chat_service = ObsidianChatService()
11
    chat_service.index_note(file_path)

No manual re-indexing required. Ask a question about a video you just processed, and it’s already in the index.

Bulk Indexing: The Initial Setup#

For existing vaults, we provide a bulk indexing script:

1
def main():
2
    service = ObsidianChatService()
3
    stats = service.index_all_notes()
4

5
    print(f"Indexed {stats['indexed_count']} notes")
6
    print(f"Total chunks: {stats['total_chunks']}")
7
    print(f"Errors: {len(stats['errors'])}")

Running it on our vault:

1
python scripts/index_vault_for_chat.py --verbose
2

3
Indexing vault: C:\Users\...\Obsidian\Vault\Notes
4
Found 1051 markdown files
5
...
6
Indexed 1051 notes
7
Total chunks: 5047
8
Errors: 0
9
Time elapsed: 8m 32s

~5,000 chunks indexed in under 9 minutes. That’s the entire vault ready for questions.

Performance Characteristics#

The bottleneck is LLM generation, not retrieval. Qdrant’s sub-millisecond search means the experience feels snappy.

The Complete Stack#

Everything working together:

Figure 4 - Complete chat architecture diagram showing the full flow from user question through FastAPI, OpenAI embedding, Qdrant search, context building, Claude Haiku generation, and response display with source attribution

What Makes This Work#

Sample Interactions#

User: “What techniques help with trading discipline?”

Assistant: Based on your notes, several techniques can help with trading discipline:

1. Pre-trade checklists - From “Trading Psychology Fundamentals”: Having a written checklist before each trade forces you to verify your setup meets all criteria.

2. Journal reviews - “My Trading Journal Process” describes reviewing each trade weekly to identify emotional patterns.

3. Position sizing rules - “Risk Management 101” emphasizes that fixed position sizing removes emotional decision-making.

Sources: Trading Psychology Fundamentals (78%), My Trading Journal Process (72%), Risk Management 101 (68%)

User: “How does LangGraph differ from CrewAI?”

Assistant: According to your notes, the key differences are:

LangGraph (from “LangGraph Architecture Deep Dive”):

• Graph-based workflow definition
• Explicit state management
• Fine-grained control over agent transitions

CrewAI (from “CrewAI Framework Overview”):

• Role-based agent definition
• Automatic task delegation
• Simpler setup for common patterns

Your note “Comparing Agent Frameworks” summarizes: “LangGraph when you need control, CrewAI when you need speed.”

Sources: LangGraph Architecture Deep Dive (82%), CrewAI Framework Overview (79%), Comparing Agent Frameworks (71%)

Key Takeaways#

1. Chunking is crucial. 2,000 chars with 400 overlap works well. Include frontmatter in first chunk.

2. Lower your threshold. 0.50 retrieves more than note-linking’s 0.70. The LLM handles noise.

3. Deduplicate results. One chunk per note prevents monopolization.

4. Show sources. Attribution builds trust. Let users verify.

5. Complete the loop. “Open in Obsidian” makes the chatbot useful, not just interesting.

6. Auto-index on save. Manual re-indexing kills the user experience.

The Full Journey#

This chatbot is the culmination of our knowledge management system:

1. YouTube transcript extraction
2. AI-powered note generation
3. Semantic indexing in Qdrant
4. Auto-linking related notes
5. Tag curation and taxonomy
6. Batch processing with Anthropic API
7. Modern React frontend
8. And now: RAG-powered chat

Each phase built on the previous. The semantic index that powers note similarity also powers chunk retrieval. The frontmatter that organizes tags also enriches embeddings. The infrastructure that serves the UI also serves the chatbot.

This is what compound progress looks like.

Related Articles#

• From YouTube to Knowledge Graph - System overview
• Building a Semantic Note Network - Vector search foundation
• Anthropic Batch API in Production - Cost-effective processing
• Modern Python Web Stack 2025 - Architecture patterns

This article is part of our series on building AI-powered knowledge management tools. Written with assistance from Claude Code.