Markdown Chunker for LLMs
Split long Markdown documents into token-budget chunks for RAG pipelines and LLM context windows.
Loading tool...
What is Markdown Chunker for LLMs?
The Markdown Chunker for LLMs splits long Markdown documents into token-budget chunks built for RAG pipelines, embeddings, and LLM context windows. It parses the document structure first — headings, fenced code blocks, and paragraphs — then packs blocks into chunks that respect your token budget, optionally repeating trailing blocks as overlap and prepending heading breadcrumbs so every chunk carries its own context. Everything runs offline in your browser.
Key Benefits
- Fits any context window: choose budgets from 256 to 8192 tokens per chunk.
- Markdown-aware: code fences and tables are never cut mid-block.
- Retrieval-friendly: heading breadcrumbs give embeddings topical context.
- Boundary-safe: overlap keeps sentences from being severed at chunk edges.
- Private and offline: documents are processed locally, never uploaded.
- Pipeline-ready: JSONL export plugs straight into LangChain and LlamaIndex.
Common Use Cases
- •Preparing internal documentation for a RAG chatbot.
- •Splitting long API references into embeddable sections.
- •Chunking README files and design docs for semantic search.
- •Feeding documentation into an LLM with a limited context window.
- •Building evaluation datasets of document chunks for LLM testing.
- •Preprocessing Markdown notes for local vector databases.
How to Use the Markdown Chunker for LLMs
- Paste or upload your Markdown: Paste your documentation into the input box, or click Upload .md to load a file from your device. Nothing is sent anywhere — parsing happens in your browser.
- Set the token budget: Choose how many tokens each chunk may hold, from 256 to 8192. Common RAG recipes use 512 or 1024 with 50-100 tokens of overlap.
- Pick an estimator and options: Use characters ÷ 4 for English documents, or words × 1.3 for Thai and other non-English text. Enable heading context and code-block protection as needed.
- Copy or export the chunks: Review the generated chunks with their token counts and heading breadcrumbs, then copy any chunk, copy all, or download everything as JSONL or Markdown.
Key Features
- Token-budget chunking with presets from 256 to 8192 tokens per chunk.
- Markdown-aware splitting that keeps fenced code blocks and tables intact.
- Two token estimators: characters ÷ 4 for English, words × 1.3 for Thai and other languages.
- Configurable overlap that carries trailing blocks into the next chunk for continuity.
- Heading breadcrumbs (e.g. Guide / Install / Linux) prepended to each chunk for retrieval context.
- Export chunks as JSONL for LangChain/LlamaIndex, as Markdown, or copy one by one.
Frequently Asked Questions
How does the token count work?
The tool uses two offline heuristics: characters ÷ 4 (the classic GPT-style estimate that fits English well) and words × 1.3 (which better matches Thai and other languages where tokens per word vary). For exact billing-grade counts, run the output through your provider tokenizer — the estimates here are for planning chunk budgets, not invoicing.
Why keep code blocks intact instead of splitting them?
A code block cut in half loses its meaning: the opening fence, language tag, and logic end up in different chunks, and embeddings for each half are near-useless. The chunker treats fenced blocks as atomic units and only splits them by lines when a single block is larger than your whole budget.
What is chunk overlap for?
Overlap repeats the trailing blocks of one chunk at the start of the next, so sentences and ideas near a boundary survive in both chunks. This improves retrieval recall in RAG pipelines because a query can match either side of the seam. 10-20% of the budget is a common starting point.
What is the JSONL export for?
The JSONL (NDJSON) file has one JSON object per line with index, token estimate, heading breadcrumb, and text — exactly the shape that LangChain JSONLoader and LlamaIndex readers expect for building vector indexes.
Is my document uploaded anywhere?
No. The chunker runs entirely in your browser with client-side JavaScript. Your Markdown never leaves your device, which makes it safe for internal documentation, private API specs, and confidential policies.