# chunker
A RocketRide preprocessor node ("Text Chunker") that splits documents into smaller, overlapping chunks for downstream embedding, retrieval, and LLM-based generation.
## What it does
Receives documents on the `documents` lane and emits one document per chunk, each carrying metadata that ties it back to the source. Two strategies are available, selected by the `profile` field:
- **Sentence boundary** (default) — groups whole sentences up to `chunk_size`. Sentence boundaries take priority over the size limit, so a single sentence longer than `chunk_size` is emitted whole rather than cut mid-sentence. Pure Python (stdlib `re` only): unlike the NLTK and Spacy profiles of the General Text node, it pulls in no third-party package and downloads no language model.
- **Token-based** — splits by token count using the real `tiktoken` BPE tokenizer, sized for model context windows. The encoder is imported lazily and the `tiktoken` dependency is probed only when this strategy is selected.
### Which node do I want?
For **recursive character splitting**, use the **General Text** (`preprocessor_langchain`) node — it already exposes LangChain's `RecursiveCharacterTextSplitter` through its `default` and `recursive` profiles. This node does not reimplement it.
Reach for Text Chunker when you need something General Text does not provide:
| Need | Text Chunker | General Text (`preprocessor_langchain`) |
|---|---|---|
| Recursive character splitting | not provided | yes (`default` / `recursive` profiles) |
| Token sizing | real `tiktoken` BPE counts | byte-length estimate (`bytes/3`), UI-labelled "Estimated tokens" |
| Chunk overlap | configurable (`chunk_overlap`) | not available — fixed at `0` |
| Per-chunk character offsets | `start_char` / `end_char` on every chunk | not emitted; returns text only |
| Sentence splitting | stdlib regex, no extra deps | NLTK / Spacy profiles (extra deps + model download) |
| Dependency footprint | `tiktoken` only, and only for the token strategy | `langchain`, `langchain-core`, `langchain-text-splitters`, `transformers`, `accelerate`, `tokenizers`, `huggingface-hub` |
`chunk_overlap` characters (or tokens) are shared between consecutive chunks to preserve context across boundaries. The overlap is reserved inside `chunk_size`, so an emitted chunk never exceeds `chunk_size`, and it is honored even when a chunk fills that budget (including the hard-split path).
Each emitted chunk copies the source document (metadata is copied per chunk, never shared) and sets `chunkId`, `parentId` (the source `objectId`), `chunk_index`, `start_char`, `end_char`, and `total_chunks`. `chunkId` resets to `0` for every incoming object. Documents whose text is empty or whitespace-only are consumed and not forwarded downstream.
---
## Configuration
### Lanes
| Lane in | Lane out | Description |
|-------------|-------------|---------------------------------------------------------|
| `documents` | `documents` | Split each incoming document into one document per chunk |
### Strategies
| Profile | Strategy | Chunk size | Overlap | Best for |
|-------------------------------|------------|-------------|---------|---------------------------------------------------------------|
| Sentence Boundary *(default)* | `sentence` | 1000 chars | 200 | Coherent chunks that never split mid-sentence |
| Token-based | `token` | 512 tokens | 50 | Fitting LLM/embedding context windows (`cl100k_base` default) |
`chunk_size` is measured in characters for the sentence strategy and in tokens for the token strategy. `chunk_overlap` must be less than `chunk_size`. `encoding_name` applies only to the token strategy.
Configuring `strategy: recursive` raises at startup with a pointer to the General Text node rather than silently falling back.
### Picking a strategy for your input
The sentence strategy treats a sentence as indivisible, so `chunk_size` is a grouping target rather than a hard cap. Input with no sentence-ending punctuation — log lines, CSV rows, minified text, OCR dumps, prose in scripts that do not use `.`/`!`/`?` — contains no boundaries to group on and is emitted as a single oversized chunk.
For those inputs use the **token** strategy, which caps every chunk at `chunk_size` tokens unconditionally, or the **General Text** node's recursive splitter. The sentence strategy is the right default for ordinary punctuated prose, which is what most document pipelines carry.
---
## Schema
| Field | Type | Description | Default |
|---|---|---|---|
| `chunker.chunk_overlap` | `integer` | **Chunk overlap** Number of characters or tokens to overlap between consecutive chunks; must be less than chunk size. | `200` |
| `chunker.chunk_size` | `integer` | **Chunk size** Maximum size of each chunk (characters for the sentence strategy, tokens for the token strategy) | `1000` |
| `chunker.encoding_name` | `string` | **Token encoding** Tiktoken encoding name (only used with token strategy) | `"cl100k_base"` |
| `chunker.profile` | `string` | **Chunking strategy** Select the text chunking strategy | `"sentence"` |
## Dependencies
- `tiktoken` `>=0.7.0,<1.0.0`
## Source
[ View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/chunker)