Text Chunker

View as Markdown

A RocketRide preprocessor node ("Text Chunker") that splits documents into smaller, overlapping chunks for downstream embedding, retrieval, and LLM-based generation.

What it does

Receives documents on the documents lane and emits one document per chunk, each carrying metadata that ties it back to the source. Two strategies are available, selected by the profile field:

• Sentence boundary (default) — groups whole sentences up to chunk_size. Sentence boundaries take priority over the size limit, so a single sentence longer than chunk_size is emitted whole rather than cut mid-sentence. Pure Python (stdlib re only): unlike the NLTK and Spacy profiles of the General Text node, it pulls in no third-party package and downloads no language model.
• Token-based — splits by token count using the real tiktoken BPE tokenizer, sized for model context windows. The encoder is imported lazily and the tiktoken dependency is probed only when this strategy is selected.

Which node do I want?

For recursive character splitting, use the General Text (preprocessor_langchain) node — it already exposes LangChain's RecursiveCharacterTextSplitter through its default and recursive profiles. This node does not reimplement it.

Reach for Text Chunker when you need something General Text does not provide:

Need	Text Chunker	General Text (preprocessor_langchain)
Recursive character splitting	not provided	yes (default / recursive profiles)
Token sizing	real tiktoken BPE counts	byte-length estimate (bytes/3), UI-labelled "Estimated tokens"
Chunk overlap	configurable (chunk_overlap)	not available — fixed at 0
Per-chunk character offsets	start_char / end_char on every chunk	not emitted; returns text only
Sentence splitting	stdlib regex, no extra deps	NLTK / Spacy profiles (extra deps + model download)
Dependency footprint	tiktoken only, and only for the token strategy	langchain, langchain-core, langchain-text-splitters, transformers, accelerate, tokenizers, huggingface-hub

chunk_overlap characters (or tokens) are shared between consecutive chunks to preserve context across boundaries. The overlap is reserved inside chunk_size, so an emitted chunk never exceeds chunk_size, and it is honored even when a chunk fills that budget (including the hard-split path).

Each emitted chunk copies the source document (metadata is copied per chunk, never shared) and sets chunkId, parentId (the source objectId), chunk_index, start_char, end_char, and total_chunks. chunkId resets to 0 for every incoming object. Documents whose text is empty or whitespace-only are consumed and not forwarded downstream.

---

Configuration

Lanes

Lane in	Lane out	Description
documents	documents	Split each incoming document into one document per chunk

Strategies

Profile	Strategy	Chunk size	Overlap	Best for
Sentence Boundary (default)	sentence	1000 chars	200	Coherent chunks that never split mid-sentence
Token-based	token	512 tokens	50	Fitting LLM/embedding context windows (cl100k_base default)

chunk_size is measured in characters for the sentence strategy and in tokens for the token strategy. chunk_overlap must be less than chunk_size. encoding_name applies only to the token strategy.

Configuring strategy: recursive raises at startup with a pointer to the General Text node rather than silently falling back.

Picking a strategy for your input

The sentence strategy treats a sentence as indivisible, so chunk_size is a grouping target rather than a hard cap. Input with no sentence-ending punctuation — log lines, CSV rows, minified text, OCR dumps, prose in scripts that do not use ./!/? — contains no boundaries to group on and is emitted as a single oversized chunk.

For those inputs use the token strategy, which caps every chunk at chunk_size tokens unconditionally, or the General Text node's recursive splitter. The sentence strategy is the right default for ordinary punctuated prose, which is what most document pipelines carry.

---

Schema

Field	Type	Description	Default
chunker.chunk_overlap	integer	Chunk overlap Number of characters or tokens to overlap between consecutive chunks; must be less than chunk size.	200
chunker.chunk_size	integer	Chunk size Maximum size of each chunk (characters for the sentence strategy, tokens for the token strategy)	1000
chunker.encoding_name	string	Token encoding Tiktoken encoding name (only used with token strategy)	"cl100k_base"
chunker.profile	string	Chunking strategy Select the text chunking strategy	"sentence"

Dependencies

• tiktoken >=0.7.0,<1.0.0