Text Chunker
A RocketRide preprocessor node ("Text Chunker") that splits documents into smaller, overlapping chunks for downstream embedding, retrieval, and LLM generation. Pick it when you need sentence-boundary splitting with no extra dependencies, or token-accurate splitting sized with the real tiktoken BPE tokenizer.
What it does
The node receives documents on its documents lane, splits each one's text with the configured strategy, and emits one document per chunk on the same lane. Every emitted chunk copies the source document and gets its own metadata — chunkId, parentId (the source objectId), chunk_index, start_char, end_char, and total_chunks — so downstream nodes can trace a chunk back to its origin or reassemble the original. The incoming documents themselves are never forwarded; only chunks continue down the pipeline, and documents whose text is empty or whitespace-only are consumed and produce nothing.
It deliberately covers only the two strategies the engine does not otherwise have: sentence-boundary grouping and real token counting. For recursive character splitting use the General Text (preprocessor_langchain) node, which already exposes LangChain's RecursiveCharacterTextSplitter; configuring recursive here fails at startup with a pointer to that node rather than falling back silently.
Lanes
| Lane in | Lane out | Description |
|---|---|---|
documents | documents | Split each incoming document and emit one document per chunk. |
Profiles
Default: Sentence Boundary - Splits at sentence endings for coherent chunks (sentence).
| Profile | Strategy | Chunk size | Overlap | Best for |
|---|---|---|---|---|
sentence (default) | sentence | 1000 characters | 200 | Ordinary punctuated prose that should never be cut mid-sentence. |
token | token | 512 tokens | 50 | Chunks that must fit a model's context or an embedding input limit. |
The sentence strategy splits on sentence-ending punctuation (., !, ?) followed by whitespace, using only the Python standard library — no third-party package and no language-model download. It treats a sentence as indivisible, so chunk_size is a grouping target rather than a hard cap: a single sentence longer than chunk_size is emitted whole.
The token strategy encodes the text with tiktoken and slices it by token count, so every chunk is capped at chunk_size tokens unconditionally. Its dependency is probed and its encoder imported only when this strategy is selected. Choose it for input with no sentence-ending punctuation — log lines, CSV rows, OCR dumps, minified text — where the sentence strategy finds no boundaries to group on and emits one oversized chunk.
Configuration
Pick the profile first: it sets the strategy along with chunk size, overlap, and (for token) the encoding, and the defaults are sensible for each. The configuration panel then shows only the fields that apply to the chosen strategy, so there is nothing else most pipelines need to touch. All three values are validated at startup — a non-positive chunk size, a negative overlap, or an overlap that is not smaller than the chunk size stops the node rather than degrading quietly.
Chunk size
The maximum size of a chunk, measured in characters for the sentence strategy and in tokens for the token strategy — the same field means different units depending on the profile. Larger values preserve more context per chunk but retrieve less precisely; smaller values sharpen retrieval and cost more chunks. The profile defaults (1000 characters, 512 tokens) suit general documents; when the chunks feed an embedding model or an LLM prompt, size them against that model's input limit using the token strategy so the count is exact rather than estimated.
Chunk overlap
How much of each chunk is repeated at the start of the next one, in the same units as chunk size, so a sentence or idea straddling a boundary still appears whole in at least one chunk. It must be less than the chunk size. With the token strategy the next window starts at a progressing character boundary whose exact source suffix fits within the token overlap, so the effective overlap can be smaller than configured but never larger. With the sentence strategy the trailing sentences of the finished chunk are carried forward as long as their combined span fits within the overlap, so the effective overlap likewise lands on a sentence boundary. Set it to 0 to disable overlap entirely; the profile defaults of 200 characters and 50 tokens are roughly a fifth and a tenth of their chunk sizes.
Token encoding
The tiktoken encoding used to count and slice tokens. It applies only to the token strategy and is ignored by sentence. Match it to the model the chunks are destined for, since token boundaries differ between encodings: cl100k_base (the default) for GPT-4, GPT-3.5-turbo and text-embedding-ada-002, o200k_base for GPT-4o and newer models, p50k_base for Codex models, and r50k_base for GPT-3 models. The wrong encoding still produces chunks, but their token counts will not match what the consuming model measures.
Notes
Chunk metadata
chunkId is a running counter across everything emitted for one incoming object and resets to 0 when the next object opens; chunk_index restarts at 0 for each source document, and total_chunks is that document's chunk count. start_char and end_char are exact character offsets into the source text for both strategies.
Unicode boundaries
A token boundary can fall in the middle of a multi-byte Unicode character, and BPE merges can change when a substring is encoded on its own. The token strategy uses the original token stream only to estimate each window, then chooses a character boundary and independently tokenizes the exact source slice before emitting it. This prevents replacement characters, corrupted provenance offsets, and oversized chunks. If chunk_size is too small to contain even one complete character (for example, a one-token limit for a character represented by two tokens), chunking stops with a clear configuration error.
Upstream docs
Schema
| Field | Type | Description | Default |
|---|---|---|---|
chunker.chunk_overlap | integer | Chunk overlap Number of characters or tokens to overlap between consecutive chunks; must be less than chunk size. | 200 |
chunker.chunk_size | integer | Chunk size Maximum size of each chunk (characters for the sentence strategy, tokens for the token strategy) | 1000 |
chunker.encoding_name | string | Token encoding Tiktoken encoding name (only used with token strategy) | "cl100k_base" |
chunker.profile | string | Chunking strategy Select the text chunking strategy | "sentence" |
Dependencies
tiktoken>=0.7.0,<1.0.0