General Text

View as Markdown

A RocketRide preprocessor node that splits general text into documents for embedding or LLM processing. Choose it over the code preprocessor for prose and over the LLM preprocessor when deterministic, local splitter behavior is enough.

About LangChain

LangChain provides the langchain_text_splitters classes used by this node. The node selects one of those classes from its profile and supplies the chunk size and length function it uses for splitting.

What it does

Splits incoming text and table content on the documents lane. It offers general, Markdown, LaTeX, sentence-oriented, and fixed-separator splitters, whereas the code preprocessor extracts code syntax and the LLM preprocessor asks a connected model to make chunking decisions. Text is accumulated until an object closes; tables are split as they arrive.

Lanes

Lane in	Lane out	Description
text	documents	Split accumulated text when the object closes.
table	documents	Split each incoming table immediately.

Profiles

Default: default, the recursive character splitter.

Profile	Splitter
default (default)	RecursiveCharacterTextSplitter
recursive	RecursiveCharacterTextSplitter
character	CharacterTextSplitter
markdown	MarkdownTextSplitter
latex	LatexTextSplitter
nltk	NLTKTextSplitter
spacy	SpacyTextSplitter
custom	RecursiveCharacterTextSplitter

Configuration

Start with the default profile and string-length mode for ordinary prose. Pick a content-specific profile when its boundaries matter, then set the size in characters or estimated tokens. The generated schema lists the profile-owned fields; the choices below explain their operational effects.

Text splitter

The profile fixes the splitter class; changing the class field to another value fails configuration validation. Use markdown for Markdown, latex for LaTeX, character for one fixed separator, nltk for the NLTK splitter, or spacy for the spaCy splitter. The custom selector is present in the configuration but is also fixed to RecursiveCharacterTextSplitter, so it currently behaves like default.

Split by and chunk size

String-length mode uses the configured string length, 512 by default. Token mode estimates tokens as UTF-8 byte length divided by 3, rounded up; this is a conservative estimate rather than a tokenizer's count. Choose token mode when a downstream model or embedding service has a token budget, and lower the size when chunks are rejected for being too long.

Split separators

For the recursive profile, write a comma-separated sequence of quoted Python string literals, such as '\n\n', '\n', ' ', ''. The node parses that sequence with ast.literal_eval; every item must be a string or startup fails. For the character profile, supply exactly one such literal. Change recursive separators to prefer domain boundaries before falling back to shorter ones.

Model

The spaCy profile uses the selected spaCy pipeline, defaulting to en_core_web_sm. Choose one of the listed English, German, French, or Spanish models for text in that language. If the pipeline is absent, the node downloads it before creating the splitter; this may delay the first run.

Notes

Downloads and output

The NLTK profile installs its extra dependencies when selected and downloads punkt when it cannot find it; it also attempts to download punkt_tab when needed. All splitters use zero chunk overlap. Every emitted document receives a sequential chunkId per object; table documents also receive a tableId.

Token-budget safety net

When token mode also has a positive max_model_tokens configuration value, the node subtracts token_safety_margin (32 by default), caps the requested chunk size to the result, and force-splits any remaining oversized output. The byte-per-token estimate defaults to 3.0 and can be changed through bytes_per_token; lowering it estimates more tokens and is the safer choice.

Upstream docs

• LangChain text splitters

---

Schema

Field	Type	Description	Default
langchain.splitter.character.separator	string	Split separator	"\"\\n\""
langchain.splitter.character.splitter	string	Splitter class (set by profile) Fixed to CharacterTextSplitter by the 'Character Text Splitter' profile and cannot be changed here. To use a different splitter, change the 'Text splitter' selector above to the matching profile. Editing this to another class fails validation with 'must be equal to constant'.	const: "CharacterTextSplitter"
langchain.splitter.custom.splitter	string	Splitter class (set by profile) Fixed to RecursiveCharacterTextSplitter by the 'Custom' text-splitter profile and cannot be changed here; the custom profile currently behaves like default. To use a different splitter, change the 'Text splitter' selector above to the matching profile. Editing this to another class fails validation with 'must be equal to constant'.	const: "RecursiveCharacterTextSplitter"
langchain.splitter.default.splitter	string	Splitter class (set by profile) Fixed to RecursiveCharacterTextSplitter by the 'Default' text-splitter profile and cannot be changed here. To use a different splitter, change the 'Text splitter' selector above to the matching profile (for example Markdown for MarkdownTextSplitter). Editing this to another class fails validation with 'must be equal to constant'.	const: "RecursiveCharacterTextSplitter"
langchain.splitter.latex.splitter	string	Splitter class (set by profile) Fixed to LatexTextSplitter by the 'Latex Text Splitter' profile and cannot be changed here. To use a different splitter, change the 'Text splitter' selector above to the matching profile. Editing this to another class fails validation with 'must be equal to constant'.	const: "LatexTextSplitter"
langchain.splitter.markdown.splitter	string	Splitter class (set by profile) Fixed to MarkdownTextSplitter by the 'Markdown Text Splitter' profile and cannot be changed here. To use MarkdownTextSplitter, select 'Markdown Text Splitter' in the 'Text splitter' selector above rather than editing this field. Editing this to another class fails validation with 'must be equal to constant'.	const: "MarkdownTextSplitter"
langchain.splitter.mode	string	Split by	"strlen"
langchain.splitter.nltk.splitter	string	Splitter class (set by profile) Fixed to NLTKTextSplitter by the 'NLTK Text Splitter' profile and cannot be changed here. To use a different splitter, change the 'Text splitter' selector above to the matching profile. Editing this to another class fails validation with 'must be equal to constant'.	const: "NLTKTextSplitter"
langchain.splitter.profile	string	Text splitter Selects the splitter profile. Each profile locks one LangChain splitter class and shows only that splitter's options, so the splitter class is not chosen independently of the profile. Pick the profile that matches your content: Markdown for .md, Latex for LaTeX, Character/NLTK/Spacy as needed. Editing a profile's splitter field to a different class fails schema validation with a 'must be equal to constant' error; change this selector instead.	"default"
langchain.splitter.recursive.separators	string	Split separators	"'\\n\\n', '\\n', ' ', ''"
langchain.splitter.recursive.splitter	string	Splitter class (set by profile) Fixed to RecursiveCharacterTextSplitter by the 'Recursive Character Text Splitter' profile and cannot be changed here. To use a different splitter, change the 'Text splitter' selector above to the matching profile. Editing this to another class fails validation with 'must be equal to constant'.	const: "RecursiveCharacterTextSplitter"
langchain.splitter.spacy.model	string	Model	"en_core_web_sm"
langchain.splitter.spacy.splitter	string	Splitter class (set by profile) Fixed to SpacyTextSplitter by the 'Spacy Text Splitter' profile and cannot be changed here. To use a different splitter, change the 'Text splitter' selector above to the matching profile. Editing this to another class fails validation with 'must be equal to constant'.	const: "SpacyTextSplitter"
langchain.splitter.strlen	number	String length	512
langchain.splitter.tokens	number	Number of tokens	512

Dependencies

• langchain
• langchain-text-splitters
• langchain-core
• accelerate
• transformers
• tokenizers
• huggingface-hub
• pyyaml
• filelock
• regex
• tqdm
• safetensors