# preprocessor_langchain A RocketRide preprocessor node ("General Text") that splits incoming text into chunks for downstream embedding or LLM processing. ## What it does Splits text into chunks using **LangChain text splitters** (`langchain_text_splitters`). Choose a profile tuned for the content type: general prose, markdown, LaTeX, sentence-based NLP, or any custom splitter class the library exports. No LLM is required. The splitter class is loaded dynamically by name from `langchain_text_splitters`, so the `custom` profile can target any class that library exports. Constructor kwargs are filtered against the target class signature, preventing "unexpected keyword argument" errors across splitters. Chunk overlap is fixed at `0`. Incoming text is accumulated per file and split once when the file closes. Each incoming table is split immediately as its own unit. Every chunk is emitted as a document with a sequential `chunkId` (reset per file); tables additionally carry a `tableId`. By default chunks are measured by **string length** with a maximum of **512** characters. Token mode uses a conservative byte-length estimator instead of a real tokenizer (no transformers model is loaded). See [Token mode](#token-mode) below. --- ## Configuration ### Lanes | Lane in | Lane out | Description | |---------|-------------|------------------------------------------| | `text` | `documents` | Split plain text into document chunks | | `table` | `documents` | Split table content into document chunks | ### Fields | Field | Type | Description | |---|---|---| | `strlen` | number | Default 512. | | `tokens` | number | Default 512. | | `mode` | string | Default "strlen". | | `splitter` | string | Default "RecursiveCharacterTextSplitter". | | `separators` | string | Default "'\n\n', '\n', ' ', ''". | | `separator` | string | Default ""\n"". | | `model` | string | Default "en_core_web_sm". | | `profile` | string | Default "default". | ### Separator syntax `separators` and `separator` are parsed as comma-separated Python string literals (e.g. `'\n\n', '\n', ' ', ''`). Escape sequences such as `\n` are interpreted. Every element must be a string. The `character` profile accepts exactly one element. An invalid format raises an error at startup. ### Advanced token-mode options These keys are read from the node config but are not exposed in the UI shape: | Field | Type / Default | Description | |-----------------------|------------------|----------------------------------------------------------------------------------| | `bytes_per_token` | float, `3.0` | Bytes-per-token ratio used by the estimator. Lower values estimate more tokens (safer). | | `max_model_tokens` | int, unset | Hard cap for the model's max token context. When set, caps the chunk size and enables the post-split safety net. | | `token_safety_margin` | int, `32` | Subtracted from `max_model_tokens` to leave headroom for special tokens. | --- ## Profiles | Profile | Splitter | Best for | |---------------------|----------------------------------|------------------------------------------------------------------------| | `default` (default) | `RecursiveCharacterTextSplitter` | General-purpose prose | | `recursive` | `RecursiveCharacterTextSplitter` | General-purpose prose with custom separators | | `character` | `CharacterTextSplitter` | Simple splitting on a fixed separator | | `markdown` | `MarkdownTextSplitter` | Structured Markdown documents (separators kept in chunks) | | `latex` | `LatexTextSplitter` | Scientific and academic documents (separators kept in chunks) | | `nltk` | `NLTKTextSplitter` | Sentence-based splitting | | `spacy` | `SpacyTextSplitter` | NLP-based sentence splitting (English, German, French, Spanish models) | | `custom` | `RecursiveCharacterTextSplitter` | User-defined splitter class from `langchain_text_splitters` | ### NLTK Dependencies (`nltk`) are installed lazily the first time this profile is used. The `punkt` tokenizer data (and `punkt_tab`, required by NLTK 3.9+) is downloaded automatically if missing. Pass a `language` key in the node config (e.g. `"english"`, `"spanish"`) to forward it to the splitter. ### spaCy Dependencies (`spacy`) are installed lazily the first time this profile is used. The configured pipeline model (default `en_core_web_sm`) is downloaded automatically if not already installed. Small, medium, and large models are available for English, German, French, and Spanish; an English transformer model (`en_core_web_trf`) is also supported (best accuracy, slower). ### Custom Set `splitter` to the class name of any splitter exported by `langchain_text_splitters`. An unknown class name raises `Splitter '' not found in LangChain` at startup. Only kwargs accepted by the chosen class's constructor are forwarded; unrecognized kwargs are silently dropped. --- ## Token mode With `mode: tokens`, chunk size is measured by an estimated token count. No tokenizer or transformers model is loaded. The estimate is the UTF-8 byte length of the text divided by `bytes_per_token` (default `3.0`), rounded up. This is conservative by design, so real token counts should come in at or under the estimate. When `max_model_tokens` is set: - The effective token budget is `max_model_tokens - token_safety_margin`. - The requested chunk size (`tokens`) is capped to that budget. - After splitting, any chunk that still exceeds the budget is force-subdivided by proportional character cuts until every piece fits. This guarantees no emitted chunk exceeds the model's context budget even without an exact tokenizer. --- ## Schema | Field | Type | Description | Default | |---|---|---|---| | `langchain.splitter.character.separator` | `string` | **Split separator** | `"\"\\n\""` | | `langchain.splitter.character.splitter` | `string` | | const: `"CharacterTextSplitter"` | | `langchain.splitter.custom.splitter` | `string` | **Splitter class name** | const: `"RecursiveCharacterTextSplitter"` | | `langchain.splitter.default.splitter` | `string` | | const: `"RecursiveCharacterTextSplitter"` | | `langchain.splitter.latex.splitter` | `string` | | const: `"LatexTextSplitter"` | | `langchain.splitter.markdown.splitter` | `string` | | const: `"MarkdownTextSplitter"` | | `langchain.splitter.mode` | `string` | **Split by** | `"strlen"` | | `langchain.splitter.nltk.splitter` | `string` | | const: `"NLTKTextSplitter"` | | `langchain.splitter.profile` | `string` | **Text splitter** | `"default"` | | `langchain.splitter.recursive.separators` | `string` | **Split separators** | `"'\\n\\n', '\\n', ' ', ''"` | | `langchain.splitter.recursive.splitter` | `string` | | const: `"RecursiveCharacterTextSplitter"` | | `langchain.splitter.spacy.model` | `string` | **Model** | `"en_core_web_sm"` | | `langchain.splitter.spacy.splitter` | `string` | | const: `"SpacyTextSplitter"` | | `langchain.splitter.strlen` | `number` | **String length** | `512` | | `langchain.splitter.tokens` | `number` | **Number of tokens** | `512` | ## Dependencies - `langchain` - `langchain-text-splitters` - `langchain-core` - `accelerate` - `transformers` - `tokenizers` - `huggingface-hub` - `pyyaml` - `filelock` - `regex` - `tqdm` - `safetensors` ## Source [ View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/preprocessor_langchain)