Code

View as Markdown

A RocketRide preprocessor node that splits supported source code into syntax-aware documents. Choose it over a general-text or LLM preprocessor when you want code structures, rather than prose boundaries, to drive chunking.

About Tree-sitter

Tree-sitter is the parsing library this node uses to build source-code syntax trees. The node installs its language-specific grammar packages and creates a parser for the selected language.

What it does

Receives text and emits documents made from syntax-tree nodes. It supports the C, C++, Python, JavaScript, and TypeScript profiles; in automatic mode it detects one of those languages from the text itself. A matched class or function can coexist with chunks for nested constructs, so output may overlap.

Lanes

Lane in	Lane out	Description
text	documents	Split source-code text into syntax-aware documents.

Profiles

Default: auto, which detects the language from the source text.

Profile	Language
auto (default)	Detect from source text
c	C
cpp	C++
python	Python
javascript	JavaScript
typescript	TypeScript

Configuration

Select the language profile that matches the input when its language is known; otherwise leave the default automatic profile. The maximum string length is present in the configuration, but the current implementation determines output boundaries from syntax nodes.

Code splitter profile

The automatic profile samples up to the first 5,000,000 characters and scores regex signals for the five supported languages. It needs a score of at least 3 and a lead of at least 2 over the runner-up; otherwise the input produces no documents and a warning is emitted. Use an explicit profile for short snippets, unusual code, or content that automatic detection cannot classify reliably.

For C and C++, automatic detection favors C for header-like ties and for extern "C" without C++-only markers. Pick cpp explicitly when that conservative choice is wrong for your input.

Maximum string length

This value defaults to 512, but the splitter does not use it to divide a syntax node. A large class or function is therefore emitted as one chunk even when it exceeds that value. Change it only if a downstream component reads the same configuration; it does not constrain this node's output today.

Notes

Output details

Both text and table callbacks are parsed. Table-derived documents are marked as tables; each object starts its chunkId at 0. C and C++ header declarations and whole extern "C" linkage blocks are included, while preprocessor lines such as #include and #define are skipped.

The parser cache is a class attribute on PreProcessor, so a grammar loaded once is reused by every instance in the process and outlives any single global node instance. If the optional tree_sitter_languages package is available it is used to load a grammar; otherwise the node loads the installed per-language grammar module.

Upstream docs

• Tree-sitter documentation

---

Schema

Field	Type	Description	Default
code.splitter.auto.language	string		const: "auto"
code.splitter.c.language	string		const: "c"
code.splitter.cpp.language	string		const: "cpp"
code.splitter.javascript.language	string		const: "javascript"
code.splitter.profile	string	Code splitter profile	"auto"
code.splitter.python.language	string		const: "python"
code.splitter.strlen	number	Maximum string length	512
code.splitter.typescript.language	string		const: "typescript"

Dependencies

• tree-sitter
• tree-sitter-c
• tree-sitter-cpp
• tree-sitter-javascript
• tree-sitter-python
• tree-sitter-typescript