Context Optimizer

View as Markdown

A RocketRide text node that fits an LLM request into a model's context window by budgeting tokens across the system prompt, question, retrieved documents and conversation history. Put it in front of an LLM node when retrieval or a long conversation can push a prompt past the window and you want deterministic, local trimming instead of a provider-side rejection.

What it does

The node counts the tokens of the prompt the engine will actually render with tiktoken, and only when that rendering exceeds the model's context window does it allocate a per-component budget and trim. The system prompt and the question are truncated on sentence boundaries, falling back to a token-level cut when a single sentence is still over budget; documents are ranked by their vector-DB score (descending, or by keyword overlap with the question when no scores are present) and selected greedily until the document budget is spent; conversation history keeps the first message and as many recent messages as fit, replacing the middle with a single summarized placeholder. A question carrying several QuestionText entries keeps every entry — the entries share the query allowance in proportion to their own size — so the per-entry embedding metadata that the embedding nodes write and the document stores read survives the node. Requests whose rendered prompt already fits pass through unchanged, and no extra model call is ever made. This node is marked experimental.

Lanes

Lane in	Lane out	Description
questions	questions	Questions with optimized context fitting within model token limits

Profiles

Default: Auto - Use default model limits (auto). Both user-facing profiles ship the same budget split; they differ only in where the context window comes from.

Profile	Budget split (system / query / documents / history)	When to pick it
auto (default)	10 / 15 / 50 / 25	The usual choice: name the model and let the node resolve its window.
manual	10 / 15 / 50 / 25	A deployment whose real window neither the live catalog nor the fallback table knows; set Max context tokens yourself (ships at 128 000).
ci_tiny	10 / 20 / 35 / 35	Automated node tests only. It pins a 112-token window so every case is forced over budget; it is deliberately absent from the Mode dropdown.

Configuration

Pick a Mode and, in auto, name the model. The four budget percentages have workable defaults (10 / 15 / 50 / 25) and most pipelines never touch them — reach for them only when one component of your prompt is systematically starved.

Mode

auto resolves the context window for the model named in Model name; manual ignores the model entirely and uses Max context tokens. auto is right whenever the model is one an llm_* node in this deployment publishes, which is the common case; manual is the escape hatch for a gateway, a fine-tune, or a self-hosted deployment whose window this build cannot know. How auto resolves the window — and what it does when it cannot — is described under Where the context-window size comes from.

Model name

The model identifier used for the window lookup in auto mode, e.g. gpt-5.4, claude-sonnet-4-6, models/gemini-3.1-pro-preview. Use the id an llm_* node publishes; a provider-scoped id (openai/gpt-5) also resolves under its bare name when that name is unambiguous, and the abbreviated family aliases claude-sonnet, claude-opus, gemini-pro and gemini-flash resolve from the built-in table. An id nothing recognizes does not fail the run — it is budgeted against the smallest window this build knows about, with a warning — so a typo here costs context rather than correctness. Set the id to match the LLM node the pipeline actually calls; budgeting for a larger window than the model has is what gets a request rejected downstream.

Max context tokens

The explicit window, in tokens, used by manual mode; 0 falls back to the model default. Set it to the model's real total window, not the share you want the prompt to use — the percentages below divide this number. In auto mode the field is hidden and ignored.

System prompt, query, document and history budget percentages

These four fields divide whatever window is left after the un-budgeted sections are counted (see What is budgeted, and what is only counted). They are percentages of that remainder, and the node never lets them over-commit it: a set summing to more than 100 is scaled proportionally back to 100, and a set summing to less than 90 is honoured but logs a warning that the window is being under-used. The system prompt, query and document allowances are taken from their percentages in that order and history absorbs whatever is left, so history is the component that gives way first. Raise Document budget % for retrieval-heavy pipelines where history is short; raise History budget % for long chat sessions with few documents. A component given 0 is emptied whenever the request is over budget, which is a legitimate way to say "drop history before you touch my documents".

Notes

What is budgeted, and what is only counted

Exactly four fields of the incoming Question are budgeted and trimmed: role (the system prompt), questions, documents and history. Everything else Question.getPrompt() renders is counted against the window but never trimmed:

• instructions, including the JSON-format boilerplate expectJson injects (the prompt node appends here);
• examples;
• context (retrieval nodes fill this);
• goals;
• the section markup itself — the ### System Instructions: / ### Documents: / ### Current Task: headers, the Document N) Content: and role: line prefixes, and the CRLF joins between them.

That second group is measured, not estimated: the incoming question is rendered through getPrompt() with the four budgeted fields emptied, the result is counted with the node's own tokenizer, and the total is subtracted from the context window before the per-component percentages are applied. A request is therefore judged against the prompt that will actually be sent — a question whose role, questions, documents and history fit but whose instructions and context push the rendering over the window is trimmed instead of being waved through. The figure errs high: it keeps the markup of every document, history message and question entry even though trimming may later drop some of them, and it adds a two-token margin for the line break getPrompt() appends after a non-empty role, which the blanked rendering cannot see.

If the un-budgeted fields alone fill the window, nothing is left to allocate: every budget becomes 0, all four budgeted fields are emptied, and the node warns once naming the cost. This node cannot shorten instructions, examples, context or goals — shorten them upstream, or raise Max context tokens.

Where the context-window size comes from

auto mode resolves the window for the configured model in this order:

1. The live model catalog — the modelTotalTokens value on the matching preconfig profile of any LLM node, read through the engine's own service registry (rocketlib.getServiceDefinitions() to enumerate the registered types, then getServiceDefinition() for each raw definition; whichever definitions publish modelTotalTokens contribute — the chat llm_* nodes, the image_vision_* nodes, and the embedding/search nodes alike). Because the registry is the same path the engine used to load the nodes, the catalog is found whether this node runs from the source tree, from an installed node store, or from a --node_path materialisation with no sibling directories; it is re-read at every pipeline start, so a reloaded registry is never masked by a stale cache. Those values are kept current by the sync-models tool, so gpt-5.4, claude-sonnet-4-6, models/gemini-3.1-pro-preview and every other id the LLM nodes publish resolve without this node hard-coding them. Provider-scoped ids also resolve under their bare name (openai/gpt-5 → gpt-5), but only when that name is unambiguous: an unscoped profile always wins, and a bare name that two gateways disagree about is dropped so it falls through to the table below rather than silently picking one. Where the same full id appears in two services files with different windows, the smaller is used.
2. A built-in fallback table (ContextOptimizer.MODEL_LIMITS), refreshed against that catalog on 2026-09-04. It covers deployments where no LLM node is registered with the engine, and it is where the abbreviated family aliases that no provider actually publishes — claude-sonnet, claude-opus, gemini-pro, gemini-flash — are defined.
3. The conservative fallback — the smallest context window published by any registered chat-LLM profile (llm_* nodes / classType: ["llm"]) or by the fallback table — 8 191 tokens on develop today, the smallest published chat windows (gpt-4, codestral-latest, mistral-medium-2505) — with a one-time warning naming the unresolved model id. Embedding input limits and vision models are catalogued for lookup but never lower this floor. An unknown id is usually a typo or a model this build predates; budgeting it low costs some unused context, whereas the previous optimistic 128 000 default made the provider reject the request outright for any smaller-window model. Set Max context tokens for the real window.

manual mode sets Max context tokens explicitly and bypasses all three.

The engine also exposes the connected LLM's own limit and tokenizer to upstream nodes through IInvokeLLM.GetContextLength() / GetTokenCounter() (as used by summarization and preprocessor_llm). That channel is only reachable after beginFilter, so this node does not consume it today; manual mode is the escape hatch when both the catalog and the fallback table are wrong for a given deployment.

Offline use

Token counting uses tiktoken, which downloads its BPE vocabulary from openaipublic.blob.core.windows.net the first time a given encoding is requested (o200k_base for the GPT-5 / GPT-4o families, cl100k_base otherwise) and then caches it on disk. In an air-gapped or egress-restricted deployment, pre-seed that cache and point TIKTOKEN_CACHE_DIR at it before starting the engine. The node's own services.json test cases exercise the same path, so a CI runner without egress needs the same pre-seeded cache.

Schema

Field	Type	Description	Default
context_optimizer.document_budget_pct	number	Document budget (%) Percentage of context window for retrieved documents	50
context_optimizer.history_budget_pct	number	History budget (%) Percentage of context window for conversation history	25
context_optimizer.max_context_tokens	number	Max context tokens Override model token limit (0 = use model default)	0
context_optimizer.model_name	string	Model name Model identifier for context-window lookup. Use an id published by an llm_* node (e.g. gpt-5.4, claude-sonnet-4-6, models/gemini-3.1-pro-preview); a short family alias such as claude-sonnet also resolves.
context_optimizer.profile	string	Mode Context optimization mode	"auto"
context_optimizer.query_budget_pct	number	Query budget (%) Percentage of context window for the query/question	15
context_optimizer.system_prompt_budget_pct	number	System prompt budget (%) Percentage of context window for system prompt	10

Dependencies

• tiktoken >=0.7.0,<1.0.0