Performance

View as Markdown

Performance

Practical levers for making pipelines faster and cheaper. For why these levers exist — the engine's threading, streaming, and batching behaviour — see the Execution Model.

Preprocessor chunk size

Chunk size directly affects embedding throughput and retrieval quality:

Smaller chunks	Larger chunks
More precise retrieval	Fewer embedding API calls
More vectors to store and query	Lower storage cost
More embedding requests (cost)	Less precise retrieval for long queries

A chunk size of 512–1024 tokens is a reasonable starting point for most text content. Reduce chunk size if retrieval recall is poor on short queries; increase if you're hitting embedding API rate limits.

LLM context and cost

LLM nodes send the full accumulated context (system prompt, retrieved chunks, conversation history) on every call. Costs scale with context size:

• Retrieved chunks: more chunks retrieved from the vector store = more tokens per LLM call. The retrieval limit is set by the question's docFilter limit, not a store node config field.
• Memory nodes: conversation history grows each turn. Use memory_persistent with max_history to cap history length (memory_internal has no window-limit config).
• Model selection: larger models (GPT-5, Claude Opus) cost more per token. Use them for reasoning-heavy tasks; use smaller models for classification and extraction where a cheaper model performs just as well.

Vector store batch sizing

Vector stores flush chunks in batches (see Execution Model for the mechanics). Batch size affects throughput: larger batches reduce round-trip overhead but increase memory usage per run. The defaults suit most workloads.

Profiling a pipeline

The engine emits per-node timing in its WebSocket event stream. The CLI does not stream it; subscribe from an SDK client or the platform's monitor apps. The WebSocket Events page documents the event schema. To find bottlenecks, look for the node with the longest gap between its start and complete events — that is usually the LLM call or the embedding step.

Related

• Execution Model: threading, streaming, and batching.
• Production: deployment topology for throughput.
• WebSocket Events: timing events.