Skip to main content

Running Pipelines

View as Markdown

Running Pipelines

Start a pipeline, watch its progress, and stop it. Method tables live in the API reference; this page covers the workflow.

Start with use()​

use() starts a pipeline from a file (Node only) or an in-memory config and resolves to an object whose token identifies the running task — every data and control call takes it.

const { token } = await client.use({ filepath: './pipeline.pipe', ttl: 3600 });

Beyond filepath/pipeline, the options accept source, threads, useExisting, args, ttl, pipelineTraceLevel (trace verbosity for the run log), name (a display name for the task), and env (per-run variable overrides). Pass the pipeline config as-is — do not wrap it in { pipeline: ... }; the client sends it to the server, which resolves ${ROCKETRIDE_*} variables.

Check reused before trusting the result. useExisting returns the instance that is already running under that token rather than starting the one you submitted, and the result's reused flag is true when that happened. A reused instance keeps the configuration it was created with — the pipeline in this call is ignored, edits included — along with whatever state it has accumulated. Benchmarks and A/B comparisons are where an unnoticed reuse costs the most. Call restart() to apply new configuration to a live token.

Why a token: the server runs each pipeline as a separate task. The token targets send(), sendFiles(), pipe(), chat(), getTaskStatus(), and terminate() at the correct pipeline.

Watch progress​

Poll getTaskStatus(token) — it returns completedCount, totalCount, completed, state, exitCode, and more. A per-call { timeout } option defaults to 15000 ms; pass false to disable the bound:

while (true) {
	const status = await client.getTaskStatus(token);
	console.log(`Progress: ${status.completedCount}/${status.totalCount}`);
	if (status.completed) break;
	await new Promise((r) => setTimeout(r, 2000));
}

Events​

For push-style progress instead of polling, add a monitor subscription; events arrive at your onEvent callback:

await client.addMonitor({ token }, ['apaevt_status_upload', 'apaevt_status_processing']);
// ... later:
await client.removeMonitor({ token }, ['apaevt_status_upload', 'apaevt_status_processing']);

addMonitor(key, types) / removeMonitor(key, types) are reference-counted — adding the same key merges types, removing unsubscribes a type only when its count reaches zero. The MonitorKey is { token } for a running task, or { projectId, source, pipeId?, teamId? } (a team ID addresses that team's deployed run). The older setEvents(token, eventTypes, pipeId?) still works but is deprecated in favor of the monitor pair.

Validate before you run​

validate({ pipeline, source? }) checks a pipeline config server-side without starting it and returns errors and warnings — cheap insurance before use().

Stop with terminate()​

terminate(token) stops the pipeline and frees server resources. Long-lived tasks without a ttl run until terminated.

Discover services​

getServices() returns lightweight summaries of every service the server supports (plus a deduplicated icon table and the server version). For a full definition — config schema included — fetch one by name with getService(name), which returns a ServiceDefinition and throws on failure (it never resolves to undefined).

const services = await client.getServices();
const ocr = await client.getService('ocr'); // throws if unknown

Liveness​

ping(token?) performs a liveness check and throws on failure; the optional token scopes the ping to a task.

Deploying a pipeline so it persists server-side and runs on a schedule is a separate surface — see Deployments.

Last updated on