Ollama Vision
A RocketRide image node that sends images to locally-hosted Ollama vision models and returns text analysis.
What it does
Connects to open-source multimodal models (Llama 3.2 Vision, LLaVA, Moondream, MiniCPM-V, Qwen 2.5 VL, or any custom model) served by a local Ollama instance. No API key is required: models run entirely on your own hardware, making the node suitable for privacy-sensitive workloads. It accepts either a single image or a stream of image documents (e.g. from a frame grabber); metadata such as frame number and timestamp is preserved on the documents output.
Uses langchain-openai (ChatOpenAI) against Ollama's OpenAI-compatible /v1 endpoint. The configured server base URL is normalized to end with /v1 automatically, and a placeholder API key ("ollama") is sent because Ollama ignores it. Requests run with temperature: 0.
Each inference attempt is capped by a 30-second hard timeout; a timed-out or retryable failure is retried once with exponential backoff (a fresh HTTP client is created per attempt so a hung request cannot exhaust the connection pool). API errors are translated into actionable user-facing messages (see Troubleshooting).
Lanes
| Lane in | Lane out | Description |
|---|---|---|
image | text | Analyze a single image, receive text |
documents | documents | Analyze image documents, return text analysis with original metadata preserved |
Profiles
Default: Llama 3.2 Vision 11B (llama3_2-vision-11b).
| Profile | Model | Context tokens |
|---|---|---|
custom | (user-specified) | 16,385 |
llama3_2-vision-11b (default) | llama3.2-vision:11b | 128,000 |
llama3_2-vision-90b | llama3.2-vision:90b | 128,000 |
llava-7b | llava:7b | 32,768 |
llava-13b | llava:13b | 4,096 |
llava-34b | llava:34b | 4,096 |
moondream | moondream | 2,048 |
minicpm-v | minicpm-v | 8,192 |
qwen2_5vl-3b | qwen2.5vl:3b | 128,000 |
qwen2_5vl-7b | qwen2.5vl:7b | 128,000 |
Configuration
Choose a bundled profile for a known Ollama model or custom for another multimodal
model. All profiles expose the server URL and prompts; custom also exposes the model
name and context limit. The default server is http://localhost:11434/v1.
Image to text
Raw image bytes arrive in chunks over the AVI protocol, are accumulated, encoded as a
base64 data URL with the incoming MIME type, and sent to the model with the configured
analysis prompt. The model's answer is written to the text lane.
Documents to documents
Each incoming Doc of type Image is analyzed individually. Its page_content is
expected to be base64-encoded PNG. The answer is emitted as a Text document that
preserves the original metadata (chunkId, time_stamp, and other fields). Non-image
documents and images with empty content are skipped with a warning; a per-document
failure is logged and skipped instead of failing the batch. The original image
documents do not flow downstream.
System Instructions and Analysis Prompt
Use System Instructions to define the model's role and Analysis Prompt for the
image task. If the analysis prompt is empty, the node uses question text from the
request; if that is also empty, it uses Describe this image.
The selected model must already be pulled into Ollama (ollama pull <model>).
Notes
Troubleshooting
The node maps common API failures to clear messages:
| Symptom | Meaning / fix |
|---|---|
| "Cannot connect to Ollama server" | Ollama is not running, or serverbase points to the wrong host/port |
| "Model '...' is not loaded in Ollama" | Pull the model first: ollama pull <model> |
| "Too many requests to Ollama" | Rate limited: wait a moment and retry |
| "Ollama returned a server error" | Check the Ollama server logs |
| "Vision request timed out" | Inference exceeded the 30 s hard timeout; large models may need a warm-up run or a smaller model |
| "Image processing error" | Use a supported image format: JPEG, PNG, GIF, WEBP |
Upstream docs
Schema
| Field | Type | Description | Default |
|---|---|---|---|
image_vision_ollama.profile | string | Vision Model Select the Ollama vision model to use | "llama3_2-vision-11b" |
model | string | Model Ollama vision model name | |
modelTotalTokens | number | Tokens Total Tokens | |
vision.prompt | string | Analysis Prompt Describe what you want to analyze or extract from the image | |
vision.systemPrompt | string | System Instructions Define the model's role and behavior for image analysis |
Dependencies
langchain-openailangchain-corelangchain