LangChain plus Ollama: Local Qwen Chains, Tools, and Structure

Local Qwen via Ollama, wired into LangChain

Run open-weight chat models such as qwen3.8:27b on your own machine and plug them into LangChain chains, agents, and tools — no cloud API keys and no per-token invoice.

Ollama in one paragraph

Ollama packages weights, config, and runtime behind a Modelfile-friendly UX and handles GPU acceleration details for common setups. Pull a tag, run a prompt, expose an HTTP API on port 11434.

LangChain’s Ollama integration

The langchain-ollama package exposes chat models with tool calling, structured output, multimodal inputs, streaming, async, and logprobs — the same surface you expect from cloud chat wrappers, pointed at localhost.

Setup sketch

Install Ollama, pull the chat model you need, and confirm the API answers. In Python, install LangChain’s Ollama integration and construct a chat model pointed at your local base URL.

ollama pull qwen3.8:27b

ollama list

Chains and prompts

Build a simple prompt → model chain for summarization or Q&A without leaving the machine:

from langchain_ollama import ChatOllama

llm = ChatOllama(
    model="qwen3.8:27b",
    temperature=0.3,
)

Stream tokens if your UI needs incremental updates:

messages = [
    ChatMessage(role="control", content="thinking"),
    HumanMessage("What is 3^3?"),
]

Tools and agents

Bind tools the same way you would with a hosted model. The local model must support tool calling for reliable agent loops — pick tags known to work with Ollama’s tool protocol.

messages = [
    SystemMessage(
        content="""
You are an expert at reading screenshots and technical documentation.

Rules:
- Extract information only from the image.
- Do not hallucinate missing values.
- Return concise answers.
- Preserve commands exactly as shown.
"""
    ),
    HumanMessage(
        content=[
            {
                "type": "image_url",
                "image_url": f"data:image/png;base64,{image_b64}",
            },
            {
                "type": "text",
                "text": "What is the download command for the Gemma 7B model?",
            },
        ],
    ),
]

SystemMessage
    ↓
"You are a technical documentation analyst.
 Extract information only from the image.
 Do not guess.
 Return concise answers."

HumanMessage #1
    ↓
"What is the download command?"

HumanMessage #2
    ↓
"What version of the model is shown?"

HumanMessage #3
    ↓
"What is the model's parameter count?"

Structured output

When you need JSON, use structured output helpers rather than brittle “please return JSON” prompting alone:

HumanMessage(
    content=[
        {
            "type": "image_url",
            "image_url": f"data:image/png;base64,{image_b64}",
        },
        {
            "type": "text",
            "text": """
The image is a screenshot from an Ollama model page.

Task:
1. Find the section containing model download instructions.
2. Locate the Gemma 7B model.
3. Return only the exact Ollama pull command.

Output format:
Return JSON only:

{
    "model_name": "",
    "download_command": "",
    "confidence": ""
}

Do not include explanations.
"""
        },
    ]
)

Why this stack

You keep data on-box, iterate without rate limits, and still speak LangChain’s chain/agent vocabulary. Trade peak quality or speed for privacy and cost predictability — a good deal for prototypes, internal tools, and offline demos.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.

Prefer explicit model tags in every example so upgrades do not silently change behavior mid-tutorial.