GCA Explained: A Task Contract That Fixes Spatial Reasoning in VLMs
The Geometrically-Constrained Agent splits task formalization from geometric execution, so VLMs reason about space better with no training. How it works.
This article is published in English.
Sources: Paper Walkthrough — Geometrically-Constrained Agent for Spatial Reasoning
Picture a vision-language model asked to imagine sitting on a sofa and say where the coffee table is. In principle that's doable: determine which way the sofa faces, rotate the imagined viewpoint to match, then place the table relative to it. In practice it tends to fall apart, because the classic ReAct loop has nothing built in to settle the ambiguity before acting on it. The cause isn't missing knowledge. It's a translation failure between loose language and precise geometry.
The CVPR 2026 paper Geometrically-Constrained Agent for Spatial Reasoning (GCA) proposes a fix: before any calculation, the VLM must produce a formal task constraint and stick to it. That splits the job into a semantic formalization stage followed by a constrained execution stage. No training is involved either. The VLM and its tools receive no fine-tuning, no extra dataset and no gradient updates. Fine-tuning would try to teach the model grounded geometry through weight changes. GCA reshapes the prompting instead, so a reference frame and an objective are committed to ahead of any tool call. Any off-the-shelf VLM can then act as backbone, with the discipline supplied purely by the two-stage layout.
The sections below cover the constraint itself, the constrained computation that follows, and the results.
Two roles joined by a contract
The core idea separates the VLM's reasoning into two jobs linked by an explicit, machine-readable agreement:
C_task ← F_formalize(q, v)
r_t = F_compute(C_task, T, r_{t-1})
Here q is the ambiguous natural-language question, v is the visual input, T is the toolset and r is the running state of the computation.
Stage 1, task formalization. Here the VLM acts as a semantic analyst. It's handed the vague question plus the image, and instead of answering it, it pins down what the question means. With the sofa, the model has to state, before touching a tool, what "relative to you" amounts to in geometric terms (the sofa's heading and not the camera's) and what quantity is wanted (the table's position in that frame). The output is a specification and not a number: C_task = (C_R, C_O), one constraint for the reference frame and one for the objective, recorded before any computation. After that commitment, the meaning can't be silently swapped later, which is what keeps the agent from falling back on the camera's view.
Stage 2, constrained geometric computation. The VLM now turns into a task solver that creates and runs tool calls, always inside the deterministic limits C_task established. In the sofa example, C_R already binds the frame to the sofa's heading, and C_O already names the table's position within it, so both are settled. What's left resembles a mechanical routine: find the sofa, segment it, estimate its orientation, do the same for the table, then apply the coordinate transform linking them. If two tables turn up, the VLM must still pick one, but it's deciding which object fits the objective and not what the objective is. That keeps a noisy tool result from pulling the agent away from the problem already defined.
The reference frame, C_R
C_R is the coordinate system that every measurement refers to. It consists of an origin and three mutually perpendicular axes laid out the OpenCV way: z points forward, y points down, and x completes a right-handed set. GCA anchors it to one of three geometric primitives found in the scene:
- Object-based frame. Tied to an object's own orientation. In a phrase like "when washing hands", the +z axis of the frame points opposite the sink's, because a person faces the sink to use it.
- Camera-based frame. Tied to a particular camera's viewpoint. "From the viewpoint of Figure 1" sets the frame's +z to that camera's +z.
- Direction-based frame. Tied to a vector between two places. "Owen is north of sink" sets +z to the normalized difference between the two centroids.
The objective, C_O
Once the frame is set, C_O states what to measure inside it. Consider a query asking whether a chair sits west of a toaster. The toaster anchors C_R, while the chair-to-toaster relationship becomes C_O. Splitting the query turns one tangled, ambiguous question into two small ones: which frame applies, and what gets measured in it.
Because the VLM interprets language qualitatively here and computes nothing numerically, this stage matches what it does best. The paper reports roughly 70% accuracy from formalization alone, which suggests the weak point lies in what comes after understanding the query.
Running inside the contract
With C_task fixed, the VLM operates in a ReAct-style loop in which every step is checked against an unchangeable agreement and nothing is free-form. Three mechanisms stand out.
Data acquisition. C_task lists the geometric ingredients required. An object-based frame anchored to a sink, for example, needs that sink's orientation. The agent issues tool calls for exactly those variables and nothing improvised.
Ambiguity resolution. Real scenes are messy. Suppose C_O means "the leftmost chair" and the detector reports five chair boxes. The VLM studies the visualized detections and selects the correct index. This closed-loop check stops noisy perception from quietly corrupting the final result, while the choice still has to match what C_task first asked for.
Knowledge-augmented code generation. Recalling intricate 3D transformation formulas from memory is a classic hallucination trap. GCA instead keeps a fixed library of pre-verified geometric formulas, such as local-to-world transforms. When all variables are bound, the system picks the relevant formulas based on the data types in play and injects them into the code generator's context. It's a static retrieval-augmented setup for math and not for text. The VLM then writes and runs Python on that scaffolding and doesn't produce an opaque numeric guess.
The toolbox has two families. Geometry and perception tools (3D reconstruction with VGGT, open-vocabulary detection, segmentation) turn the visual scene into parameters. Computation and utility tools (a Python sandbox, coordinate projection helpers) convert those parameters into a deterministic answer.
Results
Using Qwen3-VL-Thinking as its backbone, GCA was evaluated on five spatial reasoning benchmarks (MMSI-Bench, MindCube-tiny, OmniSpatial, SPBench and CV-Bench) and reached a new state of the art. It tops the best foundation-model baseline, Gemini-2.5-Pro, by roughly 12%, the training-based specialist SpatialLadder by roughly 27% and the tool-integrated agent TIGeR by roughly 38%.
The margin grows on the hardest multi-step tests. MMSI-Bench asks four-choice questions, and most rival VLMs hover just above the 25% chance level. GCA scores 47.6%, a 28% relative gain on Gemini-2.5-Pro. The paper's authors point to C_task as the reason, because it blocks the model from adopting a faulty semantic shortcut, such as reading "your perspective" as the camera's, before any real computation.
Backbone choice matters little. GLM-4.5V, GPT-4o and Gemini-2.5-Pro each improve on their own chain-of-thought baselines, with MMSI-Bench gains from +19% for GPT-4o up to +49% for Gemini-2.5-Pro, loosely following each model's existing agentic and spatial ability.
The wider lesson
The point reaches beyond spatial reasoning. A model can fail because it solved an ill-posed problem correctly, as opposed to solving a well-posed one incorrectly, and in that case more training data or fancier tools won't help. What helps is making the model externalize its own problem definition as something verifiable before it computes anything. Because GCA is training-free, the fix is architectural and not statistical. That it transfers cleanly across four quite different foundation VLMs hints that the semantic-to-geometric gap is structural, a consequence of how these models are asked to reason, and not a weakness of any single model's weights.