A Model Router on GCP: One Governed API in Front of Multiple Open LLMs

Most enterprise LLM projects begin with one question: which model should the application call? Once an organization runs several models, that question turns into a problem. Models differ in strengths, latency, resource needs and uptime. Hard-coding a model into each consuming app pushes those decisions into the wrong layer of the system.

A cleaner rule is this: applications ask for an AI capability, and the AI infrastructure decides which model carries it out. In a design built for Workato's Open LLM integration on GCP, a model router sits in front of four open-weight models and exposes them through a single governed API. The router ended up being a more interesting piece than the model deployment itself.

From integrating models to abstracting them

The conventional setup is a direct line:

Application
    │
    ▼
Specific LLM

Add more models and it fragments into per-application wiring:

Application A ──► Model A
Application B ──► Model B
Application C ──► Model C

Now each application has to know model names, endpoints, authentication, capabilities, availability, fallback models and probably inference parameters. The better shape reverses the relationship:

┌──► Model A
                    │
Enterprise          ├──► Model B
Application ─► Router
                    ├──► Model C
                    │
                    └──► Model D

From the application's side there's a single AI service. Only the router knows a whole portfolio of models stands behind it. Everything else in the design hangs on that dividing line.

The overall architecture

From the outside, the layout stays simple:

Enterprise Consumers
                 │
       Workato / Applications
                 │
                 ▼
          GCP API Gateway
                 │
                 ▼
           Model Router
                 │
       ┌─────────┼─────────┐
       │         │         │
       ▼         ▼         ▼
    Model A   Model B   Model C   Model D
       │         │         │         │
       └─────────┴─────────┴─────────┘
                    │
               GPU Inference

Underneath sit GCP, containerized inference and vLLM. They matter for operations, but they aren't what makes the design interesting. What gives the design its value is the routing step that sits between the public API contract and the engines doing inference.

What the router really does

A router is more than a reverse proxy. A proxy decides where an HTTP request goes. The model router decides which model should run an AI workload. Every request moves through four stages:

Request
   │
   ▼
1. Validation
   │
   ▼
2. Workload classification
   │
   ▼
3. Candidate model selection
   │
   ▼
4. Execution + fallback
   │
   ▼
Normalized response

Because of this, clients don't have to name a physical deployment. Instead of a payload like this:

{
  "model": "physical-model-name",
  "prompt": "..."
}

the public contract can describe a logical capability:

{
  "task": "general_generation",
  "messages": [...]
}

Inside the platform, that capability fans out to whichever model gets picked:

general_generation
        │
        ▼
     Router
   /    |    \
  ▼     ▼     ▼
M1     M2     M3
        │
        ▼
selected model

Model names become an infrastructure detail and stop being an application concern.

Selection is a constraint problem

Round-robin doesn't suit this job, because LLMs aren't interchangeable copies of one service. The first stage is eligibility. From all deployed models, the router derives the subset that can legitimately serve the request, based on factors such as:

• whether the model is up
• whether it supports the requested task
• the context length required
• spare infrastructure capacity
• the character of the workload

Only after that does ranking begin. A general scoring function combines expected task suitability, expected latency, inference cost or resource pressure and current availability, each multiplied by a weight that expresses its importance. The actual coefficients are a matter of implementation policy. What matters architecturally is that model choice turns into explicit policy and stops being an accident of application configuration.

An example with four models

Imagine four models with different jobs:

Model A → general-purpose generation
Model B → stronger reasoning workload
Model C → lightweight / low-latency requests
Model D → alternate or fallback capability

A short classification or transformation request doesn't need the biggest model:

Short deterministic task
        │
        ▼
     Router
        │
        ▼
 Lightweight model

A harder request takes another route:

Complex reasoning task
        │
        ▼
     Router
        │
        ▼
 Higher-capability model

This also prevents a familiar anti-pattern, which is running the most expensive model for everything just because it exists.

Fallback belongs in the router

Selection gives failure handling a natural home. Say the preferred route is:

Request
   │
   ▼
Model B

If that model is unhealthy, overloaded or gone, the service shouldn't necessarily give up right away. The routing table can define a chain of alternatives:

Preferred
   │
   ▼
Model B
   │
 failure
   ▼
Model A
   │
 failure
   ▼
Model D

The router can then tell apart two situations:

MODEL failure

and

AI SERVICE failure

They aren't the same, and treating them differently pays off with open models, where new runtimes or versions get tried out and none of that instability should reach enterprise users.

A stable contract over changing models

The most valuable property is what stays fixed. From Workato's point of view, this endpoint never changes:

POST /v1/chat/completions

Behind it, the picture evolves over time. At the start:

Today

then a progression like this:

Router ──► Model A
LaterRouter ──► Model B
Later stillRouter ──► Models A/B/C/D
           selected dynamically

The business workflow doesn't change at all. That's decoupling in practice. The AI team is free to swap model versions, runtime settings, GPU allocation, routing rules, fallback priorities or even the whole model family without touching a single enterprise workflow.

Why an OpenAI-compatible interface

The inference layer runs vLLM, which exposes an OpenAI-compatible HTTP interface. The benefit isn't a dependency on OpenAI. That API shape has simply become a practical interoperability standard. The flow looks like this:

Client
   │
   │ standard request
   ▼
Router
   │
   │ model-specific decision
   ▼
Inference runtime
   │
   ▼
Open model

Responses can be normalized on the way back as well. The router therefore isolates two kinds of change:

Northbound
Stable enterprise API

──────── Router boundary ────────Southbound
Changing models and runtimes

It's the same job an abstraction layer does elsewhere in distributed systems.

A deliberately boring GCP layer

The supporting infrastructure is conventional on purpose:

Workato
   │
   ▼
API Gateway
   │
   ▼
Containerized Router
   │
   ▼
vLLM inference
   │
   ▼
Open-weight models

Model artifacts can live in Cloud Storage and be mounted into the inference environment, so they aren't baked into application images. API Gateway marks the external boundary. For a proof of concept, API-key authentication is enough to prove the integration path. Production workloads with sensitive data call for more: real identity controls, authenticated services, private networking, quotas, central logs and a proper secrets store.

The aim isn't to invent a new GPU deployment pattern. It's to keep everything beneath the routing abstraction replaceable.

Workato as a consumer, not a runtime

This changes what Workato is responsible for. It should understand business flow:

business event
     ↓
business workflow
     ↓
AI capability request

It shouldn't have to care about any of the following:

GPU
model checkpoint
vLLM process
context limits
model health
fallback order
routing policy

Those belong to the AI platform. The integration shrinks to a plain call:

Workato Recipe
      │
      ▼
HTTP request
      │
      ▼
Governed AI API

Other consumers can use the same pattern:

┌── Workato
                    │
AI Gateway ◄────────┼── Copilot / Agent
                    │
                    ├── Web application
                    │
                    └── Internal service

That means one model portfolio serves many platforms with no new inference integration for each.

Toward an internal AI control plane

A project that started as a way to link enterprise automation to open models points to something larger:

Enterprise Layer
                       │
      ┌────────────────┼────────────────┐
      │                │                │
   Workato          Agents          Applications
      │                │                │
      └────────────────┼────────────────┘
                       │
                       ▼
                 AI API Gateway
                       │
                       ▼
                  Model Router
                       │
          ┌────────────┼────────────┐
          ▼            ▼            ▼
       Model A      Model B      Model C      Model D

The router is the start of an AI control plane. Later policy could add workload classification, latency, model health, capacity, task fit, cost, privacy requirements, evaluation scores and, eventually, tenant-specific rules. Consumers stay shielded from all of it.

The lesson that stands out

The initial question was how to connect an open LLM to Workato. After building it, that turned out to be the wrong level of abstraction. A better one is how enterprise applications can use AI without becoming coupled to individual models. The answer is a combination:

Stable API
    +
Model Router
    +
Explicit Selection Policy
    +
Multiple Open Models
    +
Fallback

Each model becomes replaceable infrastructure, and for enterprise AI architecture that may matter more than which model gets picked.