Building an Agentic API Assistant: From Hackathon to Beta

A team that runs a fleet-management API for a payments company turned a hackathon prototype into a beta assistant sitting inside its developer portal. The idea: stop making integrators read reference pages and translate them into code, and let a conversational agent work out the integration with them. This is a look at how such an assistant is put together, what it refuses to do on purpose, and how its quality is measured.

The assistant explains endpoints, writes code in whatever language the developer uses, plans multi-step workflows and runs read-only data queries against live systems. It takes the place of language-specific static docs and search indexes that only point at pages.

Why static docs stopped being enough

Fleet and mobility software deals with a heavy stream of data: driver activity, fuel stations, card transactions and vehicle telemetry. Anyone integrating has to learn advanced REST endpoints, filter syntax, domain vocabulary and long chains of dependent calls. All that friction made it harder for partners to grow with the platform.

Developers relied on off-the-shelf documentation tools to read endpoint definitions and on test utilities to try calls by hand. The team also kept "recipes", fixed sequences of calls showing how to finish a multi-step job such as creating an account, onboarding a driver and attaching a card. They worked, but three weaknesses kept showing up:

• Tied to a language. Showing each workflow in cURL, Python, .NET, Java and Go demanded steady manual effort from writers and engineers.
• Brittle at the edges. It's impossible to document every permutation of every workflow, so recipes often missed a partner's exact case.
• A constant upkeep bill. The moment an API schema changed, the pages went stale, which stretched onboarding and loaded up the support queue.

From hackathon idea to two modes

The proof of concept grew out of a December 2025 internal hackathon and one question: why ask developers to turn docs into code when a model can reason about their intent directly? What came out of it behaves like an always-available integration consultant, not a search box.

For operational safety, the design splits the assistant into two separate processing paths.

Advisor mode handles single-turn explanations, documentation deep-dives and code generation for any framework, including mutating calls such as POST, PUT and DELETE. It produces code but never runs it, so a person stays in the loop for anything that changes data.

Plan-and-execute mode takes complex, multi-step read requests and carries them out against the live platform through read-only Model Context Protocol (MCP) tools.

What partners actually see

Ask advisor mode for a Python snippet that creates a driver and it returns a requests call with the right endpoint, headers, body schema and a comment on the scopes required. Nothing touches the production API.

Ask plan-and-execute mode for five drivers whose first name is John and a planner reduces the multi-filter call to one task: fetch drivers filtered by first name. An executor selects the matching MCP tool, builds the OData arguments ($filter=first_name eq 'John', $top=5), runs a read-only GET and drops the full response into an artifact store. The model sees only a compact summary such as an artifact id, a count of five and a short preview. A synthesizer then writes a clean list of the five matching drivers.

Architecture: a cycle of cooperating agents

The assistant is built on LangGraph as a cyclic state machine. A planner, executors, validators and a synthesizer take turns planning, acting and checking each step, passing shared state and memory around until the job is done. Four design choices stand out.

1. A read-only proxy for safety

Security shaped the design from the start. The assistant can't run a mutating call inside its own runtime. When a user says "create a new card for driver X", a router sends the request to the advisor node, which returns a structured snippet that the developer can copy, review and run locally. Anything executed for real is limited to GET requests. Write-capable tools exist in the MCP server code, but they're filtered out before the model ever gets to see them.

2. Supervisor-style orchestration

Reasoning and execution sit on different layers, coordinated by a single supervisor that decides which specialist goes next inside one global loop:

• A mode router classifies intent and sends each message to advisor or plan-and-execute.
• A planner breaks a complicated request into a checklist of tasks tied to the target parameters.
• An executor picks the MCP tool for each task, assembles the arguments (OData filters included) and iterates with the tool layer until the task completes or retries run out.
• A synthesizer merges every task's output, real data pulls and fallback code snippets alike, into one answer.

3. A loop that repairs its own OData

Models are prone to inventing query options or mangling syntax when they meet a structured filter language like OData. The team picked OData because it allows flexible partial-match queries over complex mobility objects without hand-coding every combination, but that flexibility invites errors, so a multi-layer repair loop was added.

If a preflight parser flags a syntax problem, or a live call comes back with an API error, the graph catches the event, raises a per-task retry counter and feeds the exact error message back into the model's context. The executor reads its own failure and writes a corrected call. When retries are used up, the task drops back to advisor mode so the user still leaves with something useful.

4. Memory split into artifacts

Pushing a large JSON payload, say thousands of asset records, into the conversation burns through the token budget, hurts reasoning accuracy and raises cost. So the raw responses stay in an isolated backend artifact store, and the model is shown only short summary metadata that points at artifact ids. Follow-up questions drill into specific items by looking up the ids in the application layer, which keeps the model's context small and focused.

Measuring quality

Changing a prompt or a model without regressions needs an automated benchmark. The team's pipeline uses an LLM as a judge and answers one question for every change: is it better or worse than before?

The dataset is a hand-built set of test cases in four categories: code generation, factual knowledge, reasoning and safety. Each case has an explicit grading rubric and not just a question, so the judge rates the substance of an answer instead of matching a brittle expected string. The judge is always a different model from the one being tested and scores on a three-point scale of 10 for correct, 5 for partial and 0 for wrong. Every run adds a row to a history file tagged with the git commit, the tested model, the judge model, the OpenAPI hash, the dataset hash and the prompt version, so any score traces back to the exact code, spec and prompt behind it.

According to the team, the assistant averages about 9.4 out of 10 on this rubric across runs, and that figure serves as a baseline to defend before changes ship. Accuracy isn't the only axis, though. Each run also records latency, to keep the experience responsive, and token usage, which maps directly to model spend. A change that worsens either one gets flagged ahead of release.

Protecting budget and capacity

An always-on assistant that touches a production API needs limits that guard both the model budget and the upstream services. Three layers are in place:

• Quotas per user. Demo accounts get 10 messages a day, which lets curious partners try it without open-ended model spend and nudges them toward a partner account when they want real work.
• Calls that are scoped and attributable. Partner traffic runs on the partner's own credentials, so each call counts against their quota and appears in their own audit trail. The assistant never poses as a shared service account, which is what keeps usage attribution and abuse detection possible.
• Retries that understand failure. When a request fails, including a transient throttling response, the repair loop passes the error back to the model, bumps the retry counter and re-plans up to a configured maximum. Exhausted retries fall back to advisor mode.

Making access easy

Building the assistant was half the work. Secure, painless access mattered just as much.

The platform's APIs traditionally used standard client credentials, which meant provisioning each new customer by hand. For an assistant aimed at outside partners, "file a request and wait for engineering approval" wasn't acceptable. Asking developers to paste tokens into the chat was no better, because it made assistant-generated calls impossible to tell from manual ones and clouded usage attribution and security monitoring. The team needed one strategy that served two kinds of user: partners who want a friction-free sandbox, and those ready to use live data, with no manual set-up tickets in between.

The answer was a two-tier, self-serve model behind the regular developer login.

• Demo mode is the default. New users can talk to the assistant over synthetic data straight away, without provisioning, waiting or tickets.
• Partner mode switches on when a developer wants live integration. Credentials come from a one-time, self-serve authorization flow linked to the developer's own account and permissions. After connecting, every request is scoped to that identity and fully attributable, so it's clear who is calling and by which path.

That removed the main onboarding bottleneck. Going from "I'm curious" to "I'm integrating" no longer involves an engineering queue.

Where it goes next

The project started as an internal experiment and is now a beta that shortens integration time for corporate developer teams. It runs in the developer portal and has onboarded its first group of partners.

The orchestrator reads standard OpenAPI schemas, so the architecture is meant to scale sideways. Current effort stays focused on validating deeply within the mobility domain, but the routing core is intentionally generic. The roadmap includes taking the same conversational layer to other divisions, such as corporate payments and benefits platforms, adding richer deterministic OData validation so more bad queries are stopped before leaving the assistant, and exploring human-in-the-loop confirmation for write operations so that the assistant can eventually help with mutating workflows safely too.