From Text Predictor to Software Component: Schemas and Function Calling

Before agents, RAG and MCP, there's a more basic question: what is an LLM, and what does it actually do? A large language model does exactly one thing. It predicts the next token. It reads the text so far, picks the likeliest next token, appends it, and predicts again, looping until it's finished.

This loop is the whole mechanism. Everything else people build around LLMs is layered on top of it. Two consequences follow. First, an LLM knows nothing about types, numbers, objects or databases. It produces text and only text. Second, its predictions come from training data, which is a snapshot of the past, so answers can be stale unless you put fresh information into the prompt yourself.

Being a plain text generator becomes a real problem as soon as you build software on it. Software is deterministic and typed, with objects, methods and numbers. A raw LLM hands you a string.

The problem in concrete terms

Say you ask a model for the weather. Any of these could come back:

• "It's 22°C and sunny"
• "Around 22 degrees, clear skies"
• "I'd say about 22°"

All three are correct and all three are useless to your code. Your code wants response.temp_c to be a number every time, so it can store it, calculate with it or plot it. You can't do any of that with a sentence, and you can't JSON.parse() English. A regex might cope today and fail on tomorrow's phrasing.

Notice the failure mode here. Nothing is broken and nobody made a mistake; the model did its job, and the job happened to produce prose when your program wanted a value. So the trouble isn't that the model is wrong. The output has no fixed shape. To build on it you need predictable output, not just correct output. Solving that took a few ideas that stack on each other.

JSON Schema

Start with JSON Schema. It's a recipe for what a JSON object should look like once filled in. It holds no data. It only describes shape: which fields exist, what type each one has, and which are required.

{
  "type": "object",
  "properties": {
    "city":     { "type": "string",  "description": "City name, e.g. Istanbul" },
    "temp_c":   { "type": "number",  "description": "Temperature in Celsius" },
    "is_sunny": { "type": "boolean" }
  },
  "required": ["city", "temp_c"]
}

Each field gets a type such as string, number or boolean. A description gives the model more context about a field, and the required list says which fields must be present. In this example is_sunny is optional.

This is the kind of thing a weather lookup needs: a city as a string, a temperature as a number, and a flag that may or may not be there. A schema does nothing by itself. Treat it as a contract that says "valid data looks like this." The next step is handing that contract to the model.

Structured output

In 2024 LLMs got a feature called structured output. You give the model a JSON Schema and it has to fill it in, nothing more and nothing less. (An earlier, weaker version only guaranteed valid JSON, with no promise about fields or types. Structured output closed that gap.)

It works through two things together, and neither is enough alone.

The first is training. From 2024 on, new models were trained to understand schemas and to emit output that matches them. But a model still guesses one token at a time, so even a well-trained one followed the schema around 90% of the time. For real software, 90% is a bug waiting to happen.

The second is plain code in the inference engine, called constrained decoding. The model emits tokens one by one, and at every step this code checks the schema and removes any token that would break it by setting its probability to zero. The schema is turned into a grammar, a set of rules for which tokens may come next, and the model can only choose from what's left.

The trained model does most of the work, and constrained decoding turns that into something close to a 100% guarantee. Think of the split this way: training makes the right answer likely, and the decoding step makes a wrong shape impossible. The model still outputs a string, but one that always fits the recipe. You then parse it into a real object and trust it.

// the recipe, the same kind of JSON Schema we saw above
const schema = {
  type: "object",
  properties: {
    city:   { type: "string" },
    temp_c: { type: "number" }
  },
  required: ["city", "temp_c"]
};

// ask the model, and pass the schema to force the shape
const report = await askModel({
  prompt: "Weather in Istanbul, sunny, 22°",
  schema: schema,
});

report.temp_c; // 22, always a number. no parsing, no regex.

Now typed code and the model can finally cooperate, because the output has a shape you can rely on.

There's a bigger consequence. A function's parameters are just fields with types, which is exactly what a structured output is. If the model can fill in any schema, it can fill in the arguments of a function. And once it can produce correct arguments, it can effectively tell your code which function to run and with which values. That gives you more than clean data. It gives you method calls, and it has a name.

Function calling

With function calling you send the prompt together with a list of named functions. Each has a name, a description saying what it does, and a JSON Schema for its parameters.

{
  "name": "get_weather",
  "description": "Get the current weather for a city.",
  "parameters": {
    "type": "object",
    "properties": { "city": { "type": "string" } },
    "required": ["city"]
  }
}

The description carries a lot of weight, because it's how the model decides this function is the right one. The model reads your prompt, concludes it needs the weather, and replies with the call it wants made:

{ "name": "get_weather", "input": { "city": "Istanbul" } }

That reply is nothing but a structured output. input holds the arguments and name identifies the function. The model ran nothing, and it can't. On your side you parse the JSON, find the matching function from name, pass input as its arguments, run it, and send the result back to the model. Put simply, function calling is a structured output holding the arguments, combined with the model's choice of which function fits.

That's what opens things up. Whatever you can wrap in a function becomes reachable: a weather API, a payment system, a search engine, a database, your own services. Any feature described as a function with a schema can be plugged into an LLM. Notice how little the model needs for this. It never touches your systems directly; it only proposes a call, and your code decides whether and how to run it. That's how a text box becomes something that does things in the real world, through your code.

The hard part was never describing what you want. It was getting a text generator to follow the rules. Once that was solved, an LLM could become a normal building block in your code, like any other tool.