From GPT-1 to Reasoning Models: What Each Generation Changed for Developers

Every API call you make to a modern language model rests on a handful of ideas that arrived one generation at a time: generative pre-training, scale, in-context learning, alignment from human feedback, native multimodality and extra computation spent at inference time. Knowing which idea came from where makes it much easier to reason about what a model can do, why it fails, and which API features to reach for. This walkthrough traces the GPT line from its 2018 origins to the reasoning-focused models and closes with two practical Python examples for GPT-4o.

Before GPT: why the Transformer mattered

Until around 2018, most natural language processing relied on recurrent neural networks and their LSTM variant. These models read text one token after another, which made training hard to parallelise and caused information from early in a long passage to fade before the end.

In 2017 a Google research paper titled "Attention Is All You Need" presented the Transformer architecture. It swapped recurrence for self-attention, letting every position in a sequence attend directly to every other. OpenAI built its GPT (Generative Pre-trained Transformer) series on this architecture, and in June 2018 published the paper "Improving Language Understanding by Generative Pre-Training", which few people at the time recognised as the start of a major shift in computing.

2018 to 2019: pre-training and the payoff of scale

GPT-1: pre-train first, fine-tune second

The first model was modest by today's standards:

• 117 million parameters
• a decoder-only Transformer with 12 layers
• trained on BooksCorpus, roughly 5 GB of text

Its contribution was a two-stage recipe. First, unsupervised pre-training on raw text let the model pick up grammar, factual associations and the structure of language simply by predicting the next token. Second, a small amount of supervised fine-tuning on a particular task, such as sentiment analysis or classification, was enough to reach state-of-the-art results, without designing a custom architecture for each task.

GPT-2: zero-shot behaviour emerges

The next step was mainly about size:

• 1.5 billion parameters, about ten times GPT-1
• trained on WebText, around 8 million web pages or 40 GB of text

With enough scale, fine-tuning became optional for many tasks. GPT-2 showed zero-shot learning: a prompt such as "Translate to French:" could steer it into a task it had never been trained on explicitly, purely because continuing that text plausibly meant producing a translation.

GPT-2 is also remembered for how it was released. OpenAI initially held back the full model, citing concerns that it could be used to mass-produce fake news and spam, and released it in stages. That decision sparked a lasting public debate about AI safety and about whether powerful models should be openly published.

2020 to 2022: in-context learning and alignment

GPT-3: examples inside the prompt

GPT-3 moved the line from research to commercial product:

• 175 billion parameters, more than a hundred times GPT-2
• trained on a mix of filtered web text, books and Wikipedia, drawn from roughly 45 TB of raw data before filtering

Its headline ability was few-shot learning at scale. Put two or three worked examples in the prompt and the model would continue the pattern, whether writing code, verse or extracting fields from messy data. This is in-context learning: the model adapts from the prompt alone, with no weight updates.

Base GPT-3 had a serious limitation, though. It was trained only to continue text, not to act as a helpful assistant, so it often invented facts, drifted off topic or ignored what it had been asked. The diagram below summarises the fix that followed: alignment from human feedback turns a raw text predictor into an instruction-following model.

[Raw Pre-training]  --> Predicts likely text continuations (often rambling or toxic)
        │
        ▼  + RLHF (Human Feedback Alignment)
[InstructGPT Engine] --> Follows instructions safely and concisely

InstructGPT and ChatGPT: learning from human preferences

The fix was Reinforcement Learning from Human Feedback (RLHF). People ranked model outputs, a reward model learned those preferences, and the language model was then optimised to produce answers humans preferred. The resulting InstructGPT models followed instructions far more directly and usefully.

On 30 November 2022 OpenAI launched ChatGPT, based on the GPT-3.5 series. Pairing multi-turn conversation with RLHF-tuned behaviour made it widely reported as the fastest-growing consumer application up to that point, with an estimated 100 million users within about two months.

2023 to 2024: professional reasoning and multimodality

GPT-4: stronger reasoning, images and tools

GPT-4, released in March 2023, moved the focus from fluent text to reasoning on academic and professional tasks. OpenAI reported results such as a score around the 90th percentile on a simulated Uniform Bar Exam and strong scores on many AP exams. It also brought image understanding, and the GPT-4 family gained features developers now rely on daily: JSON-formatted output, function calling and tool use such as code execution and web search.

Two details are often misstated. OpenAI has not officially published GPT-4's architecture; the widely repeated claim that it is a Mixture of Experts model comes from unconfirmed reports. And the 128k-token context window arrived with GPT-4 Turbo later in 2023, not with the original March release, which offered much smaller windows.

GPT-4o: one network for text, vision and audio

Earlier voice experiences were pipelines: one model transcribed speech, a language model produced text, and a separate engine synthesised audio. GPT-4o, where the "o" stands for "omni", handles text, images and audio in a single end-to-end network. Removing the hand-offs cut voice latency to roughly 200 ms and allowed the model to perceive and produce tone of voice rather than just words.

Reasoning models: spending compute at inference time

As gains from ever-larger training runs became harder to achieve, with high-quality training data increasingly the constraint, attention shifted to test-time compute: letting a model deliberate before it answers. The contrast is simple. A classic GPT response starts streaming tokens as soon as the prompt arrives, while a reasoning model first works through an internal chain of thought and only then produces its final answer.

Traditional GPT Response:  Prompt ──> Immediate Word-by-Word Streaming
o-Series Response:         Prompt ──> Internal Chain-of-Thought (Deliberation) ──> Final Answer

OpenAI's o1 and o3 series are built this way. Instead of committing to the next word immediately, the model spends additional computation exploring edge cases, checking intermediate steps and debugging its own code, which pays off most on mathematics, competitive programming and multi-step logic. The price is higher latency and cost, so reserve them for problems that need deliberation.

Working with GPT-4o in Python

The history explains why GPT-4o changed day-to-day development: instead of chaining a vision model, a language model and a speech engine, a single API call can accept mixed inputs. Two examples show the most useful developer-facing features.

Sending an image and text in one request

With GPT-4o you can put text and images side by side in the content array of a single user message. An image can be passed as a public URL or as base64-encoded data. The example below uses the official Python SDK, reads the API key from an environment variable, asks the model to inspect a photo of concrete for cracks and rate their severity, caps the reply with max_tokens, and prints the text of the first choice.

import os
from openai import OpenAI
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
# Sending a visual inspection query directly to GPT-4o
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Analyze this image for architectural structural damage. Identify any cracks and rate severity."
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://upload.wikimedia.org/wikipedia/commons/e/e1/Concrete_crack_example.jpg"
                    },
                },
            ],
        }
    ],
    max_tokens=500,
)
print(response.choices[0].message.content)

The content is a list of typed parts (text and image_url), which is what lets you mix several images and instructions in one turn. Treat the output as an assessment to review; anything safety-related still needs an expert.

Getting schema-valid JSON with Structured Outputs

Reliably extracting JSON from a model used to mean careful prompting plus defensive parsing with regular expressions. Structured Outputs solves this at the decoding stage: the model's sampling is constrained so that output must conform to the JSON schema you supply. With strict schemas, OpenAI describes this as guaranteeing schema compliance, which is stronger than older "JSON mode", which only guaranteed syntactically valid JSON.

In Python the most convenient route is Pydantic. The example defines an Entity model and an ArticleAnalysis model with a title, summary, a sentiment score between -1.0 and 1.0 and a list of entities, then passes the class as response_format to the SDK's parse helper. The helper converts the class to a JSON schema, sends it with the request, and returns a typed object in message.parsed.

from openai import OpenAI
from pydantic import BaseModel, Field
from typing import List
client = OpenAI()
# Define the exact JSON schema required by your database
class Entity(BaseModel):
    name: str
    category: str = Field(description="Person, Organization, Location, or Date")
class ArticleAnalysis(BaseModel):
    title: str
    summary: str
    sentiment_score: float = Field(description="Score between -1.0 and 1.0")
    key_entities: List[Entity]
# Pass the schema directly into the completion request
completion = client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are an automated intelligence parser."},
        {"role": "user", "content": "OpenAI CEO Sam Altman announced new developer tooling in San Francisco today."}
    ],
    response_format=ArticleAnalysis,
)
# Extract type-safe output directly as a Python object
result: ArticleAnalysis = completion.choices[0].message.parsed
print(f"Summary: {result.summary}")
print(f"Entities: {result.key_entities}")

Two practical notes. The helper lived under client.beta when this feature launched, and newer SDK versions also expose it outside the beta namespace, so check the SDK version your project uses. And while the schema is guaranteed, the values are not: a sentiment score will be a number, but whether it is a sensible number is still a model judgement. Handle refusals and validate business rules as you would for any external input.

The GPT family at a glance

• GPT-1 (2018): 117M parameters; showed that generative pre-training plus light fine-tuning works.
• GPT-2 (2019): 1.5B parameters; zero-shot task performance and a staged, safety-motivated release.
• GPT-3 (2020): 175B parameters; few-shot, in-context learning at scale.
• InstructGPT and ChatGPT (2022): RLHF alignment turned a text predictor into an assistant.
• GPT-4 (2023): professional-level reasoning, image input and tool use.
• GPT-4o (2024): a single network for text, vision and audio with low-latency voice.
• o1 and o3: reasoning models that trade inference time for accuracy on hard problems.

What this evolution means for your stack

• Scale alone stopped being the story. The early jumps came from parameter counts (117M, then 1.5B, then 175B); later progress came as much from alignment techniques like RLHF, multimodal unification and test-time deliberation.
• Models moved from autocomplete to action. Current models plan, call functions and coordinate multi-step tool workflows, which is why agent design is now a mainstream engineering concern.
• Predictability is a feature you can use. Structured Outputs lets you plug model output into typed code paths with a fixed schema instead of parsing free text.
• Efficiency now competes with raw size. Smaller variants such as GPT-4o mini and targeted reasoning models often beat the biggest model on cost and latency for a given task, so choose per workload rather than by headline capability.

Building with these models is increasingly about composing systems of models, tools and schemas rather than sending a single prompt.