What Really Runs a Neural Network: AI Compilers Beneath Python

Anyone with a compiler background and some machine-learning knowledge may never have linked the two fields. Lexers, parsers, ASTs and LLVM IR belong to one world, and neural networks belong to another. They meet in AI/ML compilers, and the first surprise is this: a deep learning model isn't really running on Python.

That sounds odd, because a line like the following looks like Python doing everything:

output = model(input)

Python isn't performing billions of matrix multiplications itself, though. A much longer trip is happening underneath.

From a formula to hardware instructions

Picture a tiny network that flows from input through a matrix multiplication and onward:

Input
  ↓
Matrix Multiplication
  ↓
ReLU
  ↓
Matrix Multiplication
  ↓
Output

In Python, its core might read like this:

y = relu(x @ W + b)

It looks simple, but think about what the machine needs. A CPU or GPU has no concept of this function:

relu()

or of this operator:

x @ W

as they appear in code. Hardware executes instructions. Somewhere, the high-level description of the model must be transformed into something the chip can run efficiently, and that transformation is the job of AI/ML compilers.

A simplified picture of the chain, from the model down to the processor:

AI Model
   ↓
PyTorch / TensorFlow
   ↓
Computational Graph
   ↓
Intermediate Representation (IR)
   ↓
Optimisations
   ↓
Lowering
   ↓
Hardware-specific Code
   ↓
Machine Instructions
   ↓
CPU / GPU / NPU

That's a lot of stages, so it's worth taking them one at a time.

What the framework does

With PyTorch, the same idea is written like this:

y = torch.relu(x @ W + b)

A developer doesn't want to deal with GPU registers, memory layout, instruction scheduling or thousands of hardware threads. They state what computation they want. Python and PyTorch provide a comfortable way to express it, while something else has to bring that description close to the hardware.

The computation graph

A neural network can be viewed as a computation graph. The expression:

y = relu(x @ W + b)

can be drawn as a graph of connected operations:

Now the compiler has something concrete to work on. Once the operations are visible, it can ask whether the computation can be made faster. This is where optimization becomes central.

Operator fusion and memory traffic

Take a model containing a chain like this:

MatMul
   ↓
Add
   ↓
ReLU

A compiler may notice that those steps can be merged or tuned for the target hardware. A key technique is operator fusion: instead of treating each operation as a separate step, some are combined, cutting unnecessary work, especially the shuttling of data between memory and compute units.

That matters because moving data is expensive in modern AI workloads. The bottleneck isn't only how fast a result can be calculated. It's also how efficiently the inputs reach the calculation. This is one of the reasons the field is interesting.

Intermediate representations

Next comes a concept familiar from classic compiler courses, the intermediate representation (IR). A traditional compiler rarely goes straight from this:

Source Code

to this:

Machine Code

It passes through IRs. AI compilers follow the same principle, but their representations must capture ideas important to machine learning. Where a general compiler thinks about:

variables
functions
loops

an ML compiler also needs to represent things such as:

tensors
shapes
matrix operations
convolutions
attention
memory layouts

That's what makes projects like MLIR notable. MLIR supplies infrastructure for representing and transforming computations at several levels of abstraction.

Lowering step by step

The compiler gradually moves from a high-level form toward ones that sit closer to the target hardware. Think of descending a staircase:

High-level ML operation
          ↓
Tensor operation
          ↓
Lower-level operation
          ↓
Hardware-oriented operation
          ↓
Machine instructions

At every level, abstract ideas become a bit more concrete, until the compiler emits something the target hardware can use.

The hardware at the bottom

Now we reach the part nobody thinks about when writing:

model(input)

The final computation might run on a CPU, GPU, NPU, TPU or another AI accelerator. None of them care that the original code said:

y = relu(x @ W + b)

They need executable instructions and data laid out sensibly for that device. A GPU may need GPU-specific code and execution models. A CPU uses CPU instructions and vectorization. An accelerator may need an entirely different backend. The same high-level model can therefore be compiled differently depending on its target.

So where does Python fit?

Python hasn't gone away, and it's still very useful. It's the interface for defining models, controlling execution, loading data and building applications. It simply doesn't necessarily carry out every numerical operation itself. A simplified mental model:

This changes the way AI looks. Viewed only from the model side, the picture is short:

Data → Model → Prediction

Add the compiler and there's another layer in between:

Model
  ↓
Compiler
  ↓
Hardware

The model says what computation must happen. The compiler works out how to represent and optimize it. The hardware does the executing.

Why another layer is needed

If frameworks already exist, why add a compiler layer? Because AI hardware is increasingly specialized. There are CPUs, GPUs, TPUs, NPUs, inference accelerators and custom chips, each with different characteristics. A model that runs well on one architecture may need very different tuning on another. The compiler bridges two sides. One is the question of what the model wants to compute:

WHAT
The ML model wants to compute

The other is how a particular device should do it efficiently:

HOW
A specific piece of hardware should execute it efficiently

That bridge keeps growing in importance.

A familiar philosophy in a new domain

Anyone who knows traditional compilers recognizes the basic pipeline:

Source Code
    ↓
Lexer
    ↓
Parser
    ↓
AST
    ↓
IR
    ↓
Optimisation
    ↓
Machine Code

Machine learning applies a similar philosophy to models:

ML Model
    ↓
ML Representation
    ↓
ML IR
    ↓
Optimisation
    ↓
Lowering
    ↓
Hardware Code

The domain differs and the problems are far harder, yet the mindset carries over: represent, transform, optimize, lower, execute.

A natural next idea is building a tiny AI compiler to learn how it works. It wouldn't rival LLVM, XLA or MLIR, and the goal would just be something small enough to understand, perhaps starting from a simple ML language:

Simple ML Language
        ↓
       Parser
        ↓
      ML AST
        ↓
       ML IR
        ↓
   Simple Optimiser
        ↓
    Lowering
        ↓
   LLVM / Backend

The stack under one line

AI usually shows up through a very simple interface:

model(input)

Underneath that single call sits a whole software and hardware stack: frameworks, graphs, intermediate representations, compilers, optimizations, backends, hardware and finally machine instructions. Running a model means running a full compiler and hardware ecosystem, with Python only on top.