Eight Linear Algebra Ideas Behind Embeddings, LoRA and PCA

You don't need a math degree to build with AI, but a little linear algebra clears up a lot of mystery. Models can't read text, images or sound directly. Everything is first turned into numbers. Once that's done, you still need a way to arrange the numbers so a model can compare items, spot patterns and make predictions, and linear algebra supplies that toolkit. Here are eight concepts worth knowing before you call yourself an AI engineer.

1. Vectors hold the data

A vector is an ordered list of numbers describing something. A house could be:

[300000, 150, 3]

Those three entries stand for the price ($300,000), the size (150 square meters) and the number of bedrooms (3).

Why care? Once things are vectors, a model can compare them. Two houses with similar price, size and bedroom count end up with similar vectors, and that's how the model sees they're related.

2. Matrices stack many vectors

Think of a matrix as a grid of rows and columns. One vector describes one item, and a matrix holds a whole set:

[ 300000   150   3 ]  → House 1
[ 450000   200   4 ]  → House 2
[ 250000   120   2 ]  → House 3

Each row is a house and each column is a feature: price, size, bedrooms. Matrices let a system handle plenty of data in one go, not one vector at a time, which keeps calculations on big datasets efficient.

3. Eigenvalues and eigenvectors point to what matters

An eigenvector is a special vector that, after a transformation, still points along the same line. It may get longer or shorter, but it doesn't swing toward a new direction. Its eigenvalue tells you how much it was stretched or squeezed. The relationship is written as:

Av=λv

A = the transformation
v = the eigenvector (special direction)
λ = the eigenvalue (how much it stretches or shrinks)

To make sense of that, remember that a vector has a direction and a magnitude, meaning its length.

        ●
       ↗
      /
     /
    /
   /
  ●

Magnitude is how long the arrow is, and direction is where it points. Multiplying a vector by a matrix transforms it; the result can be stretched, shrunk, rotated or changed in some other way. A tiny case:

[3].[2 0] = [6]
[4] [0 2]   [4]

The vector [3, 4] has magnitude 5 and points at roughly 53.13 degrees. After the multiplication it becomes [6, 8], with magnitude 10 and the same 53.13 degrees. It got twice as long without turning, so it's an eigenvector with eigenvalue 2.

This matters because eigen-analysis exposes the dominant patterns or directions in data. Principal Component Analysis (PCA) relies on it to shrink datasets while keeping the important information.

4. Dot product and cosine similarity measure likeness

Take two vectors of the same length, combine them, and the dot product returns one number:

A · B = (a₁ × b₁) + (a₂ × b₂) + ... + (aₙ × bₙ)

Run it on two houses:

House 1 (A) = [300000, 150, 3]
House 2 (B) = [450000, 200, 4]

A · B = (300000 × 450000) + (150 × 200) + (3 × 4)
      = 135,000,030,012

That result has two flaws.

The first is scale. Price numbers are huge next to bedrooms, so price dominates the answer and the bedroom count barely registers. The cure is feature scaling: squeeze every feature into a similar range, such as 0 to 1, where 0 is the smallest value seen and 1 the largest.

// All three features are on roughly the same scale (0–1),
// so price doesn't dominate simply because its original number is much larger.

A' = [0.60,0.50,0.60]
B' = [0.90,0.67,0.80]

The second is length. With a raw dot product, a longer vector scores higher just for being longer. Two fixes exist: normalize the vectors first and then take the dot product, or compute the dot product and finish with cosine similarity. Normalizing needs the L2 norm, the vector's straight-line length:

# L2 Norm (Euclidean Distance) → Straight-line distance from the origin.
# It measures the overall length of a vector.

||x||₂ = √(x₁² + x₂²+...+xₙ²)

||A'||₂ = √(0.60²+0.50²+0.60²) ≈ 0.985
A'' = [0.60/0.985, 0.50/0.985 ,0.60/0.985] ≈ [0.609,  0.508,  0.609]

||B'||₂ = √(0.90²+0.67²+0.80²) ≈ 1.378
B'' = [0.90/1.378, 0.67/1.378, 0.80/1.378] ≈[0.653,  0.486,  0.581]


A''⋅B'' = (0.609)(0.653)+(0.508)(0.486)+(0.609)(0.581)
        ≈ 0.398+0.247+0.354
        ≈0.999

And the cosine calculation itself:

# cosine similarity
A' · B' = (0.60×0.90)+(0.50×0.67)+(0.60×0.80)
        ≈ 1.355

||A'||₂ ≈ 0.985

||B′||₂ ≈ 1.378

Cosine Similarity = (A' · B')/||A'||||B'||
                  ≈ 1.355/(0.985*1.378)
                  ≈ 0.999

A common reading of the score:

• 1.0 means the same direction, maximum similarity
• 0.9 up to 1.0 is extremely similar
• 0.7 up to 0.9 is fairly similar
• 0.4 up to 0.7 is moderately similar
• above 0 and below 0.4 is weak similarity
• 0 means perpendicular, no shared direction
• below 0 means increasingly opposite
• -1.0 means exactly opposite, maximum dissimilarity

Don't treat those bands as law. What counts as similar depends on your data and your model. The idea matters because similarity search, recommenders and neural networks all depend on how strongly two sets of numbers relate.

5. Norms measure size and error

Vectors can stand for houses, users, products, images or words. A norm gives you a single number for a vector's size, which lets you compare vectors by magnitude. Three common ones:

The L1 norm, also called Manhattan distance, adds up the absolute values of all entries, giving total magnitude.

||x||₁ = ∣x₁∣+∣x₂∣+⋯+∣xₙ∣

The L2 norm, or Euclidean distance, is the straight-line distance from the origin, so it captures overall length.

||x||₂ = √(x₁² + x₂²+...+xₙ²)

The L-infinity norm looks only at the largest absolute value.

||x||∞ = max⁡(∣x₁∣,∣x₂∣, ⋯ , ∣xₙ∣)

For "how big," picture the vector as an arrow: its norm is the arrow's length, and a longer vector has a larger norm. That's what powers the normalization step above. Normalizing rescales a vector to length 1, so comparisons depend on direction and not on size.

For "how wrong," treat the gap between prediction and truth as an error vector. Its norm says how far off you were.

yactual=[10,20]
ypredicted=[8,23]

Error     = [10,20]−[8,23]=[2,−3]
||error||₁=∣2∣+∣−3∣=5
So the prediction has a total absolute error of 5.

A bigger error norm means a worse prediction. A value of 5 isn't good or bad by itself; you judge it against other predictions or a baseline.

Norms show up in practice in three places:

• Regularization (L1 and L2) fights overfitting by penalizing large weights in the loss.
• Loss functions use them: Mean Absolute Error is built on the L1 norm, and Mean Squared Error on the squared L2 norm.
• Gradient clipping uses the L2 norm to keep exploding gradients under control while training deep networks.

6. Matrix multiplication combines information

A matrix does more than store numbers. Applied to an input vector, it produces an output vector by scaling, rotating, reflecting or otherwise transforming it. Multiplying matrices mixes numbers from both to create new ones. Suppose three houses are each described by three scaled features:

Houses =
    [ 0.50 0.50 0.50 ]
    [ 0.75 1.00 1.00 ]
    [ 0.25 0.00 0.00 ]

Now say the model has learned that features differ in importance, with price at 0.5, size at 0.3 and bedrooms at 0.2:

Weights =
  [ 0.5 ]
  [ 0.3 ]
  [ 0.2 ]

Multiply the house data by the weights:

Houses × Weights → Scores

The outcome:

House 1 → 0.500
House 2 → 0.875
House 3 → 0.125

A higher score means more weighted value under the importance you assigned. Matrix multiplication is everywhere in machine learning, neural networks, data processing and computer graphics, because it combines and transforms data efficiently.

7. SVD simplifies matrices

Many matrices aren't square, and non-square matrices have no eigenvectors. Singular Value Decomposition works on any matrix of any shape by splitting it into three simpler ones:

A=UΣVᵀ

Read it as U rotating the output, Sigma scaling the important directions and V transpose rotating the input. SVD lets you shrink large matrices while keeping their most important content. It's used for dimensionality reduction, compression, recommender systems and low-rank methods such as LoRA. Keep only the strongest components and data and models get smaller, faster and cheaper.

8. Rank explains low-rank tricks

Rank tells you how much unique information a matrix carries. A matrix can have many rows and columns and still repeat itself, which gives it a low rank. Look at this one:

[1 2]
[2 4]

The second row is simply twice the first, so only one row is independent and the rank is 1.

That's why low-rank methods are popular. Rather than storing every number, you represent the essential content with fewer numbers, cutting memory and compute while keeping most of what matters.

How it fits together

Between them, these eight ideas let you encode data, reshape it, gauge its size, check how alike two things are and boil it down to something smaller. Don't try to memorize formulas. Aim to know what each concept does, why it's useful and where it appears in real systems. That intuition makes the tools, libraries and models you use far less opaque. Is there a ninth concept you'd add to the list?