Shapes first — vectors, matrices, and why matmul is the whole game

Almost every bug I hit in my first term was a shape bug, not a maths bug. So this note starts where the errors start: what the objects are, and what multiplying them is allowed to mean.

The three objects

A vector is a list of numbers with a direction attached, written bold: x ∈ ℝⁿ. In code it is a 1-D array of length n.

A matrix is a table, W ∈ ℝ^(m×n): m rows, n columns. Rows first, always — W[i, j] is row i, column j.

A tensor is the same idea with more axes. A batch of 32 RGB images at 224×224 is shape (32, 3, 224, 224). Nothing new, just more indices.

The dot product is a similarity score

For a, b ∈ ℝⁿ:

a · b = Σᵢ aᵢbᵢ        for i = 1 … n

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
a @ b          # 1*4 + 2*5 + 3*6 = 32

Two facts to carry everywhere: a·a = ‖a‖², and a·b = ‖a‖‖b‖cos θ. The second is why the dot product turns up whenever something needs to be scored for similarity — cosine similarity in retrieval, and the QKᵀ term in attention, are both this line.

Matrix–vector: a weighted sum of columns

Wx is not a mysterious operation. It is a linear combination of the columns of W, with the entries of x as the weights:

W = [[2, 0],      x = [3,
     [1, 4],           5]
     [0, 3]]

Wx = 3*[2,1,0] + 5*[0,4,3] = [6, 3, 0] + [0, 20, 15] = [6, 23, 15]

Row-wise it reads as “one dot product per row”, which is the version to hold when you think about a linear layer: each output neuron dots its own weight row with the input.

The shape rule, which is the entire debugging strategy

(m × n) · (n × p)  ->  (m × p)

The inner dimensions must match and they vanish; the outer ones survive. A (32, 784) batch times a (784, 128) weight gives (32, 128). That is a whole layer.

When something throws, write the shapes down in that order before touching the code. Nine times out of ten a transpose is missing.

X = np.random.randn(32, 784)     # batch of 32
W = np.random.randn(784, 128)
H = X @ W                        # (32, 128)  ✓
W @ X                            # ValueError: 784 vs 32

Broadcasting, and the bug it hides

NumPy stretches size-1 axes to match, right-aligned:

H = X @ W          # (32, 128)
b = np.zeros(128)  # (128,)  -> broadcast to (32, 128)
H + b              # fine: one bias per feature

That is the intended use. The failure case looks almost identical:

y      = np.array([1, 2, 3])        # (3,)
y_pred = np.array([[1], [2], [3]])  # (3, 1)
y - y_pred                          # (3, 3), silently

No error, wrong answer, and a loss that trains to nonsense. Any time a residual looks suspicious, print .shape on both sides first.

Cost

Multiplying (m×n) by (n×p) costs O(mnp) multiply-adds. For a 32×784 batch through a 784×128 layer that is about 3.2M operations — trivial on a GPU, which is exactly why deep learning is a pile of matmuls rather than a pile of loops. The rule that follows: if you wrote a Python for over an array, there is a matrix version, and it is usually 50× faster.

What to take forward

Shapes before maths. Write them down.
Wx = weighted sum of columns = one dot product per row.
Inner dims match and cancel; outer dims survive.
Broadcasting is convenient and is also how silent bugs get in.

— Ishaan SandhwarLinear algebraNumPyFundamentals