A First Taste of Algorithms · Lab 9: The Engine

6. The machine speaks

Generate names — feed the model a start, take its prediction, feed it back, repeat:

def sample(temp=1.0):
    ctx = [stoi["."]] * CONTEXT; out = []
    for _ in range(20):
        emb = C[np.array([ctx])].reshape(1, -1)
        h = np.tanh(emb @ W1 + b1)
        logits = (h @ W2 + b2)[0] / temp
        logits -= logits.max(); pr = np.exp(logits); pr /= pr.sum()
        ix = rng.choice(V, p=pr)
        if itos[ix] == ".": break
        out.append(itos[ix]); ctx = ctx[1:] + [ix]
    return "".join(out)

known = set(names)
print("generated names (★ = novel):")
for _ in range(25):
    name = sample(1.0)
    print(f"   {name:<16} {'★ never existed' if name not in known else '(in training)'}")

Names like Riemanujan, Dantor, Wilevin — nobody’s, everybody’s plausible. The model learned the music of the names without copying them, which is learning itself. Some exact training names may appear too; Problem 3.3 asks what that does and doesn’t prove. Then run §6 again with temp=0.4 and temp=1.6 (Problem 2.3’s prediction): cold sampling plays it safe and wordlike, hot sampling gets adventurous and strange.

7. A real attention head

The toy model read a flat window. Now build the unit behind the transformer — one self-attention head, from the chapter’s five equations, in numpy. This is the architecture of your era, small enough to read whole.

The pieces, dimensioned (D = model dimension, V = vocabulary, BLOCK = longest sequence):

```python import numpy as np rng = np.random.default_rng(0)

reuse names / chars / stoi / itos / V from §5; set up dimensions

✦ ✦ ✦

BLOCK = 16 D = 32; Dk = 32 Wtok = rng.standard_normal((V, D)) * 0.1 # character -> vector Wpos = rng.standard_normal((BLOCK, D)) * 0.1 # position -> vector WQ = rng.standard_normal((D, Dk)) * 0.1 # query projection WK = rng.standard_normal((D, Dk)) * 0.1 # key projection WV = rng.standard_normal((D, D)) * 0.1 # value projection Wout = rng.standard_normal((D, V)) * 0.1 # context -> next-char scores

def softmax_rows(M): M = M - M.max(axis=-1, keepdims=True) e = np.exp(M) return e / e.sum(axis=-1, keepdims=True)

The forward pass *is* the five equations, in order — embed, project to Q/K/V, score, mask, weight, combine, read out:

```python
def forward(seq):
    T = len(seq)
    X = Wtok[seq] + Wpos[:T]                  # (T,D)  symbols + positions
    Q = X @ WQ; K = X @ WK; Vv = X @ WV       # the three projections
    scores = Q @ K.T / np.sqrt(Dk)            # (T,T)  query · key, scaled
    mask = np.triu(np.ones((T, T)), k=1).astype(bool)
    scores = np.where(mask, -1e9, scores)     # causal: no peeking ahead
    A = softmax_rows(scores)                  # (T,T)  attention weights
    context = A @ Vv                          # (T,D)  weighted values
    logits = context @ Wout                   # (T,V)  next-char scores
    return logits, A

seq = [stoi[c] for c in "." + "riemann"]
logits, A = forward(seq)
print("attention matrix shape:", A.shape, " (should be square, one row per position)")

Read A and confirm it is lower-triangular — each position attends only to itself and earlier, the mask at work. That triangle is the difference between generating and blank-filling.

Training is the same loop (the full gradient code is provided in the notebook — it is the chapter’s backward sweep applied to each matrix, batched). Run it, then inspect what the model learned to look at:

```python

All 32 chapters of A First Taste of Algorithms · Foundations · All courses