A First Taste of Algorithms · Lab 9: The Engine

3. More operations, tested against reality

The model needs squashing and probabilities. Add these inside the class, each three lines, each just its derivative:

    def __pow__(self, p):
        out = Value(self.data ** p, (self,))
        def _backward():
            self.grad += p * (self.data ** (p - 1)) * out.grad
        out._backward = _backward
        return out

    def tanh(self):
        t = math.tanh(self.data)
        out = Value(t, (self,))
        def _backward():
            self.grad += (1 - t * t) * out.grad
        out._backward = _backward
        return out

    def exp(self):
        e = math.exp(self.data)
        out = Value(e, (self,))
        def _backward():
            self.grad += e * out.grad
        out._backward = _backward
        return out

    def log(self):
        out = Value(math.log(self.data), (self,))
        def _backward():
            self.grad += (1 / self.data) * out.grad
        out._backward = _backward
        return out

    def __truediv__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        return self * other ** -1

    def __neg__(self): return self * -1
    def __sub__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        return self + (-other)
    def __radd__(self, other): return self + other
    def __rmul__(self, other): return self * other

Now verify the whole engine the way professionals verify gradients: against finite differences. Calculus says the derivative is the limit of (f(x+h) − f(x−h)) / 2h; compute that numerically and check the engine matches:

def f(av, bv, cv):
    return (math.tanh(av * bv) + cv ** 2) * (av + cv)

a = Value(0.5); b = Value(1.3); c = Value(-0.7)
out = ((a * b).tanh() + c ** 2) * (a + c)
out.backward()

h = 1e-6
ga = (f(0.5 + h, 1.3, -0.7) - f(0.5 - h, 1.3, -0.7)) / (2 * h)
print("a.grad engine:", round(a.grad, 5), "  finite diff:", round(ga, 5))

They agree to five decimals — your sixty lines compute the same derivatives the limit does, on an expression you’d never want to differentiate by hand. This is the test that makes the engine trustworthy: not a proof, but the check every practitioner actually runs.

4. A tiny model, trained by your own engine

Before numpy, prove the engine can learn — train a tiny next-letter model with nothing but Values and your backward sweep. Small enough to be pure Value, real enough to work:

```python random.seed(0) names = [“ada”, “gauss”, “euler”, “abel”, “nash”, “hardy”, “klein”, “peano”, “boole”, “bayes”] chars = sorted(set(“”.join(names))) + [“.”] stoi = {c: i for i, c in enumerate(chars)} V = len(chars)

All 32 chapters of A First Taste of Algorithms · Foundations · All courses