Everything is a UOp – reading the deep learning stack behind comma[dot]AI

i’ve used PyTorch for years. a @ b, .backward(), .cuda(). works every time. i never thought about what’s underneath. then i came across tinygrad. it’s the deep learning stack behind comma.ai’s openpilot, the open-source self-driving system. 17K lines of Python. PyTorch is 3 million lines of C++. this thing fits the whole pipeline, including the compiler, in less code than some test suites. i figured i’d spend an afternoon reading it. to give an outline of how tinygrad works, we would explore these 4 stages: the first thing i noticed to start with, we need to know how a tensor operation works. the fundamental tensor operation used in DL libraries is matrix multiplication. so, i opened a terminal and typed this out. from tinygrad import Tensor a = Tensor.rand(4, 4) b = Tensor.rand(4, 4) c = a.matmul(b) print(c.shape) # (4, 4)