KV Cache, Flash Attention & Inference Optimization

Training is parallel and FLOP-bound. Inference is serial and memory-bound. Different bottleneck, different tricks.