Knowledge note
Attention Foundations
Foundational sequence-modeling concepts behind attention-based LLMs: the Transformer, softmax, and RNN-to-LSTM background.
Attention Foundations
- The Transformer: Attention Is All You Need — The foundational architecture: scaled dot-product attention, multi-head self-attention, sinusoidal positional encoding, encoder-decoder stacks, and the training recipe that launched modern LLMs.
- Layer Normalization in Transformers (term) — Why each token is normalized across hidden features rather than across a batch, with equations, axis diagrams, implementation cautions, and a worked trace.
- The Softmax Function: Properties, Motivation, and Interpretation — Tutorial covering score-difference semantics, α parameter interpretation, three conceptual justifications, IO vs. IM model taxonomy, and complete mathematical properties.
- Perplexity (PPL): From Next-Token Loss to Reliable Evaluation — Explains exponentiated average next-token negative log likelihood, a worked example, fixed-length sliding-window evaluation, and tokenizer/protocol comparison caveats.
- Gray Code: One-Bit Transitions for Safe State Changes — Binary-reflected Gray code, one-bit adjacency, reflect-and-prefix construction, shift/XOR conversion, and transition-safe hardware and communication uses.
- Recurrent Neural Networks: From RNN to LSTM — Sequence processing through shared recurrent weights and hidden state, the long-term dependency problem, LSTM gating, and the conceptual bridge to linear attention’s RNN mode.