Glossary · term175 of 202
softmax
A normalization function that converts a set of raw scores into weights that sum to 1, which is how a model divides its attention across every token in the window.
Related terms
attention budgetThe fixed total of attention weight a model has to spend across a window, forced by softmax to sum to 1, so every added token draws share away from the others.view termattention mechanismThe transformer component that computes a model's output as a weighted sum over every token in its input, with weights set by how strongly each position matches the current query.view term
At a glance
- cited by
- 1 post
- categories
- 1
- first used
- jul 2026
