Glossary · term14 of 202
attention budget
The fixed total of attention weight a model has to spend across a window, forced by softmax to sum to 1, so every added token draws share away from the others.
also written as fixed attention budget
Related terms
attention mechanismThe transformer component that computes a model's output as a weighted sum over every token in its input, with weights set by how strongly each position matches the current query.view termcontext poisoningA failed attempt or wrong answer left sitting in a conversation keeps competing for attention on every later turn, dragging output back toward the same mistake even after correction.view termdistractor sensitivityA model's tendency to lose accuracy as irrelevant or off-task tokens accumulate in its context, even when the genuinely relevant information is still present.view termsoftmaxA normalization function that converts a set of raw scores into weights that sum to 1, which is how a model divides its attention across every token in the window.view term
At a glance
- cited by
- 1 post
- categories
- 1
- first used
- jul 2026
