An Ignored Target Token Still Exists in the Input
Loss masking removes a position from cross entropy. It does not remove its token from a transformer input or prevent other positions from attending to it.
Topic · 10 stories
10 stories tagged transformers on AI Tamer, newest first.
Loss masking removes a position from cross entropy. It does not remove its token from a transformer input or prevent other positions from attending to it.
RMSNorm divides a vector by its root mean square and learns a gain for each feature. A small numeric example shows what changes when LayerNorm’s mean subtraction is removed.
A sparse mixture-of-experts router can choose an expert that has no remaining token slots. Here is what overflow means in Switch Transformer and how capacity changes the tradeoff.
Grouped-query attention gives several query heads one key-value head pair. That shrinks decoding caches, with speed and quality gains that depend on the model and workload.
RoPE rotates query and key coordinate pairs so their attention score depends on relative position. The math helps explain context extension and its limits.
A repetition penalty discourages reused tokens. An n-gram rule can rule out a repeated sequence entirely, including one the text needs.
Padding to the longest input lets each batch set its length. Padding to a fixed length uses a chosen target, while truncation handles inputs that exceed a limit.
A chat template turns role-labeled messages into the token sequence a model expects. The markers and the start of a reply depend on the model.
A cut-off answer can result from a stop string, a token cap, or a time limit. Check each setting before changing it.
Padding lets prompts of different lengths share a batch. An attention mask marks which positions the model should attend to.