Papers › A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit...
A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)
Yan Li, Tianyi Zhang, Zechuan Li, Soyeon Caren Han
Transformer-based Large Language Models (LLMs) struggle to process inputs exceeding their training context window, with performance degrading due to positional out-of-distribution (O.O.D.) that disrupt attention computations. Existing solutions, fine-tuning and training-free methods, are limited by computational inefficiency, attention logit outliers or loss of local positional information. To address this, we propose Greedy Attention Logit Interpolation (GALI), a training-free length extrapolation method that maximizes the utilization of pretrained positional intervals while avoiding attention logit outliers through attention logit interpolation. The result demonstrates that GALI consistently outperforms state-of-the-art training-free methods. Our findings reveal that LLMs interpret positional intervals unevenly within their training context window, suggesting that extrapolating within a smaller positional interval range yields superior results-even for short-context tasks. GALI represents a significant step toward resolving the positional O.O.D. challenge, enabling more reliable long-text understanding in LLMs. Our implementation of GALI, along with the experiments from our paper, is open-sourced at https://github.com/AcademyCityL/GALI.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Long-Context Understanding | L-Eval | GALI(Llama3-8b-ins-4k-to-16k) | Average Score | 59.21 | #1 of 4 | Archive leaderboard | report |
| Long-Context Understanding | L-Eval | GALI(Llama3-8b-ins-4k-to-32k) | Average Score | 59.10 | #2 of 4 | Archive leaderboard | report |
| Long-Context Understanding | L-Eval | GALI(Llama3-8b-ins-8k-to-32k) | Average Score | 42.79 | #3 of 4 | Archive leaderboard | report |
| Long-Context Understanding | L-Eval | GALI(Llama3-8b-ins-8k-to-16k) | Average Score | 42.32 | #4 of 4 | Archive leaderboard | report |
| Long-Context Understanding | LongBench | GALI(Llama3-8b-ins-4k-to-16k) | Average Score | 46.22 | #1 of 3 | Archive leaderboard | report |
| Long-Context Understanding | LongBench | GALI(Llama3-8b-ins-8k-to-32k) | Average Score | 45.38 | #2 of 3 | Archive leaderboard | report |
| Long-Context Understanding | LongBench | GALI(Llama3-8b-ins-8k-to-16k) | Average Score | 45.17 | #3 of 3 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections