https://k4i.top/tags/rope/Recent content in Rope on k4i's blogHugo -- gohugo.ioensky_io@outlook.com (K4i)sky_io@outlook.com (K4i)All content is subject to the license of <a rel="license noopener" href="https://creativecommons.org/licenses/by-nc-sa/4.0/" target="_blank">CC BY-NC-SA 4.0</a> .Thu, 28 May 2026 21:53:12 +0800https://k4i.top/posts/positional-encoding-to-rope/Thu, 28 May 2026 21:53:12 +0800sky_io@outlook.com (K4i)Thu, 04 Jun 2026 01:12:37 +0800https://k4i.top/posts/positional-encoding-to-rope/<h2 id="introduction">Introduction</h2> <p>Self-attention has a surprising weakness: by itself, it does not know word order.</p> <p>Consider these two sentences:</p> <ul> <li>I like you</li> <li>You like me</li> </ul> <p>They contain almost the same tokens, but their meanings are different. RNNs read inputs step by step, and CNNs preserve local neighborhoods through convolution windows. Standard self-attention, however, mostly compares all token vectors with all other token vectors. Its core formula is:</p> <p>$$\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V$$</p> <p>If we permute the input tokens and permute the output in the same way, the attention computation does not naturally resist that permutation. Attention is good at modeling content relationships, but order must be injected separately.</p>K4ifeatured imagellmtransformerattentionposition-encodingropeAI