better tp

content/article.md

@@ -72,7 +72,7 @@ So, what are the principles of `transformers`? We will try to summarize the foun
 </li>  
 <li class="tenet">
 <a id="modular-toolbox"></a>
-<strong>Modular Toolbox (Not Framework)</strong>
+<strong>Modular Toolbox (Not A Framework)</strong>
 <p>We ARE a toolbox. What we are not is a framework: you should not be FORCED to rewrite every modeling, but it is <em>better</em> for your model to be able to inherit from PreTrainedModel and have enabled TensorParallel, from_pretrained, sharding, push_to_hub, loss, as well as PEFT/TRL/SGLang/vLLM.</p>
 <em>This is the largest change. Provide tools and utilities, but don't force users into a rigid framework.</em>
 </li>
@@ -136,7 +136,21 @@ For better _information_, we plan to use `python` features such as `Annotated` f
 
 ## <a id="simpler-tensor-parallelism"></a> Simpler Tensor Parallelism
 
-We want to touch minimally to the modeling code, and only modify it when _architectural changes_ are involved. For instance, for tensor parallelism, we instead now specify a simple `tp_plan`.
+We want to touch minimally to the modeling code, and only modify it when _architectural changes_ are involved. For instance, for tensor parallelism, we instead now specify a simple `tp_plan`. 
+
+It is written once in the config and passed to `.from_pretrained()`. 
+
+The plan maps module name patterns to partitioning strategies. Strategies are resolved by the internal `ParallelInterface`, which wires to sharding implementations `ColwiseParallel`, `RowwiseParallel`, packed variants, and so on.
+
+{{{fragment-tp-plan}}}
+
+
+Which allows a user to run with multiple processes per node, e.g. 4 GPUs:
+
+`torchrun --nproc-per-node 4 demo.py`
+
+Semantics stay in the model (a Linear stays a Linear), distribution is orthogonal and declared via strings: "colwise" splits columns of weights/bias across ranks; "rowwise" splits rows; packed variants shard fused weights; The mapping keys accept glob patterns like `layers.*.mlp.down_proj` to target repeated submodules.
+
 
 ## <a id="layers-attentions-caches"></a> Layers, attentions and caches
 With th
@@ -170,7 +184,6 @@ So what do we see? Llama is a basis for many models, and it shows.
 Radically different architectures such as mamba have spawned their own dependency subgraph.
 {{{fragment-d3-graph}}}
 
-![Graph showing modular related models](static/graph_modular_related_models.png)
 
  But there is no similar miracle for VLMs across the board.
 As you can see, there is a small DETR island, a little llava pocket, and so on, but it's not comparable to the centrality observed. 
@@ -255,8 +268,7 @@ But this is _within_ the modeling file, not in the `PreTrainedModel` base class.
 So the question abounds naturally: How can we modularize more?
 I took again a similarity measure and looked at the existing graphs. The tool is available on this [ZeroGPU-enabled Space](https://huggingface.co/spaces/Molbap/transformers-modular-refactor). It scans the whole transformers repository, and outputs a graph of candidates across models, using either a Jaccard similarity index (simple) or a SentenceTransformers embedding model. It is understandable that [encoder models still have a lion's share of the game.](#encoders-ftw) See also [Tom Aarsen and Arhur Bresnu's great blog post on the topic of sparse embeddings.](https://huggingface.co/blog/train-sparse-encoder).
 
-
-![Modular candidates analysis](static/modular_candidates.png)
+{{fragment-space-embed}}
 
 ## <a id="encoders-ftw"></a> The neverending stories of encoder models.
 
@@ -324,19 +336,8 @@ Adding a model to transformers means:
 
 Having a clean _external_ API allows us to work on the true inner workings of transformers. One of the few recent additions was the _CUDA warmup_ via `caching_allocator_warmup` which improved massively the loading footprint by pre-allocating GPU memory to avoid malloc bottlenecks during model loading. 
 
-{{{fragment-memory-profiler}}}
-
-### Linkedin post (to remove)
-Linkedin post for videos:
-
-In transformers, how do we deal with cross-model dependencies, while supporting ~400 models? Maybe you've seen the same 200-lines functions in too many _modeling_file.py_? Duplication isn’t inevitable.
-
-The “one‑model/one‑file” rule keeps every model readable and runnable. It also means identical code is copied hundreds of times. Maintenance hurts, contributor PRs snowball, and vision–language models especially end up in siloed forks.
-
-modular_*.py fixes the trade‑off, by auto-generating the modeling file from a modular file, which can use inheritance.
-
-With a small analyser I’ve mapped which models already share modular pieces and which 100‑plus still repeat themselves. Red nodes in the graph = lowest‑hanging fruit for refactor; blue = already modular.
+{{{fragment-warmup_demo}}}
 
-The result: contributors can focus on novel layers instead of boilerplate, reviews shrink from “new file diff” to “does this override make sense?”, and the codebase stays something you can actually open and read.
+## What is coming next
 
-If you maintain or ship models on top of Transformers, take a look at modular, in 2025 it’s how we keep shipping breadth without the bloat. 🛠️
+It sounds dumb, but it's true: the future is very soon. One tenet that will be broken when the next major version is released, v5, [backwards compatibility](#backwards-compatibility) will be heavily broken. Instead, what we aim to be is way more of a [modular toolbox](#modular-toolbox), while maintaining a [consistent public surface](#consistent-public-surface).

webpack.config.js

@@ -123,7 +123,7 @@ module.exports = {
                             const article = document.querySelector('d-article');
                             const toc = document.querySelector('d-contents');
                             if (toc) {
-                                const headings = article.querySelectorAll('h1, h2, h3, h4');
+                                const headings = [...article.querySelectorAll('h1, h2, h3, h4')].filter(h => !h.hasAttribute('data-no-toc'));
                                 let ToC = '<nav role="navigation" class="l-text figcaption">';
                                 ToC += '<div class="toc-header"><span class="toc-title">Table of Contents</span></div>';
                                 ToC += '<div class="toc-content">';