match dist

content/article.md

@@ -1,5 +1,3 @@
-# Digging through tenets and time
-
 
 ## Introduction 
 
@@ -59,7 +57,8 @@ As I was looking for things to improve and make better, it's one of the iteratio
 
 However, both of these works were already pointing at some drawbacks, which have been iteratively addressed. [Transformers has gone modular](https://huggingface.co/docs/transformers/en/modular_transformers) , allowing a form of inheritance without breaking [One model, One file](#one-model-one-file). If you're familiar with this, you can [skip this section](#^attention-classes) and go to the next one.
 
-We amended the principle of [DRY*](#do-repeat-yourself) by removing progressively 
+We amended the principle of [DRY*](#do-repeat-yourself) by removing progressively all pieces of code that were "copied from" another file.
+
 It is explained in details in the documentation above, but overall it works like this, you define a `modular_` file that can inherit from _any function across all other modeling, configuration and processor files_: 
 
 <summary>Auto-generated modeling code</summary>
@@ -82,9 +81,16 @@ We often read and understand that `kwargs` are criticized, and we are typing the
 
 It is a strength of the new attention interface, where it can be plugged in various backends, because most of the signature is not enforced. We INFORM but do not ENFORCE. That way, the current system is a [minimal user api](#minimal-user-api).
 
-For a better _information_, we plan to use `python`features such as `Annoted` for example, to inform users of what we expect typically in an argument. That way, higher-level information could be included directly in the 
+For a better _information_, we plan to use `python`features such as `Annotated` for example, to inform users of what we expect typically in an argument. That way, higher-level information could be included directly in the type annotations, telling for instance the expected dimensions and contents of a tensor.
+
+## <a id="simpler-tensor-parallelism"></a> Simpler Tensor Parallelism
+
+We want to touch minimally to the modeling code, and only modify it when _architectural changes_ are involved. For instance, for tensor parallelism, we instead now specify a simple `tp_plan`.
+
+## <a id="layers-attentions-caches"></a> Layers, attentions and caches
+With th
 
-## Community Kernels
+## <a id="community-kernels"></a>Community Kernels
 
 The same principle extends to normalization, activation, and other hot paths. The model defines **semantics**; a kernel defines **how** to execute them faster. We annotate the module to borrow a community‑provided forward, keeping a [consistent public surface](#consistent-public-surface)
 
@@ -94,7 +100,7 @@ class GlmRMSNorm(nn.Module):
     ...
 ```
 
-Plus, this opened another angle of contribution for the community. People who are GPU whisperersYou can check on the [kernel community blog post](https://huggingface.co/blog/hello-hf-kernels) to learn more about it! 
+Plus, this opened another angle of contribution for the community. People who are GPU whisperers can check on the [kernel community blog post](https://huggingface.co/blog/hello-hf-kernels) to learn more about it! 
 
 ## The good modularity
 
@@ -201,7 +207,7 @@ I took again a similarity measure and looked at the existing graphs. The tool is
 
 ![Modular candidates analysis](static/modular_candidates.png)
 
-## <a id="encoders-ftw"></a> Encoders win !
+## <a id="encoders-ftw"></a> The neverending stories of encoder models.
 
 Models popularity speaks for itself! This is because the usage of encoders lies in embeddings obviously. So we have to keep the encoders part viable, usable, fine-tune-able. 
 
@@ -249,9 +255,9 @@ Adding a model to transformers means:
 - having it immediately available to the community
 - usable in vLLM, SGLang, and so on without additional code. 
 
-## Inner cooking: Cache allocator
+## Inner cooking: CUDA Warmup
 
-Having a clean _external_ API allows us to work on the true inner workings of transformers. One of the few recent additions was the _Cache pre-allocator_ which improved massively the loading footprint. 
+Having a clean _external_ API allows us to work on the true inner workings of transformers. One of the few recent additions was the _CUDA warmup_ via `caching_allocator_warmup` which improved massively the loading time by pre-allocating GPU memory to avoid malloc bottlenecks during model loading. 
 
 {{{fragment-memory-profiler}}}