big update

content/article.md

@@ -2,13 +2,24 @@
 ## Introduction 
 
 The `transformers` library, built with `PyTorch`, supports all state-of-the-art LLMs, many VLMs, task-specific vision language models, video models, audio models, table models, classical encoders, to a global count of almost 400 models. 
+
 The name of the library itself is mostly majority driven as many models are not even transformers architectures, like Mamba, Zamba, RWKV, and convolution-based models.
+
 Regardless, each of these is wrought by the research and engineering team that created them, then harmonized into a now famous interface, and callable with a simple `.from_pretrained` command.
+
 Inference works for all models, training is functional for most. The library is a foundation for many machine learning courses, cookbooks, and overall, several thousands other open-source libraries depend on it. All models are tested as part of a daily CI ensuring their preservation and reproducibility. Most importantly, it is _open-source_ and has been written by the community for a large part.
 This isn't really to brag but to set the stakes: what does it take to keep such a ship afloat, made of so many moving, unrelated parts? 
+
 The ML wave has not stopped, there's more and more models being added, at a steadily growing rate. `Transformers` is widely used, and we read the feedback that users post online. Whether it's about a function that had 300+ keyword arguments, duplicated code and helpers, and mentions of `Copied from ... ` everywhere, along with optimisation concerns. Text-only models are relatively tamed, but multimodal models remain to be harmonized.
 
-Here we will dissect what is the new design philosophy of transformers, as a continuation from the existing older [philosophy](https://huggingface.co/docs/transformers/en/philosophy) page, and an accompanying [blog post from 2022](https://huggingface.co/blog/transformers-design-philosophy) . Some time ago I dare not say how long, we discussed with transformers maintainers about the state of things. A lot of recent developments were satisfactory, but if we were only talking about these, self-congratulation would be the only goalpost. Reflecting on this philosophy now, as models pile up, is essential and will drive new developments. 
+Here we will dissect what is the new design philosophy of transformers, as a continuation from the existing older [philosophy](https://huggingface.co/docs/transformers/en/philosophy) page, and an accompanying [blog post from 2022](https://huggingface.co/blog/transformers-design-philosophy).
+
+More recently, and I recommend the read if it's not done yet, a blog post about [recent upgrades to transformers](https://huggingface.co/blog/faster-transformers) was written, explaining in particular what makes the library faster today.
+
+Some time ago I dare not say how long, we discussed with transformers maintainers about the state of features in transformers. A lot of recent developments were satisfactory, but if we were only talking about these, self-congratulation would be the only goalpost.
+
+Reflecting on this philosophy now, as models pile up, is essential and will drive new developments. 
+
 
 ### What you will learn
 
@@ -325,14 +336,15 @@ curl -X POST http://localhost:8000/v1/chat/completions \
 ```
 
 This provides an OpenAI-compatible API with features like continuous batching for better GPU utilization.
+
 ## Community reusability
  
 
 Adding a model to transformers means:
 - having it immediately available to the community
-- usable in vLLM, SGLang, and so on without additional code. 
+- usable in vLLM, SGLang, and so on without additional code.
 
-## Inner cooking: CUDA Warmup
+## Cooking faster CUDA warmups
 
 Having a clean _external_ API allows us to work on the true inner workings of transformers. One of the few recent additions was the _CUDA warmup_ via `caching_allocator_warmup` which improved massively the loading footprint by pre-allocating GPU memory to avoid malloc bottlenecks during model loading. 
 

dist/fragments/attention-visualizer.html

@@ -10,16 +10,16 @@
         <div style="display: grid; grid-template-columns: 1fr auto; gap: 1rem; align-items: start; margin-bottom: 1rem;">
             <div>
                 <label style="display: block; font-weight: 600; margin-bottom: 0.5rem; color: #374151;">Model:</label>
-                <select id=model-select style="width: 100%; padding: 0.5rem; border: 1px solid #d1d5db; border-radius: 6px; background: white;">
-                    <option value=openai-community/gpt2>openai-community/gpt2</option>
-                    <option value=google/gemma-2-2b>google/gemma-2-2b</option>
-                    <option value=microsoft/DialoGPT-small>microsoft/DialoGPT-small</option>
+                <select id="model-select" style="width: 100%; padding: 0.5rem; border: 1px solid #d1d5db; border-radius: 6px; background: white;">
+                    <option value="openai-community/gpt2">openai-community/gpt2</option>
+                    <option value="google/gemma-2-2b">google/gemma-2-2b</option>
+                    <option value="microsoft/DialoGPT-small">microsoft/DialoGPT-small</option>
                 </select>
             </div>
             
             <div>
                 <label style="display: block; font-weight: 600; margin-bottom: 0.5rem; color: #374151;">Action:</label>
-                <button id=visualize-btn style="padding: 0.5rem 1rem; background: #3b82f6; color: white; border: none; border-radius: 6px; cursor: pointer; font-weight: 500;">
+                <button id="visualize-btn" style="padding: 0.5rem 1rem; background: #3b82f6; color: white; border: none; border-radius: 6px; cursor: pointer; font-weight: 500;">
                     🚀 Visualize
                 </button>
             </div>
@@ -27,10 +27,13 @@
         
         <div style="margin-bottom: 1rem;">
             <label style="display: block; font-weight: 600; margin-bottom: 0.5rem; color: #374151;">Prompt:</label>
-            <textarea id=prompt-input style="width: 100%; padding: 0.75rem; border: 1px solid #d1d5db; border-radius: 6px; resize: vertical; font-family: monospace; font-size: 0.9em;" rows=3 placeholder="You are an assistant. Make sure you print me."></textarea>
+            <textarea id="prompt-input" 
+                      style="width: 100%; padding: 0.75rem; border: 1px solid #d1d5db; border-radius: 6px; resize: vertical; font-family: monospace; font-size: 0.9em;" 
+                      rows="3" 
+                      placeholder="You are an assistant. Make sure you print me."></textarea>
         </div>
         
-        <div id=attention-output style="min-height: 200px; background: #f8f9fa; border: 1px solid #e9ecef; border-radius: 6px; padding: 1rem;">
+        <div id="attention-output" style="min-height: 200px; background: #f8f9fa; border: 1px solid #e9ecef; border-radius: 6px; padding: 1rem;">
             <div style="text-align: center; color: #6c757d; font-style: italic;">
                 Click "Visualize" to generate attention visualization
             </div>
@@ -43,4 +46,57 @@
     </div>
 </div>
 
-<script>document.addEventListener("DOMContentLoaded",function(){let e=document.getElementById("model-select"),t=document.getElementById("prompt-input"),n=document.getElementById("visualize-btn"),i=document.getElementById("attention-output");t.value="You are an assistant. Make sure you print me.",n.addEventListener("click",function(){let o=e.value,r=t.value.trim();if(!r){i.innerHTML='<div style="color: #e53e3e;">Please enter a prompt</div>';return}n.disabled=!0,n.textContent="Processing...",i.innerHTML='<div style="text-align: center; color: #6c757d;"><em>Generating attention visualization...</em></div>',setTimeout(()=>{let e=r.split(" ").slice(0,8),t='<div style="margin-bottom: 1rem;"><strong>Model:</strong> '+o+"</div>";t+='<div style="margin-bottom: 1rem;"><strong>Tokens:</strong> '+e.join(" • ")+'</div><div><strong>Attention Matrix (Layer 0, Head 0):</strong></div><table style="margin-top: 0.5rem; border-collapse: collapse; font-family: monospace; font-size: 0.8em;">';for(let n=0;n<e.length;n++){t+="<tr>";for(let n=0;n<e.length;n++){let e=Math.random(),n=`rgba(59, 130, 246, ${e})`;t+=`<td style="border: 1px solid #ddd; padding: 4px; background: ${n}; text-align: center; min-width: 40px;">${e.toFixed(2)}</td>`}t+="</tr>"}i.innerHTML=t+='</table><div style="margin-top: 1rem; font-size: 0.9em; color: #6c757d;"><em>Darker blue = higher attention weight</em></div>',n.disabled=!1,n.textContent="\uD83D\uDE80 Visualize"},2e3)})})</script>
+<script>
+document.addEventListener('DOMContentLoaded', function() {
+    const modelSelect = document.getElementById('model-select');
+    const promptInput = document.getElementById('prompt-input');
+    const visualizeBtn = document.getElementById('visualize-btn');
+    const output = document.getElementById('attention-output');
+    
+    // Set default prompt
+    promptInput.value = "You are an assistant. Make sure you print me.";
+    
+    visualizeBtn.addEventListener('click', function() {
+        const model = modelSelect.value;
+        const prompt = promptInput.value.trim();
+        
+        if (!prompt) {
+            output.innerHTML = '<div style="color: #e53e3e;">Please enter a prompt</div>';
+            return;
+        }
+        
+        // Show loading state
+        visualizeBtn.disabled = true;
+        visualizeBtn.textContent = 'Processing...';
+        output.innerHTML = '<div style="text-align: center; color: #6c757d;"><em>Generating attention visualization...</em></div>';
+        
+        // Simulate processing time
+        setTimeout(() => {
+            // Generate mock attention visualization
+            const tokens = prompt.split(' ').slice(0, 8); // Limit tokens for demo
+            let html = '<div style="margin-bottom: 1rem;"><strong>Model:</strong> ' + model + '</div>';
+            html += '<div style="margin-bottom: 1rem;"><strong>Tokens:</strong> ' + tokens.join(' • ') + '</div>';
+            html += '<div><strong>Attention Matrix (Layer 0, Head 0):</strong></div>';
+            html += '<table style="margin-top: 0.5rem; border-collapse: collapse; font-family: monospace; font-size: 0.8em;">';
+            
+            // Generate attention matrix visualization
+            for (let i = 0; i < tokens.length; i++) {
+                html += '<tr>';
+                for (let j = 0; j < tokens.length; j++) {
+                    const attention = Math.random();
+                    const opacity = attention;
+                    const color = `rgba(59, 130, 246, ${opacity})`;
+                    html += `<td style="border: 1px solid #ddd; padding: 4px; background: ${color}; text-align: center; min-width: 40px;">${attention.toFixed(2)}</td>`;
+                }
+                html += '</tr>';
+            }
+            html += '</table>';
+            html += '<div style="margin-top: 1rem; font-size: 0.9em; color: #6c757d;"><em>Darker blue = higher attention weight</em></div>';
+            
+            output.innerHTML = html;
+            visualizeBtn.disabled = false;
+            visualizeBtn.textContent = '🚀 Visualize';
+        }, 2000);
+    });
+});
+</script>

dist/fragments/d3-graph.html

@@ -1,11 +1,11 @@
-<div class=interactive-demo>
-    <div class=demo-header>
+<div class="interactive-demo">
+    <div class="demo-header">
         <h3>🔗 Model Dependency Graph</h3>
     </div>
-    <div class=demo-content>
-        <iframe src=static/d3_dependency_graph.html width=100% height=600px frameborder=0 style="border-radius: 8px; background: white;"></iframe>
+    <div class="demo-content">
+        <iframe src="static/d3_dependency_graph.html" width="100%" height="600px" frameborder="0" style="border-radius: 8px; background: white;"></iframe>
     </div>
-    <div class=demo-footer>
+    <div class="demo-footer">
         Interactive dependency graph showing real relationships between Transformers models. 🟡 Base models (HuggingFace logo), 🔵 Derived modular models. Click and drag to explore!
     </div>
 </div>

dist/fragments/glm-compare.html

@@ -1,9 +1,9 @@
-<div class=code-compare style="display: grid; grid-template-columns: 1fr 1fr; gap: 1rem; margin: 1.5rem 0;">
-    <div class=code-column style="border: 1px solid #e2e8f0; border-radius: 8px; overflow: hidden;">
-        <div class=code-header style="background: #f8f9fa; padding: 0.75rem 1rem; font-weight: 600; color: #495057; border-bottom: 1px solid #e2e8f0;">
+<div class="code-compare" style="display: grid; grid-template-columns: 1fr 1fr; gap: 1rem; margin: 1.5rem 0;">
+    <div class="code-column" style="border: 1px solid #e2e8f0; border-radius: 8px; overflow: hidden;">
+        <div class="code-header" style="background: #f8f9fa; padding: 0.75rem 1rem; font-weight: 600; color: #495057; border-bottom: 1px solid #e2e8f0;">
             modular_glm.py
         </div>
-        <pre style="margin: 0; padding: 1rem; background: #ffffff; overflow-x: auto; font-size: 0.9em;"><code class=language-python>class GlmMLP(Phi3MLP):
+        <pre style="margin: 0; padding: 1rem; background: #ffffff; overflow-x: auto; font-size: 0.9em;"><code class="language-python">class GlmMLP(Phi3MLP):
     pass
 
 class GlmAttention(LlamaAttention):
@@ -19,11 +19,11 @@ class GlmForCausalLM(LlamaForCausalLM):
     pass</code></pre>
     </div>
     
-    <div class=code-column style="border: 1px solid #e2e8f0; border-radius: 8px; overflow: hidden;">
-        <div class=code-header style="background: #f8f9fa; padding: 0.75rem 1rem; font-weight: 600; color: #495057; border-bottom: 1px solid #e2e8f0;">
+    <div class="code-column" style="border: 1px solid #e2e8f0; border-radius: 8px; overflow: hidden;">
+        <div class="code-header" style="background: #f8f9fa; padding: 0.75rem 1rem; font-weight: 600; color: #495057; border-bottom: 1px solid #e2e8f0;">
             modeling_glm.py (auto-expanded)
         </div>
-        <pre style="margin: 0; padding: 1rem; background: #ffffff; overflow-x: auto; font-size: 0.9em; max-height: 400px;"><code class=language-python>class GlmMLP(nn.Module):
+        <pre style="margin: 0; padding: 1rem; background: #ffffff; overflow-x: auto; font-size: 0.9em; max-height: 400px;"><code class="language-python">class GlmMLP(nn.Module):
     def __init__(self, config):
         super().__init__()
         self.config = config

dist/fragments/modular-growth.html

@@ -0,0 +1,18 @@
+<section id="modular-growth-space" class="l-body">
+  <h2>Modular growth (interactive)</h2>
+  <p class="l-page">Explore live graphs and metrics from the accompanying Space.</p>
+  <div style="position:relative;padding-top:62.5%;border:1px solid rgba(255,255,255,.08);border-radius:12px;overflow:hidden">
+    <iframe
+      src="https://molbap-transformers-modular-refactor.hf.space"
+      title="Transformers Modular Refactor Space"
+      style="position:absolute;inset:0;border:0;width:100%;height:100%;"
+      loading="lazy"
+      referrerpolicy="no-referrer-when-downgrade"
+      allow="clipboard-read; clipboard-write; fullscreen; autoplay"
+    ></iframe>
+  </div>
+  <p class="l-page" style="margin-top:.6rem">
+    Open in a new tab: <a href="https://huggingface.co/spaces/Molbap/transformers-modular-refactor" target="_blank" rel="noopener">Space README & details</a>.
+  </p>
+</section>
+

dist/fragments/terminal.html

@@ -2,21 +2,42 @@
     <h4 style="margin-top: 0; color: #495057;">Interactive Terminal</h4>
     <div style="background: #2d3748; color: #e2e8f0; padding: 1rem; border-radius: 6px; font-family: 'Consolas', 'Monaco', monospace;">
         <div style="margin-bottom: 1rem;">
-            <input type=text id=terminal-input placeholder="python -c 'import torch; print(torch.__version__)'" style="width: calc(100% - 80px); padding: 0.5rem; background: #1a202c; border: 1px solid #4a5568; color: #e2e8f0; border-radius: 4px;">
-            <button id=terminal-run style="width: 70px; padding: 0.5rem; margin-left: 8px; background: #3182ce; color: white; border: none; border-radius: 4px; cursor: pointer;">Run</button>
+            <input type="text" 
+                   id="terminal-input" 
+                   placeholder="python -c 'import torch; print(torch.__version__)'" 
+                   style="width: calc(100% - 80px); padding: 0.5rem; background: #1a202c; border: 1px solid #4a5568; color: #e2e8f0; border-radius: 4px;">
+            <button id="terminal-run" 
+                    style="width: 70px; padding: 0.5rem; margin-left: 8px; background: #3182ce; color: white; border: none; border-radius: 4px; cursor: pointer;">Run</button>
         </div>
-        <pre id=terminal-output style="background: #1a202c; padding: 1rem; border-radius: 4px; min-height: 100px; margin: 0; overflow-x: auto;">$ Ready to run commands...</pre>
+        <pre id="terminal-output" style="background: #1a202c; padding: 1rem; border-radius: 4px; min-height: 100px; margin: 0; overflow-x: auto;">$ Ready to run commands...</pre>
     </div>
     <p style="font-size: 0.9em; color: #6c757d; margin-top: 0.5rem;">
         <em>Note: This is a simulated terminal. In the original Gradio app, this would execute real Python commands with proper security restrictions.</em>
     </p>
 </div>
 
-<script>document.addEventListener("DOMContentLoaded",function(){let e=document.getElementById("terminal-input"),t=document.getElementById("terminal-run"),n=document.getElementById("terminal-output");function o(){let t=e.value.trim();t&&(n.textContent=`$ ${t}
-Simulated output for: ${t}
-
-This would execute the command in the original app.
-Example outputs:
-- torch version: 2.0.1+cu117
-- import checks: Success
-- memory info: Available`)}t.addEventListener("click",o),e.addEventListener("keypress",function(e){"Enter"===e.key&&o()})})</script>
+<script>
+document.addEventListener('DOMContentLoaded', function() {
+    const input = document.getElementById('terminal-input');
+    const button = document.getElementById('terminal-run');
+    const output = document.getElementById('terminal-output');
+    
+    function runCommand() {
+        const command = input.value.trim();
+        if (!command) return;
+        
+        // Simulate command execution
+        output.textContent = `$ ${command}\nSimulated output for: ${command}\n\n` + 
+                           `This would execute the command in the original app.\n` +
+                           `Example outputs:\n` +
+                           `- torch version: 2.0.1+cu117\n` +
+                           `- import checks: Success\n` +
+                           `- memory info: Available`;
+    }
+    
+    button.addEventListener('click', runCommand);
+    input.addEventListener('keypress', function(e) {
+        if (e.key === 'Enter') runCommand();
+    });
+});
+</script>

dist/fragments/tp-plan.html

@@ -0,0 +1,24 @@
+<pre><code class="language-python"># In the model's config (example: ERNIE 4.5-style decoder blocks)
+base_model_tp_plan = {
+    "layers.*.self_attn.q_proj": "colwise",
+    "layers.*.self_attn.k_proj": "colwise",
+    "layers.*.self_attn.v_proj": "colwise",
+    "layers.*.self_attn.o_proj": "rowwise",
+    "layers.*.mlp.gate_proj": "colwise",
+    "layers.*.mlp.up_proj":   "colwise",
+    "layers.*.mlp.down_proj": "rowwise",
+}
+
+# Runtime
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "your/model-or-local-checkpoint"
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    dtype=torch.bfloat16,
+    tp_plan=base_model_tp_plan,   # <-- plan defined above
+)
+tok = AutoTokenizer.from_pretrained(model_id)
+inputs = tok("Hello", return_tensors="pt").to(model.device)
+out = model(**inputs)</code></pre>

dist/fragments/warmup_demo.html

@@ -0,0 +1,398 @@
+<style>
+    .warmup-demo body {
+        font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
+        margin: 0;
+        padding: 20px;
+        background-color: #f5f5f5;
+    }
+
+    .warmup-demo .container {
+        max-width: 1200px;
+        margin: 0 auto;
+        background: white;
+        border-radius: 12px;
+        padding: 30px;
+        box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
+    }
+
+    .warmup-demo h1 {
+        text-align: center;
+        color: #333;
+        margin-bottom: 10px;
+    }
+
+    .warmup-demo .subtitle {
+        text-align: center;
+        color: #666;
+        margin-bottom: 30px;
+        font-size: 16px;
+    }
+
+    .warmup-demo .demo-container {
+        display: flex;
+        gap: 40px;
+        margin-bottom: 30px;
+    }
+
+    .warmup-demo .side {
+        flex: 1;
+        border: 2px solid #ddd;
+        border-radius: 8px;
+        padding: 20px;
+        background: #fafafa;
+    }
+
+    .warmup-demo .side h2 {
+        text-align: center;
+        margin-top: 0;
+        color: #333;
+    }
+
+    .warmup-demo .no-warmup h2 {
+        color: #d63384;
+    }
+
+    .warmup-demo .with-warmup h2 {
+        color: #198754;
+    }
+
+    .warmup-demo .memory-area {
+        height: 400px;
+        border: 2px dashed #ccc;
+        border-radius: 6px;
+        padding: 10px;
+        margin: 20px 0;
+        background: #fff;
+        position: relative;
+        overflow: hidden;
+    }
+
+    .warmup-demo .layer-box {
+        width: 80px;
+        height: 30px;
+        border: 2px solid #666;
+        border-radius: 4px;
+        margin: 3px;
+        display: inline-block;
+        position: relative;
+        background: #fff;
+        transition: all 0.3s ease;
+    }
+
+    .warmup-demo .layer-box.allocating {
+        background: #e9ecef;
+        border-color: #adb5bd;
+    }
+
+    .warmup-demo .layer-box.allocating::after {
+        content: "malloc";
+        position: absolute;
+        top: 50%;
+        left: 50%;
+        transform: translate(-50%, -50%);
+        font-size: 10px;
+        color: #666;
+        font-weight: bold;
+    }
+
+    .warmup-demo .layer-box.loaded {
+        background: #d1e7dd;
+        border-color: #198754;
+    }
+
+    .warmup-demo .layer-box.loaded::after {
+        content: "data";
+        position: absolute;
+        top: 50%;
+        left: 50%;
+        transform: translate(-50%, -50%);
+        font-size: 10px;
+        color: #198754;
+        font-weight: bold;
+    }
+
+    .warmup-demo .warmup-container {
+        width: 100%;
+        height: 60px;
+        border: 3px solid #666;
+        border-radius: 6px;
+        margin-bottom: 20px;
+        background: #fff;
+        position: relative;
+        overflow: hidden;
+    }
+
+    .warmup-demo .warmup-container.allocated {
+        border-color: #0d6efd;
+        background: #e7f1ff;
+    }
+
+    .warmup-demo .warmup-container::before {
+        content: "Pre-allocated Memory Pool";
+        position: absolute;
+        top: 50%;
+        left: 50%;
+        transform: translate(-50%, -50%);
+        font-size: 14px;
+        color: #666;
+        font-weight: bold;
+        z-index: 1;
+    }
+
+    .warmup-demo .warmup-container.allocated::before {
+        color: #0d6efd;
+    }
+
+    .warmup-demo .warmup-fill {
+        height: 100%;
+        background: linear-gradient(90deg, #198754, #20c997);
+        width: 0%;
+        transition: width 0.5s ease;
+        border-radius: 3px;
+        position: relative;
+        z-index: 2;
+    }
+
+    .warmup-demo .warmup-fill::after {
+        content: "Layer Data Loading";
+        position: absolute;
+        top: 50%;
+        left: 50%;
+        transform: translate(-50%, -50%);
+        font-size: 12px;
+        color: white;
+        font-weight: bold;
+        white-space: nowrap;
+    }
+
+    .warmup-demo .timing {
+        text-align: center;
+        font-size: 24px;
+        font-weight: bold;
+        margin: 15px 0;
+        min-height: 30px;
+    }
+
+    .warmup-demo .no-warmup .timing {
+        color: #d63384;
+    }
+
+    .warmup-demo .with-warmup .timing {
+        color: #198754;
+    }
+
+    .warmup-demo .controls {
+        text-align: center;
+        margin: 30px 0;
+    }
+
+    .warmup-demo .btn {
+        background: #0d6efd;
+        color: white;
+        border: none;
+        padding: 12px 24px;
+        border-radius: 6px;
+        font-size: 16px;
+        cursor: pointer;
+        margin: 0 10px;
+        transition: background 0.3s ease;
+    }
+
+    .warmup-demo .btn:hover {
+        background: #0b5ed7;
+    }
+
+    .warmup-demo .btn:disabled {
+        background: #6c757d;
+        cursor: not-allowed;
+    }
+
+    .warmup-demo .description {
+        background: #f8f9fa;
+        padding: 15px;
+        border-radius: 6px;
+        margin-top: 15px;
+        font-size: 14px;
+        line-height: 1.5;
+    }
+
+    .warmup-demo .phase-indicator {
+        font-size: 14px;
+        color: #666;
+        text-align: center;
+        margin-top: 10px;
+        min-height: 20px;
+    }
+
+    .warmup-demo .layer-counter {
+        text-align: center;
+        font-size: 16px;
+        color: #495057;
+        margin: 10px 0;
+    }
+</style>
+
+<div class="warmup-demo">
+    <div class="container">
+        <p class="subtitle">Mem allocation patterns during model loading</p>
+
+        <div class="controls">
+            <button class="btn" id="startBtn" onclick="startDemo()">Start Animation</button>
+            <button class="btn" id="resetBtn" onclick="resetDemo()">Reset</button>
+        </div>
+
+        <div class="demo-container">
+            <div class="side no-warmup">
+                <h2 data-no-toc>❌ Without Warmup</h2>
+                <div class="timing" id="noWarmupTime">0.00s</div>
+                <div class="layer-counter" id="noWarmupCounter">Layers loaded: 0/10</div>
+                <div class="phase-indicator" id="noWarmupPhase"></div>
+                <div class="memory-area" id="noWarmupArea"></div>
+                <div class="description">
+                    <strong>Individual Allocations:</strong><br>
+                    Each model layer triggers a separate cudaMalloc() call, creating memory fragmentation and allocation overhead.
+                    <br><br>
+                    📦 <strong>Grey "malloc"</strong> = Memory allocation overhead<br>
+                    ✅ <strong>Green "data"</strong> = Actual layer data loading
+                </div>
+            </div>
+
+            <div class="side with-warmup">
+                <h2 data-no-toc>✅ With Warmup</h2>
+                <div class="timing" id="warmupTime">0.00s</div>
+                <div class="layer-counter" id="warmupCounter">Layers loaded: 0/10</div>
+                <div class="phase-indicator" id="warmupPhase"></div>
+                <div class="memory-area" id="warmupArea">
+                    <div class="warmup-container" id="warmupContainer">
+                        <div class="warmup-fill" id="warmupFill"></div>
+                    </div>
+                    <div id="warmupLayers"></div>
+                </div>
+                <div class="description">
+                    <strong>Pre-allocated Pool:</strong><br>
+                    The warmup function calculates total memory needed and makes ONE large allocation. Subsequent layers load directly into this pool, eliminating malloc overhead.
+                    <br><br>
+                    🔵 <strong>Blue container</strong> = Single large malloc (warmup)<br>
+                    🟢 <strong>Green progress bar</strong> = Layer data loading (no malloc needed)
+                </div>
+            </div>
+        </div>
+    </div>
+</div>
+
+<script>
+    let animationSpeed = 1 / 2.4;
+    let isRunning = false;
+    const totalLayers = 10;
+
+    function startDemo() {
+        if (isRunning) return;
+        isRunning = true;
+
+        document.getElementById('startBtn').disabled = true;
+        document.getElementById('resetBtn').disabled = true;
+
+        Promise.all([
+            animateNoWarmup(),
+            animateWithWarmup()
+        ]).then(() => {
+            isRunning = false;
+            document.getElementById('startBtn').disabled = false;
+            document.getElementById('resetBtn').disabled = false;
+        });
+    }
+
+    function resetDemo() {
+        if (isRunning) return;
+
+        document.getElementById('noWarmupArea').innerHTML = '';
+        document.getElementById('warmupLayers').innerHTML = '';
+        document.getElementById('warmupFill').style.width = '0%';
+        document.getElementById('warmupContainer').classList.remove('allocated');
+
+        document.getElementById('noWarmupTime').textContent = '0.00s';
+        document.getElementById('warmupTime').textContent = '0.00s';
+
+        document.getElementById('noWarmupCounter').textContent = 'Layers loaded: 0/10';
+        document.getElementById('warmupCounter').textContent = 'Layers loaded: 0/10';
+
+        document.getElementById('noWarmupPhase').textContent = '';
+        document.getElementById('warmupPhase').textContent = '';
+    }
+
+    async function animateNoWarmup() {
+        const container = document.getElementById('noWarmupArea');
+        const timeEl = document.getElementById('noWarmupTime');
+        const counterEl = document.getElementById('noWarmupCounter');
+        const phaseEl = document.getElementById('noWarmupPhase');
+
+        let currentTime = 0;
+        const baseDelay = 200 / animationSpeed;
+
+        phaseEl.textContent = 'Loading model layers...';
+
+        for (let i = 0; i < totalLayers; i++) {
+            const layerBox = document.createElement('div');
+            layerBox.className = 'layer-box';
+            container.appendChild(layerBox);
+
+            await sleep(baseDelay * 0.3);
+            layerBox.classList.add('allocating');
+            currentTime += 0.08;
+            timeEl.textContent = currentTime.toFixed(2) + 's';
+
+            await sleep(baseDelay * 0.7);
+            layerBox.classList.remove('allocating');
+            layerBox.classList.add('loaded');
+            currentTime += 0.12;
+            timeEl.textContent = currentTime.toFixed(2) + 's';
+
+            counterEl.textContent = `Layers loaded: ${i + 1}/${totalLayers}`;
+        }
+
+        phaseEl.textContent = 'Complete!';
+    }
+
+    async function animateWithWarmup() {
+        const container = document.getElementById('warmupLayers');
+        const timeEl = document.getElementById('warmupTime');
+        const counterEl = document.getElementById('warmupCounter');
+        const phaseEl = document.getElementById('warmupPhase');
+        const warmupContainer = document.getElementById('warmupContainer');
+        const warmupFill = document.getElementById('warmupFill');
+
+        let currentTime = 0;
+        const baseDelay = 200 / animationSpeed;
+
+        phaseEl.textContent = 'Warming up allocator...';
+        await sleep(baseDelay * 2);
+        warmupContainer.classList.add('allocated');
+        currentTime += 0.3;
+        timeEl.textContent = currentTime.toFixed(2) + 's';
+
+        phaseEl.textContent = 'Loading model layers...';
+
+        for (let i = 0; i < totalLayers; i++) {
+            const layerBox = document.createElement('div');
+            layerBox.className = 'layer-box loaded';
+            layerBox.style.width = '40px';
+            layerBox.style.height = '20px';
+            container.appendChild(layerBox);
+
+            const progress = ((i + 1) / totalLayers) * 100;
+            warmupFill.style.width = progress + '%';
+
+            await sleep(baseDelay * 0.5);
+            currentTime += 0.08;
+            timeEl.textContent = currentTime.toFixed(2) + 's';
+
+            counterEl.textContent = `Layers loaded: ${i + 1}/${totalLayers}`;
+        }
+
+        phaseEl.textContent = 'Complete!';
+    }
+
+    function sleep(ms) {
+        return new Promise(resolve => setTimeout(resolve, ms));
+    }
+</script>

dist/index.html

@@ -8,21 +8,21 @@
     <script src="https://d3js.org/d3.v7.min.js"></script>
     <meta name="viewport" content="width=device-width, initial-scale=1">
     <meta charset="utf8">
-    <title>Scaling insanity: maintaining hundreds of model definitions</title>
+    <title>Scaling Insanity: maintaining hundreds of model definitions</title>
     <link rel="stylesheet" href="style.css">
     <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/prism/1.29.0/themes/prism.min.css">
 </head>
 <body>
     <d-front-matter>
         <script id='distill-front-matter' type="text/json">{
-    "title": "Scaling insanity: maintaining hundreds of model definitions",
+    "title": "Scaling Insanity: maintaining hundreds of model definitions",
     "description": "A peek into software engineering for the transformers library",
     "published": "Aug 21, 2025",
     "authors": [{"author": "Pablo Montalvo", "authorURL": "https://huggingface.co/Molbap"}]
   }</script>
     </d-front-matter>
     <d-title>
-        <h1>Scaling insanity: maintaining hundreds of model definitions</h1>
+        <h1>Scaling Insanity: maintaining hundreds of model definitions</h1>
         <p>A peek into software engineering for the transformers library</p>
     </d-title>
     <d-byline></d-byline>
@@ -48,13 +48,16 @@
             </nav>
         </d-contents>
         <h2>Introduction</h2>
-<p>The <code>transformers</code> library, built with <code>PyTorch</code>, supports all state-of-the-art LLMs, many VLMs, task-specific vision language models, video models, audio models, table models, classical encoders, to a global count of almost 400 models.
-The name of the library itself is mostly majority driven as many models are not even transformers architectures, like Mamba, Zamba, RWKV, and convolution-based models.
-Regardless, each of these is wrought by the research and engineering team that created them, then harmonized into a now famous interface, and callable with a simple <code>.from_pretrained</code> command.
-Inference works for all models, training is functional for most. The library is a foundation for many machine learning courses, cookbooks, and overall, several thousands other open-source libraries depend on it. All models are tested as part of a daily CI ensuring their preservation and reproducibility. Most importantly, it is <em>open-source</em> and has been written by the community for a large part.
-This isn’t really to brag but to set the stakes: what does it take to keep such a ship afloat, made of so many moving, unrelated parts?
-The ML wave has not stopped, there’s more and more models being added, at a steadily growing rate. <code>Transformers</code> is widely used, and we read the feedback that users post online. Whether it’s about a function that had 300+ keyword arguments, duplicated code and helpers, and mentions of <code>Copied from ... </code> everywhere, along with optimisation concerns. Text-only models are relatively tamed, but multimodal models remain to be harmonized.</p>
-<p>Here we will dissect what is the new design philosophy of transformers, as a continuation from the existing older <a href="https://huggingface.co/docs/transformers/en/philosophy">philosophy</a> page, and an accompanying <a href="https://huggingface.co/blog/transformers-design-philosophy">blog post from 2022</a> . Some time ago I dare not say how long, we discussed with transformers maintainers about the state of things. A lot of recent developments were satisfactory, but if we were only talking about these, self-congratulation would be the only goalpost. Reflecting on this philosophy now, as models pile up, is essential and will drive new developments.</p>
+<p>The <code>transformers</code> library, built with <code>PyTorch</code>, supports all state-of-the-art LLMs, many VLMs, task-specific vision language models, video models, audio models, table models, classical encoders, to a global count of almost 400 models.</p>
+<p>The name of the library itself is mostly majority driven as many models are not even transformers architectures, like Mamba, Zamba, RWKV, and convolution-based models.</p>
+<p>Regardless, each of these is wrought by the research and engineering team that created them, then harmonized into a now famous interface, and callable with a simple <code>.from_pretrained</code> command.</p>
+<p>Inference works for all models, training is functional for most. The library is a foundation for many machine learning courses, cookbooks, and overall, several thousands other open-source libraries depend on it. All models are tested as part of a daily CI ensuring their preservation and reproducibility. Most importantly, it is <em>open-source</em> and has been written by the community for a large part.
+This isn’t really to brag but to set the stakes: what does it take to keep such a ship afloat, made of so many moving, unrelated parts?</p>
+<p>The ML wave has not stopped, there’s more and more models being added, at a steadily growing rate. <code>Transformers</code> is widely used, and we read the feedback that users post online. Whether it��s about a function that had 300+ keyword arguments, duplicated code and helpers, and mentions of <code>Copied from ... </code> everywhere, along with optimisation concerns. Text-only models are relatively tamed, but multimodal models remain to be harmonized.</p>
+<p>Here we will dissect what is the new design philosophy of transformers, as a continuation from the existing older <a href="https://huggingface.co/docs/transformers/en/philosophy">philosophy</a> page, and an accompanying <a href="https://huggingface.co/blog/transformers-design-philosophy">blog post from 2022</a>.</p>
+<p>More recently, and I recommend the read if it’s not done yet, a blog post about <a href="https://huggingface.co/blog/faster-transformers">recent upgrades to transformers</a> was written, explaining in particular what makes the library faster today.</p>
+<p>Some time ago I dare not say how long, we discussed with transformers maintainers about the state of features in transformers. A lot of recent developments were satisfactory, but if we were only talking about these, self-congratulation would be the only goalpost.</p>
+<p>Reflecting on this philosophy now, as models pile up, is essential and will drive new developments.</p>
 <h3>What you will learn</h3>
 <p>Every reader, whether an OSS maintainer, power user, or casual fine-tuner, will walk away knowing how to reason about the <code>transformers</code> code base, how to use it better, how to meaningfully contribute to it.
 This will also showcase new features you might have missed so you’ll be up-to-date.</p>
@@ -113,7 +116,7 @@ This will also showcase new features you might have missed so you’ll be up-to-
 </li>  
 <li class="tenet">
 <a id="modular-toolbox"></a>
-<strong>Modular Toolbox (Not Framework)</strong>
+<strong>Modular Toolbox (Not A Framework)</strong>
 <p>We ARE a toolbox. What we are not is a framework: you should not be FORCED to rewrite every modeling, but it is <em>better</em> for your model to be able to inherit from PreTrainedModel and have enabled TensorParallel, from_pretrained, sharding, push_to_hub, loss, as well as PEFT/TRL/SGLang/vLLM.</p>
 <em>This is the largest change. Provide tools and utilities, but don't force users into a rigid framework.</em>
 </li>
@@ -297,6 +300,35 @@ if self.config._attn_implementation != &quot;eager&quot;:
 <p>For better <em>information</em>, we plan to use <code>python</code> features such as <code>Annotated</code> for example, to inform users of what we expect typically in an argument. That way, higher-level information could be included directly in the type annotations.</p>
 <h2><a id="simpler-tensor-parallelism"></a> Simpler Tensor Parallelism</h2>
 <p>We want to touch minimally to the modeling code, and only modify it when <em>architectural changes</em> are involved. For instance, for tensor parallelism, we instead now specify a simple <code>tp_plan</code>.</p>
+<p>It is written once in the config and passed to <code>.from_pretrained()</code>.</p>
+<p>The plan maps module name patterns to partitioning strategies. Strategies are resolved by the internal <code>ParallelInterface</code>, which wires to sharding implementations <code>ColwiseParallel</code>, <code>RowwiseParallel</code>, packed variants, and so on.</p>
+<p><pre><code class="language-python"># In the model's config (example: ERNIE 4.5-style decoder blocks)
+base_model_tp_plan = {
+    "layers.*.self_attn.q_proj": "colwise",
+    "layers.*.self_attn.k_proj": "colwise",
+    "layers.*.self_attn.v_proj": "colwise",
+    "layers.*.self_attn.o_proj": "rowwise",
+    "layers.*.mlp.gate_proj": "colwise",
+    "layers.*.mlp.up_proj":   "colwise",
+    "layers.*.mlp.down_proj": "rowwise",
+}
+
+# Runtime
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "your/model-or-local-checkpoint"
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    dtype=torch.bfloat16,
+    tp_plan=base_model_tp_plan,   # <-- plan defined above
+)
+tok = AutoTokenizer.from_pretrained(model_id)
+inputs = tok("Hello", return_tensors="pt").to(model.device)
+out = model(**inputs)</code></pre></p>
+<p>Which allows a user to run with multiple processes per node, e.g. 4 GPUs:</p>
+<p><code>torchrun --nproc-per-node 4 demo.py</code></p>
+<p>Semantics stay in the model (a Linear stays a Linear), distribution is orthogonal and declared via strings: “colwise” splits columns of weights/bias across ranks; “rowwise” splits rows; packed variants shard fused weights; The mapping keys accept glob patterns like <code>layers.*.mlp.down_proj</code> to target repeated submodules.</p>
 <h2><a id="layers-attentions-caches"></a> Layers, attentions and caches</h2>
 <p>With th</p>
 <h2><a id="community-kernels"></a>Community Kernels</h2>
@@ -331,7 +363,6 @@ Radically different architectures such as mamba have spawned their own dependenc
 </div>
 
 </p>
-<p><img src="static/graph_modular_related_models.png" alt="Graph showing modular related models"></p>
 <p>But there is no similar miracle for VLMs across the board.
 As you can see, there is a small DETR island, a little llava pocket, and so on, but it’s not comparable to the centrality observed.</p>
 <p>One problem is, this is only for <code>modular</code> models. Several models do NOT have a modular file. In other words, we have a big “hidden space here.”</p>
@@ -415,7 +446,7 @@ Example outputs:
 <h2>Modularity candidates</h2>
 <p>So the question abounds naturally: How can we modularize more?
 I took again a similarity measure and looked at the existing graphs. The tool is available on this <a href="https://huggingface.co/spaces/Molbap/transformers-modular-refactor">ZeroGPU-enabled Space</a>. It scans the whole transformers repository, and outputs a graph of candidates across models, using either a Jaccard similarity index (simple) or a SentenceTransformers embedding model. It is understandable that <a href="#encoders-ftw">encoder models still have a lion’s share of the game.</a> See also <a href="https://huggingface.co/blog/train-sparse-encoder">Tom Aarsen and Arhur Bresnu’s great blog post on the topic of sparse embeddings.</a>.</p>
-<p><img src="static/modular_candidates.png" alt="Modular candidates analysis"></p>
+<p></p>
 <h2><a id="encoders-ftw"></a> The neverending stories of encoder models.</h2>
 <p>Models popularity speaks for itself! This is because the usage of encoders lies in embeddings obviously. So we have to keep the encoders part viable, usable, fine-tune-able.</p>
 <p><img src="static/popular_models_barplot.png" alt="Popular models bar plot"></p>
@@ -500,32 +531,61 @@ curl -X POST http://localhost:8000/v1/chat/completions \
 <li>having it immediately available to the community</li>
 <li>usable in vLLM, SGLang, and so on without additional code.</li>
 </ul>
-<p>## Inner cooking: CUDA Warmup</p>
+<h2>Cooking faster CUDA warmups</h2>
 <p>Having a clean <em>external</em> API allows us to work on the true inner workings of transformers. One of the few recent additions was the <em>CUDA warmup</em> via <code>caching_allocator_warmup</code> which improved massively the loading footprint by pre-allocating GPU memory to avoid malloc bottlenecks during model loading.</p>
-<p><div style="border: 1px solid #e2e8f0; border-radius: 8px; background: white; margin: 1.5rem 0;">
-    <div style="padding: 1rem; border-bottom: 1px solid #e2e8f0; background: #f8f9fa;">
-        <h4 style="margin: 0 0 0.5rem 0; color: #495057;">🚀 CUDA Warmup Efficiency Benchmark</h4>
-        <p style="margin: 0; font-size: 0.9em; color: #6c757d;">
-            Real CUDA warmup benchmarking with actual Transformers models. Measure the performance impact of the caching_allocator_warmup function.
-        </p>
-    </div>
-    
-    <div style="padding: 1rem;">
-        <iframe src=https://molbap-cuda-warmup-transformers.hf.space width=100% height=800px frameborder=0 style="border-radius: 8px; background: white;"></iframe>
-    </div>
-    
-    <div style="padding: 1rem; border-top: 1px solid #e2e8f0; background: #f8f9fa; font-size: 0.9em; color: #6c757d;">
-        Real CUDA warmup benchmarking with actual Transformers models. Measure the performance impact of the <code>caching_allocator_warmup</code> function at <code>transformers/src/transformers/modeling_utils.py:6186</code>. This interactive tool loads models twice - once with warmup disabled and once with warmup enabled - to demonstrate the significant loading time improvements.
+<p><style>.warmup-demo body{background-color:#f5f5f5;margin:0;padding:20px;font-family:Segoe UI,Tahoma,Geneva,Verdana,sans-serif}.warmup-demo .container{background:#fff;border-radius:12px;max-width:1200px;margin:0 auto;padding:30px;box-shadow:0 4px 6px #0000001a}.warmup-demo h1{text-align:center;color:#333;margin-bottom:10px}.warmup-demo .subtitle{text-align:center;color:#666;margin-bottom:30px;font-size:16px}.warmup-demo .demo-container{gap:40px;margin-bottom:30px;display:flex}.warmup-demo .side{background:#fafafa;border:2px solid #ddd;border-radius:8px;flex:1;padding:20px}.warmup-demo .side h2{text-align:center;color:#333;margin-top:0}.warmup-demo .no-warmup h2{color:#d63384}.warmup-demo .with-warmup h2{color:#198754}.warmup-demo .memory-area{background:#fff;border:2px dashed #ccc;border-radius:6px;height:400px;margin:20px 0;padding:10px;position:relative;overflow:hidden}.warmup-demo .layer-box{background:#fff;border:2px solid #666;border-radius:4px;width:80px;height:30px;margin:3px;transition:all .3s;display:inline-block;position:relative}.warmup-demo .layer-box.allocating{background:#e9ecef;border-color:#adb5bd}.warmup-demo .layer-box.allocating:after{content:"malloc";color:#666;font-size:10px;font-weight:700;position:absolute;top:50%;left:50%;transform:translate(-50%,-50%)}.warmup-demo .layer-box.loaded{background:#d1e7dd;border-color:#198754}.warmup-demo .layer-box.loaded:after{content:"data";color:#198754;font-size:10px;font-weight:700;position:absolute;top:50%;left:50%;transform:translate(-50%,-50%)}.warmup-demo .warmup-container{background:#fff;border:3px solid #666;border-radius:6px;width:100%;height:60px;margin-bottom:20px;position:relative;overflow:hidden}.warmup-demo .warmup-container.allocated{background:#e7f1ff;border-color:#0d6efd}.warmup-demo .warmup-container:before{content:"Pre-allocated Memory Pool";color:#666;z-index:1;font-size:14px;font-weight:700;position:absolute;top:50%;left:50%;transform:translate(-50%,-50%)}.warmup-demo .warmup-container.allocated:before{color:#0d6efd}.warmup-demo .warmup-fill{z-index:2;background:linear-gradient(90deg,#198754,#20c997);border-radius:3px;width:0%;height:100%;transition:width .5s;position:relative}.warmup-demo .warmup-fill:after{content:"Layer Data Loading";color:#fff;white-space:nowrap;font-size:12px;font-weight:700;position:absolute;top:50%;left:50%;transform:translate(-50%,-50%)}.warmup-demo .timing{text-align:center;min-height:30px;margin:15px 0;font-size:24px;font-weight:700}.warmup-demo .no-warmup .timing{color:#d63384}.warmup-demo .with-warmup .timing{color:#198754}.warmup-demo .controls{text-align:center;margin:30px 0}.warmup-demo .btn{color:#fff;cursor:pointer;background:#0d6efd;border:none;border-radius:6px;margin:0 10px;padding:12px 24px;font-size:16px;transition:background .3s}.warmup-demo .btn:hover{background:#0b5ed7}.warmup-demo .btn:disabled{cursor:not-allowed;background:#6c757d}.warmup-demo .description{background:#f8f9fa;border-radius:6px;margin-top:15px;padding:15px;font-size:14px;line-height:1.5}.warmup-demo .phase-indicator{color:#666;text-align:center;min-height:20px;margin-top:10px;font-size:14px}.warmup-demo .layer-counter{text-align:center;color:#495057;margin:10px 0;font-size:16px}</style>
+
+<div class=warmup-demo>
+    <div class=container>
+        <p class=subtitle>Mem allocation patterns during model loading</p>
+
+        <div class=controls>
+            <button class=btn id=startBtn onclick=startDemo()>Start Animation</button>
+            <button class=btn id=resetBtn onclick=resetDemo()>Reset</button>
+        </div>
+
+        <div class=demo-container>
+            <div class="no-warmup side">
+                <h2 data-no-toc>❌ Without Warmup</h2>
+                <div class=timing id=noWarmupTime>0.00s</div>
+                <div class=layer-counter id=noWarmupCounter>Layers loaded: 0/10</div>
+                <div class=phase-indicator id=noWarmupPhase></div>
+                <div class=memory-area id=noWarmupArea></div>
+                <div class=description>
+                    <strong>Individual Allocations:</strong><br>
+                    Each model layer triggers a separate cudaMalloc() call, creating memory fragmentation and allocation overhead.
+                    <br><br>
+                    📦 <strong>Grey "malloc"</strong> = Memory allocation overhead<br>
+                    ✅ <strong>Green "data"</strong> = Actual layer data loading
+                </div>
+            </div>
+
+            <div class="side with-warmup">
+                <h2 data-no-toc>✅ With Warmup</h2>
+                <div class=timing id=warmupTime>0.00s</div>
+                <div class=layer-counter id=warmupCounter>Layers loaded: 0/10</div>
+                <div class=phase-indicator id=warmupPhase></div>
+                <div class=memory-area id=warmupArea>
+                    <div class=warmup-container id=warmupContainer>
+                        <div class=warmup-fill id=warmupFill></div>
+                    </div>
+                    <div id=warmupLayers></div>
+                </div>
+                <div class=description>
+                    <strong>Pre-allocated Pool:</strong><br>
+                    The warmup function calculates total memory needed and makes ONE large allocation. Subsequent layers load directly into this pool, eliminating malloc overhead.
+                    <br><br>
+                    🔵 <strong>Blue container</strong> = Single large malloc (warmup)<br>
+                    🟢 <strong>Green progress bar</strong> = Layer data loading (no malloc needed)
+                </div>
+            </div>
+        </div>
     </div>
-</div></p>
-<h3>Linkedin post (to remove)</h3>
-<p>Linkedin post for videos:</p>
-<p>In transformers, how do we deal with cross-model dependencies, while supporting ~400 models? Maybe you’ve seen the same 200-lines functions in too many <em>modeling_file.py</em>? Duplication isn’t inevitable.</p>
-<p>The “one‑model/one‑file” rule keeps every model readable and runnable. It also means identical code is copied hundreds of times. Maintenance hurts, contributor PRs snowball, and vision–language models especially end up in siloed forks.</p>
-<p>modular_*.py fixes the trade‑off, by auto-generating the modeling file from a modular file, which can use inheritance.</p>
-<p>With a small analyser I’ve mapped which models already share modular pieces and which 100‑plus still repeat themselves. Red nodes in the graph = lowest‑hanging fruit for refactor; blue = already modular.</p>
-<p>The result: contributors can focus on novel layers instead of boilerplate, reviews shrink from “new file diff” to “does this override make sense?”, and the codebase stays something you can actually open and read.</p>
-<p>If you maintain or ship models on top of Transformers, take a look at modular, in 2025 it’s how we keep shipping breadth without the bloat. 🛠️</p>
+</div>
+
+<script>let animationSpeed=1/2.4,isRunning=!1,totalLayers=10;function startDemo(){isRunning||(isRunning=!0,document.getElementById("startBtn").disabled=!0,document.getElementById("resetBtn").disabled=!0,Promise.all([animateNoWarmup(),animateWithWarmup()]).then(()=>{isRunning=!1,document.getElementById("startBtn").disabled=!1,document.getElementById("resetBtn").disabled=!1}))}function resetDemo(){isRunning||(document.getElementById("noWarmupArea").innerHTML="",document.getElementById("warmupLayers").innerHTML="",document.getElementById("warmupFill").style.width="0%",document.getElementById("warmupContainer").classList.remove("allocated"),document.getElementById("noWarmupTime").textContent="0.00s",document.getElementById("warmupTime").textContent="0.00s",document.getElementById("noWarmupCounter").textContent="Layers loaded: 0/10",document.getElementById("warmupCounter").textContent="Layers loaded: 0/10",document.getElementById("noWarmupPhase").textContent="",document.getElementById("warmupPhase").textContent="")}async function animateNoWarmup(){let e=document.getElementById("noWarmupArea"),t=document.getElementById("noWarmupTime"),n=document.getElementById("noWarmupCounter"),a=document.getElementById("noWarmupPhase"),m=0,o=200/animationSpeed;a.textContent="Loading model layers...";for(let a=0;a<10;a++){let d=document.createElement("div");d.className="layer-box",e.appendChild(d),await sleep(.3*o),d.classList.add("allocating"),t.textContent=(m+=.08).toFixed(2)+"s",await sleep(.7*o),d.classList.remove("allocating"),d.classList.add("loaded"),t.textContent=(m+=.12).toFixed(2)+"s",n.textContent=`Layers loaded: ${a+1}/10`}a.textContent="Complete!"}async function animateWithWarmup(){let e=document.getElementById("warmupLayers"),t=document.getElementById("warmupTime"),n=document.getElementById("warmupCounter"),a=document.getElementById("warmupPhase"),m=document.getElementById("warmupContainer"),o=document.getElementById("warmupFill"),d=0,l=200/animationSpeed;a.textContent="Warming up allocator...",await sleep(2*l),m.classList.add("allocated"),t.textContent=(d+=.3).toFixed(2)+"s",a.textContent="Loading model layers...";for(let a=0;a<10;a++){let m=document.createElement("div");m.className="layer-box loaded",m.style.width="40px",m.style.height="20px",e.appendChild(m);let i=(a+1)/10*100;o.style.width=i+"%",await sleep(.5*l),t.textContent=(d+=.08).toFixed(2)+"s",n.textContent=`Layers loaded: ${a+1}/10`}a.textContent="Complete!"}function sleep(e){return new Promise(t=>setTimeout(t,e))}</script></p>
+<h2>What is coming next</h2>
+<p>It sounds dumb, but it’s true: the future is very soon. One tenet that will be broken when the next major version is released, v5, <a href="#backwards-compatibility">backwards compatibility</a> will be heavily broken. Instead, what we aim to be is way more of a <a href="#modular-toolbox">modular toolbox</a>, while maintaining a <a href="#consistent-public-surface">consistent public surface</a>.</p>
 
     </d-article>
     
@@ -534,7 +594,7 @@ curl -X POST http://localhost:8000/v1/chat/completions \
                             const article = document.querySelector('d-article');
                             const toc = document.querySelector('d-contents');
                             if (toc) {
-                                const headings = article.querySelectorAll('h1, h2, h3, h4');
+                                const headings = [...article.querySelectorAll('h1, h2, h3, h4')].filter(h => !h.hasAttribute('data-no-toc'));
                                 let ToC = '<nav role="navigation" class="l-text figcaption">';
                                 ToC += '<div class="toc-header"><span class="toc-title">Table of Contents</span></div>';
                                 ToC += '<div class="toc-content">';

dist/main.bundle.js

@@ -146,7 +146,7 @@ module.exports = styleTagTransform;
 var ___CSS_LOADER_EXPORT___ = _node_modules_css_loader_dist_runtime_api_js__WEBPACK_IMPORTED_MODULE_1___default()((_node_modules_css_loader_dist_runtime_sourceMaps_js__WEBPACK_IMPORTED_MODULE_0___default()));
 ___CSS_LOADER_EXPORT___.i(_node_modules_css_loader_dist_cjs_js_transformers_custom_css__WEBPACK_IMPORTED_MODULE_2__/* ["default"] */ .A);
 // Module
-___CSS_LOADER_EXPORT___.push([module.id, `/* style.css - Transformers Playthrough */
+___CSS_LOADER_EXPORT___.push([module.id, `/* style.css - Scaling Insanity */
 
 /* Import ultrascale-playbook base styles and add transformers-specific styling */
 /* Define colors */
@@ -885,7 +885,7 @@ iframe, .js-plotly-plot {
     margin-right: 0px;
 }
 
-/* Import transformers-specific styles */`, "",{"version":3,"sources":["webpack://./src/style.css"],"names":[],"mappings":"AAAA,yCAAyC;;AAEzC,iFAAiF;AACjF,kBAAkB;AAClB;IACI,kCAAkC;IAClC,wCAAwC;IACxC,0CAA0C;IAC1C,2CAA2C;IAC3C,uBAAuB;AAC3B;;AAEA,+BAA+B;AAC/B;IACI,aAAa;IACb,sBAAsB;IACtB,mBAAmB;IACnB,SAAS,EAAE,8CAA8C;AAC7D;AACA;IACI,kBAAkB;IAClB,eAAe;IACf,YAAY;AAChB;AACA;IACI,kBAAkB;IAClB,gBAAgB;AACpB;;AAEA;IACI,aAAa;IACb,eAAe;IACf,mBAAmB;IACnB,uBAAuB;IACvB,uBAAuB;IACvB,SAAS;AACb;;;AAGA;IACI,aAAa;IACb,mBAAmB;IACnB,sBAAsB;IACtB,SAAS;AACb;;AAEA,kCAAkC;AAClC;IACI,gBAAgB;IAChB,4BAA4B;IAC5B,kBAAkB;IAClB,kBAAkB;IAClB,kBAAkB;IAClB,eAAe;IACf,8CAA8C;IAC9C,aAAa;AACjB;;AAEA,8BAA8B;;AAE9B;IACI,aAAa;IACb,mBAAmB;IACnB,SAAS;AACb;;AAEA;IACI,wBAAwB;IACxB,WAAW;IACX,qCAAqC;IACrC,kBAAkB;IAClB,aAAa;AACjB;;AAEA;IACI,eAAe;IACf,kBAAkB;IAClB,eAAe;AACnB;;AAEA;IACI,wBAAwB;IACxB,gBAAgB;IAChB,WAAW;IACX,YAAY;IACZ,kBAAkB;IAClB,+BAA+B;IAC/B,eAAe;AACnB;;AAEA;IACI,WAAW;IACX,YAAY;IACZ,kBAAkB;IAClB,+BAA+B;IAC/B,eAAe;AACnB;;AAEA,yBAAyB;AACzB;IACI,eAAe;IACf,iBAAiB;AACrB;;AAEA;IACI,gBAAgB;IAChB,mBAAmB;AACvB;;AAEA;IACI,yBAAyB;IACzB,2CAA2C;IAC3C,kBAAkB;IAClB,eAAe;AACnB;AACA;IACI,wBAAwB;AAC5B;;;AAGA;IACI;QACI,wBAAwB;IAC5B;IACA;QACI,wBAAwB;IAC5B;AACJ;;AAEA;EACE,0BAA0B;EAC1B,iBAAiB;EACjB,iBAAiB;EACjB,kBAAkB;AACpB;;AAEA;EACE;IACE,kCAAkC;EACpC;AACF;;AAEA;IACI,eAAe;IACf,kBAAkB;AACtB;;AAEA;IACI,0BAA0B;AAC9B;;AAEA;IACI;QACI,aAAa;QACb,iBAAiB;QACjB,mBAAmB;QACnB,iBAAiB;QACjB,qBAAqB;QACrB,kBAAkB;QAClB,oBAAoB;QACpB,2CAA2C;QAC3C,wBAAwB;QACxB,0BAA0B;QAC1B,uCAAuC;QACvC,kBAAkB;QAClB,0BAA0B;QAC1B,qBAAqB;QACrB,aAAa;IACjB;AACJ;;AAEA;IACI,mBAAmB;AACvB;;AAEA;IACI,iBAAiB;IACjB,gBAAgB;IAChB,WAAW;AACf;;AAEA;IACI,0BAA0B;AAC9B;;AAEA;IACI,wBAAwB;AAC5B;;AAEA;IACI,gBAAgB;IAChB,gBAAgB;IAChB,wBAAwB;IACxB,oCAAoC;AACxC;;AAEA;IACI,aAAa;IACb,aAAa;AACjB;;AAEA;IACI;QACI,yDAAyD;QACzD,iBAAiB;IACrB;;IAEA;QACI,iBAAiB;QACjB,iBAAiB;QACjB,+BAA+B;QAC/B,6BAA6B;QAC7B,uBAAuB;QACvB,iBAAiB;QACjB,eAAe;QACf,kBAAkB;QAClB,iBAAiB;QACjB;;;iDAGyC;QACzC,wBAAwB,EAAE,eAAe;QACzC,gBAAgB;QAChB,SAAS,EAAE,gCAAgC;QAC3C,gBAAgB;QAChB,0BAA0B;QAC1B,qBAAqB;QACrB,oCAAoC;QACpC,aAAa;IACjB;AACJ;;AAEA;IACI,aAAa;IACb,kBAAkB;AACtB;;AAEA;IACI,yBAAyB;IACzB,iBAAiB;AACrB;;AAEA;IACI,yBAAyB;IACzB,mBAAmB;IACnB,qBAAqB;AACzB;;AAEA;IACI,qBAAqB;AACzB;;AAEA;IACI,iBAAiB;AACrB;;AAEA;IACI,oBAAoB;AACxB;;AAEA;IACI,mDAAmD;AACvD;;AAEA;IACI,aAAa;IACb,kBAAkB;AACtB;;;AAGA;IACI,cAAc;IACd,aAAa;IACb,oBAAoB;AACxB;;AAEA;IACI,eAAe;IACf,gBAAgB;AACpB;;AAEA;IACI,kBAAkB;AACtB;;AAEA;IACI,eAAe;AACnB;;AAEA;IACI;QACI,gBAAgB;IACpB;AACJ;;AAEA;;IAEI,qBAAqB;AACzB;;AAEA;IACI,yBAAyB;IACzB,8BAA8B;IAC9B,aAAa;IACb,cAAc,GAAG,4BAA4B;IAC7C,kBAAkB;IAClB,0DAA0D;IAC1D,qBAAqB;AACzB;;AAEA;IACI,SAAS;IACT,cAAc;IACd,gBAAgB;IAChB,cAAc;AAClB;;AAEA;IACI,kBAAkB;IAClB,gBAAgB,GAAG,4BAA4B;IAC/C,cAAc;IACd,gBAAgB;IAChB,kBAAkB;AACtB;;AAEA,0BAA0B;AAC1B;IACI;QACI,yBAAyB;QACzB,0BAA0B;IAC9B;IACA;QACI,cAAc;IAClB;IACA;QACI,cAAc;IAClB;AACJ;;AAEA;IACI,gBAAgB;AACpB;;AAEA;IACI,gBAAgB;IAChB,kBAAkB;IAClB,0BAA0B;IAC1B,kBAAkB;AACtB;;AAEA;IACI,eAAe;AACnB;;AAEA;QACQ,YAAY;QACZ,iBAAiB;QACjB,oBAAoB;QACpB,8BAA8B;QAC9B,+BAA+B;QAC/B,iBAAiB;QACjB,mBAAmB,EAAE,6CAA6C;QAClE,aAAa;QACb,uBAAuB,EAAE,gCAAgC;AACjE;;AAEA;IACI,kBAAkB;IAClB,iBAAiB;IACjB,oBAAoB;IACpB,oCAAoC;IACpC,kBAAkB;IAClB,oBAAoB;IACpB,qCAAqC;IACrC,uBAAuB;IACvB,mBAAmB,EAAE,6CAA6C;IAClE,aAAa;IACb,uBAAuB,EAAE,gCAAgC;AAC7D;;AAEA;IACI,eAAe;IACf,iBAAiB;IACjB,mBAAmB;IACnB,yBAAyB;IACzB,wCAAwC;AAC5C;;AAEA;IACI,oBAAoB;AACxB;;AAEA;IACI,oBAAoB;AACxB;;AAEA;IACI,oBAAoB;AACxB;;AAEA;IACI,iBAAiB;AACrB;;AAEA,yBAAyB;AACzB;IACI,gBAAgB;IAChB,kBAAkB;IAClB,mBAAmB;IACnB,eAAe;IACf,mBAAmB;IACnB,mBAAmB;IACnB,yBAAyB;IACzB,wCAAwC;AAC5C;;;AAGA,0BAA0B;AAC1B;IACI,iBAAiB;IACjB,uBAAuB;IACvB,mBAAmB;IACnB,iBAAiB;IACjB,eAAe;IACf,mBAAmB;IACnB,mBAAmB;IACnB,yBAAyB;IACzB,wCAAwC;AAC5C;;AAEA,gBAAgB;AAChB;IACI,aAAa;IACb,8BAA8B;IAC9B,eAAe;AACnB;;AAEA,gBAAgB;AAChB;IACI,qBAAqB;AACzB;;AAEA,iBAAiB;AACjB;IACI,cAAc;IACd,2BAA2B;IAC3B,iBAAiB;IACjB,gBAAgB;IAChB,cAAc;AAClB;;AAEA,mDAAmD;AACnD;IACI,aAAa;IACb,SAAS;IACT,mBAAmB;AACvB;;AAEA,wBAAwB;AACxB;IACI,OAAO;IACP,WAAW;IACX,mBAAmB;IACnB,kBAAkB;IAClB,gBAAgB;IAChB,aAAa;AACjB;;AAEA;IACI,gBAAgB;IAChB,WAAW;IACX,YAAY;IACZ,mBAAmB;IACnB,kBAAkB;IAClB,eAAe;IACf,iCAAiC;AACrC;;AAEA;IACI,mBAAmB;AACvB;;AAEA,yBAAyB;AACzB;IACI,WAAW;IACX,eAAe;IACf,yBAAyB;IACzB,kBAAkB;IAClB,iBAAiB;IACjB,cAAc;AAClB;;AAEA,mBAAmB;AACnB;IACI,WAAW;IACX,eAAe;IACf,yBAAyB;IACzB,kBAAkB;IAClB,iBAAiB;IACjB,iBAAiB;IACjB,cAAc;IACd,eAAe;AACnB;;AAEA,qBAAqB;AACrB;IACI,aAAa;IACb,cAAc;IACd,oBAAoB;IACpB,yBAAyB;IACzB,kBAAkB;IAClB,eAAe;AACnB;;AAEA,2BAA2B;AAC3B;IACI,qBAAqB;AACzB;;AAEA;IACI,oBAAoB;AACxB;;AAEA,uBAAuB;AACvB;IACI,aAAa;IACb,mBAAmB;IACnB,mBAAmB;AACvB;;AAEA,gCAAgC;AAChC;IACI,gBAAgB;IAChB,kBAAkB;IAClB,aAAa;IACb,mBAAmB;IACnB,yCAAyC;AAC7C;;AAEA;IACI,iBAAiB;IACjB,gBAAgB;IAChB,cAAc;IACd,qBAAqB;AACzB;;AAEA;IACI,iBAAiB;IACjB,gBAAgB;IAChB,cAAc;AAClB;;AAEA,2BAA2B;AAC3B;IACI;QACI,0BAA0B;QAC1B,aAAa;IACjB;;IAEA;QACI,UAAU;IACd;AACJ;;AAEA,iCAAiC;AACjC;IACI,qBAAqB;AACzB;;AAEA;IACI,qBAAqB;IACrB,aAAa;IACb,6CAA6C;AACjD;;AAEA,2BAA2B;AAC3B;IACI,0BAA0B;AAC9B;;AAEA,oCAAoC;AACpC;IACI,yBAAyB;IACzB,gBAAgB;AACpB;;AAEA,8BAA8B;AAC9B;IACI,mBAAmB;IACnB,aAAa;IACb,kBAAkB;IAClB,mBAAmB;AACvB;;AAEA;IACI,mBAAmB;IACnB,aAAa;IACb,kBAAkB;AACtB;;AAEA;YACY,qDAAqD;YACrD,YAAY;YACZ,eAAe;YACf,gBAAgB;YAChB,kBAAkB;YAClB,YAAY;YACZ,mBAAmB;YACnB,eAAe;YACf,yBAAyB;YACzB,mBAAmB;YACnB,yCAAyC;YACzC,yBAAyB;YACzB,kBAAkB;YAClB,gBAAgB;QACpB;AACR;IACI,2BAA2B;IAC3B,0CAA0C;AAC9C;;AAEA;IACI,wBAAwB;IACxB,yCAAyC;AAC7C;;AAEA;IACI,WAAW;IACX,kBAAkB;IAClB,MAAM;IACN,WAAW;IACX,WAAW;IACX,YAAY;IACZ,qFAAqF;IACrF,0BAA0B;AAC9B;;AAEA;IACI,UAAU;AACd;;AAEA;YACY,qDAAqD;YACrD,YAAY;YACZ,eAAe;YACf,gBAAgB;YAChB,kBAAkB;YAClB,YAAY;YACZ,mBAAmB;YACnB,eAAe;YACf,yBAAyB;YACzB,mBAAmB;YACnB,yCAAyC;YACzC,yBAAyB;YACzB,kBAAkB;YAClB,gBAAgB;QACpB;;AAER;IACI,2BAA2B;IAC3B,0CAA0C;AAC9C;;AAEA;IACI,wBAAwB;IACxB,yCAAyC;AAC7C;;AAEA;IACI,WAAW;IACX,kBAAkB;IAClB,MAAM;IACN,WAAW;IACX,WAAW;IACX,YAAY;IACZ,qFAAqF;IACrF,0BAA0B;AAC9B;;AAEA;IACI,UAAU;AACd;AACA;IACI,mBAAmB;IACnB,uBAAuB;IACvB,aAAa;AACjB;;AAEA;IACI,aAAa;IACb,uBAAuB;IACvB,oBAAoB;AACxB;;AAEA;IACI,qBAAqB;AACzB;;;AAGA;IACI,qBAAqB;IACrB,mBAAmB;AACvB;;AAEA;IACI,aAAa;AACjB;;AAEA;IACI,aAAa,EAAE,8BAA8B,EAAE,mBAAmB;AACtE;;AAEA;IACI;QACI,0BAA0B;IAC9B;AACJ;;AAEA;IACI;QACI,0BAA0B;IAC9B;IACA;QACI,iBAAiB;IACrB;IACA;QACI,gBAAgB;IACpB;AACJ;;AAEA;IACI,kCAAkC;AACtC;;AAEA;IACI,gBAAgB;IAChB,iBAAiB;AACrB;;AAEA,wCAAwC","sourcesContent":["/* style.css - Transformers Playthrough */\n\n/* Import ultrascale-playbook base styles and add transformers-specific styling */\n/* Define colors */\n:root {\n    --distill-gray: rgb(107, 114, 128);\n    --distill-gray-light: rgb(185, 185, 185);\n    --distill-gray-lighter: rgb(228, 228, 228);\n    --distill-gray-lightest: rgb(245, 245, 245);\n    --distill-blue: #007BFF;\n}\n\n/* Container for the controls */\n[id^=\"plot-\"] {\n    display: flex;\n    flex-direction: column;\n    align-items: center;\n    gap: 15px; /* Adjust the gap between controls as needed */\n}\n[id^=\"plot-\"] figure {\n    margin-bottom: 0px;\n    margin-top: 0px;\n    padding: 0px;\n}\n.plotly_caption {\n    font-style: italic;\n    margin-top: 10px;\n}\n\n.plotly_controls {\n    display: flex;\n    flex-wrap: wrap;\n    flex-direction: row;\n    justify-content: center;\n    align-items: flex-start;\n    gap: 30px;\n}\n\n\n.plotly_input_container {\n    display: flex;\n    align-items: center;\n    flex-direction: column;\n    gap: 10px;\n}\n\n/* Style for the select dropdown */\n.plotly_input_container > select {\n    padding: 2px 4px;\n    /* border: 1px solid #ccc; */\n    line-height: 1.5em;\n    text-align: center;\n    border-radius: 4px;\n    font-size: 12px;\n    background-color: var(--distill-gray-lightest);\n    outline: none;\n}\n\n/* Style for the range input */\n\n.plotly_slider {\n    display: flex;\n    align-items: center;\n    gap: 10px;\n}\n\n.plotly_slider > input[type=\"range\"] {\n    -webkit-appearance: none;\n    height: 2px;\n    background: var(--distill-gray-light);\n    border-radius: 5px;\n    outline: none;\n}\n\n.plotly_slider > span {\n    font-size: 14px;\n    line-height: 1.6em;\n    min-width: 16px;\n}\n\n.plotly_slider > input[type=\"range\"]::-webkit-slider-thumb {\n    -webkit-appearance: none;\n    appearance: none;\n    width: 18px;\n    height: 18px;\n    border-radius: 50%;\n    background: var(--distill-blue);\n    cursor: pointer;\n}\n\n.plotly_slider > input[type=\"range\"]::-moz-range-thumb {\n    width: 18px;\n    height: 18px;\n    border-radius: 50%;\n    background: var(--distill-blue);\n    cursor: pointer;\n}\n\n/* Style for the labels */\n.plotly_input_container > label {\n    font-size: 14px;\n    font-weight: bold;\n}\n\n.main-plot-container {\n    margin-top: 21px;\n    margin-bottom: 35px;\n}\n\n.main-plot-container > figure {\n    display: block !important;\n    /* Let this be handled by graph-container */\n    margin-bottom: 0px;\n    margin-top: 0px;\n}\n.main-plot-container > div {\n    display: none !important;\n}\n\n\n@media (min-width: 768px) {\n    .main-plot-container > figure {\n        display: none !important;\n    }\n    .main-plot-container > div {\n        display: flex !important;\n    }\n}\n\nd-byline .byline {\n  grid-template-columns: 1fr;\n  grid-column: text;\n  font-size: 0.9rem;\n  line-height: 1.8em;\n}\n\n@media (min-width: 768px) {\n  d-byline .byline {\n    grid-template-columns: 5fr 1fr 1fr;\n  }\n}\n\n#title-plot {\n    margin-top: 0px;\n    margin-bottom: 0px;\n}\n\nd-contents > nav a.active {\n    text-decoration: underline;\n}\n\n@media (max-width: 1199px) {\n    d-contents {\n        display: none;\n        background: white;\n        justify-self: start;\n        align-self: start;\n        padding-bottom: 0.5em;\n        margin-bottom: 1em;\n        padding-left: 0.25em;\n        border-bottom: 1px solid rgba(0, 0, 0, 0.1);\n        border-bottom-width: 1px;\n        border-bottom-style: solid;\n        border-bottom-color: rgba(0, 0, 0, 0.1);\n        overflow-y: scroll;\n        height: calc(100vh - 40px);\n        scrollbar-width: none;\n        z-index: -100;\n    }\n}\n\nd-contents a:hover {\n    border-bottom: none;\n}\n\ntoc-title {\n    font-weight: bold;\n    font-size: 1.2em;\n    color: #333;\n}\n\ntoggle-icon {\n    transition: transform 0.3s;\n}\n\ntoggle-icon.collapsed {\n    transform: rotate(90deg);\n}\n\n.toc-content {\n    margin-top: 15px;\n    overflow: hidden;\n    /* max-height: 1000px; */\n    transition: max-height 0.3s ease-out;\n}\n\n.toc-content.collapsed {\n    max-height: 0;\n    margin-top: 0;\n}\n\n@media (min-width: 1200px) {\n    d-article {\n        /* Ensure d-article does not prevent sticky positioning */\n        overflow: visible;\n    }\n\n    d-contents {\n        align-self: start;\n        background: white;\n        grid-column-start: 1 !important;\n        grid-column-end: 4 !important;\n        grid-row: auto / span 6;\n        justify-self: end;\n        margin-top: 0em;\n        padding-right: 3em;\n        padding-left: 2em;\n        /* border-right: 1px solid rgba(0, 0, 0, 0.1);\n        border-right-width: 1px;\n        border-right-style: solid;\n        border-right-color: rgba(0, 0, 0, 0.1); */\n        position: -webkit-sticky; /* For Safari */\n        position: sticky;\n        top: 10px; /* Adjust this value if needed */\n        overflow-y: auto;\n        height: calc(100vh - 40px);\n        scrollbar-width: none;\n        transition: max-height 0.3s ease-out;\n        z-index: -100;\n    }\n}\n\nd-contents nav h3 {\n    margin-top: 0;\n    margin-bottom: 1em;\n}\n\nd-contents nav div div {\n    color: rgba(0, 0, 0, 0.8);\n    font-weight: bold;\n}\n\nd-contents nav a {\n    color: rgba(0, 0, 0, 0.8);\n    border-bottom: none;\n    text-decoration: none;\n}\n\nd-contents li {\n    list-style-type: none;\n}\n\nd-contents ul, d-article d-contents ul {\n    padding-left: 1em;\n}\n\nd-contents nav ul li {\n    margin-bottom: .25em;\n}\n\nd-contents nav a:hover {\n    text-decoration: underline solid rgba(0, 0, 0, 0.6);\n}\n\nd-contents nav ul {\n    margin-top: 0;\n    margin-bottom: 6px;\n}\n\n\nd-contents nav > div {\n    display: block;\n    outline: none;\n    margin-bottom: 0.5em;\n}\n\nd-contents nav > div > a {\n    font-size: 13px;\n    font-weight: 600;\n}\n\nd-article aside {\n    margin-bottom: 1em;\n}\n\nd-article img {\n    max-width: 100%;\n}\n\n@media (min-width: 768px) {\n    d-article aside {\n        margin-bottom: 0;\n    }\n}\n\nd-contents nav > div > a:hover,\nd-contents nav > ul > li > a:hover {\n    text-decoration: none;\n}\n\n.note-box {\n    background-color: #f6f8fa;\n    border-left: 4px solid #444444;\n    padding: 1rem;\n    margin: 1rem 0;  /* Keep this modest margin */\n    border-radius: 6px;\n    /* Add this to ensure the box only takes up needed space */\n    display: inline-block;\n}\n\n.note-box-title {\n    margin: 0;\n    color: #444444;\n    font-weight: 600;\n    font-size: 1em;\n}\n\n.note-box-content {\n    margin-top: 0.5rem;\n    margin-bottom: 0;  /* Ensure no bottom margin */\n    color: #24292f;\n    font-size: 0.9em;\n    line-height: 1.5em;\n}\n\n/* For dark mode support */\n@media (prefers-color-scheme: dark) {\n    .note-box {\n        background-color: #1c1c1c;\n        border-left-color: #888888;\n    }\n    .note-box-title {\n        color: #888888;\n    }\n    .note-box-content {\n        color: #d4d4d4;\n    }\n}\n\nd-article {\n    font-size: 1.0em;\n}\n\n.figure-legend {\n    font-size: 0.9em;\n    font-style: italic;\n    color: var(--distill-gray);\n    line-height: 1.5em;\n}\n\nd-code {\n    font-size: 12px;\n}\n\n.large-image-background {\n        width: 100vw;\n        padding-top: 10px;\n        padding-bottom: 10px;\n        margin-left: calc(-50vw + 50%);\n        margin-right: calc(-50vw + 50%);\n        background: white;\n        height: fit-content; /* This will make it match the image height */\n        display: flex;\n        justify-content: center; /* This will center your image */\n}\n\n.large-image-background-transparent {\n    /* width: 100vw; */\n    padding-top: 10px;\n    padding-bottom: 10px;\n    /* margin-left: calc(-50vw + 50%); */\n    margin-left:-100px;\n    margin-right: -100px;\n    /* margin-right: calc(-50vw + 50%); */\n    /* background: white; */\n    height: fit-content; /* This will make it match the image height */\n    display: flex;\n    justify-content: center; /* This will center your image */\n}\n\n.boxed-image {\n    padding: 0.5rem;\n    background: white;\n    border-radius: 12px;\n    border: 1px solid #e5e7eb;\n    box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);\n}\n\nd-article li {\n    margin-bottom: 0.0em;\n}\n\nd-article ul ul {\n    margin-bottom: 0.0em;\n}\n\nd-article ol ol {\n    margin-bottom: 0.0em;\n}\n\nd-article hr {\n    grid-column: text;\n}\n\n/* Memory visualization */\n#graph-all {\n    min-width: 500px;\n    margin-right: 10px;\n    margin-bottom: 2rem;\n    padding: 0.5rem;\n    background: #f9fafb;\n    border-radius: 12px;\n    border: 1px solid #e5e7eb;\n    box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);\n}\n\n\n/* Main container styles */\n#controls {\n    max-width: 1200px;\n    /* margin: 2rem auto; */\n    margin-bottom: 2rem;\n    margin-left: 10px;\n    padding: 0.6rem;\n    background: #f9fafb;\n    border-radius: 12px;\n    border: 1px solid #e5e7eb;\n    box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);\n}\n\n/* Grid layout */\n#controls {\n    display: grid;\n    grid-template-columns: 1fr 1fr;\n    /* gap: 2rem; */\n}\n\n/* Cell styles */\n.cell {\n    margin-bottom: 0.2rem;\n}\n\n/* Label styles */\nlabel {\n    display: block;\n    /* margin-bottom: 0.5rem; */\n    font-size: 0.8rem;\n    font-weight: 500;\n    color: #374151;\n}\n\n/* Input container for range + number combination */\n.input-container {\n    display: flex;\n    gap: 1rem;\n    align-items: center;\n}\n\n/* Range input styling */\ninput[type=\"range\"] {\n    flex: 1;\n    height: 6px;\n    background: #e5e7eb;\n    border-radius: 3px;\n    appearance: none;\n    outline: none;\n}\n\ninput[type=\"range\"]::-webkit-slider-thumb {\n    appearance: none;\n    width: 16px;\n    height: 16px;\n    background: #3b82f6;\n    border-radius: 50%;\n    cursor: pointer;\n    transition: background 0.15s ease;\n}\n\ninput[type=\"range\"]::-webkit-slider-thumb:hover {\n    background: #2563eb;\n}\n\n/* Number input styling */\ninput[type=\"number\"] {\n    width: 80px;\n    padding: 0.5rem;\n    border: 1px solid #e5e7eb;\n    border-radius: 6px;\n    font-size: 0.9rem;\n    color: #374151;\n}\n\n/* Select styling */\nselect {\n    width: 100%;\n    padding: 0.5rem;\n    border: 1px solid #e5e7eb;\n    border-radius: 6px;\n    background: white;\n    font-size: 0.9rem;\n    color: #374151;\n    cursor: pointer;\n}\n\n/* Checkbox styling */\ninput[type=\"checkbox\"] {\n    width: 1.2rem;\n    height: 1.2rem;\n    margin-right: 0.5rem;\n    border: 2px solid #e5e7eb;\n    border-radius: 4px;\n    cursor: pointer;\n}\n\n/* Column specific styles */\n.column-1 {\n    padding-right: 0.5rem;\n}\n\n.column-2 {\n    padding-left: 0.5rem;\n}\n\n/* Checkbox container */\n.checkbox-container {\n    display: flex;\n    align-items: center;\n    margin-bottom: 1rem;\n}\n\n/* Memory visualization styles */\n.memory-block {\n    background: #fff;\n    border-radius: 8px;\n    padding: 1rem;\n    margin-bottom: 1rem;\n    box-shadow: 0 2px 4px rgba(0, 0, 0, 0.05);\n}\n\n.memory-title {\n    font-size: 1.1rem;\n    font-weight: 500;\n    color: #374151;\n    margin-bottom: 0.5rem;\n}\n\n.memory-value {\n    font-size: 1.5rem;\n    font-weight: 600;\n    color: #3b82f6;\n}\n\n/* Responsive adjustments */\n@media (max-width: 768px) {\n    #controls {\n        grid-template-columns: 1fr;\n        padding: 1rem;\n    }\n\n    .column-1, .column-2 {\n        padding: 0;\n    }\n}\n\n/* Hover states and transitions */\ninput:hover, select:hover {\n    border-color: #3b82f6;\n}\n\ninput:focus, select:focus {\n    border-color: #2563eb;\n    outline: none;\n    box-shadow: 0 0 0 2px rgba(59, 130, 246, 0.1);\n}\n\n/* Add smooth transitions */\ninput, select, button {\n    transition: all 0.15s ease;\n}\n\n/* Preset dropdown special styling */\nselect[name=\"presets\"] {\n    background-color: #f3f4f6;\n    font-weight: 500;\n}\n\n/* Memory graph enhancements */\n.activation-memory {\n    background: #dbeafe;\n    padding: 1rem;\n    border-radius: 8px;\n    margin-bottom: 1rem;\n}\n\n.gradient-memory {\n    background: #ede9fe;\n    padding: 1rem;\n    border-radius: 8px;\n}\n\n.order-button-second {\n            background: linear-gradient(135deg, #6DB4C4, #D4A5B8);\n            color: white;\n            font-size: 18px;\n            font-weight: 600;\n            padding: 20px 20px;\n            border: none;\n            border-radius: 12px;\n            cursor: pointer;\n            text-transform: uppercase;\n            letter-spacing: 1px;\n            box-shadow: 0 4px 15px rgba(0, 0, 0, 0.2);\n            transition: all 0.3s ease;\n            position: relative;\n            overflow: hidden;\n        }\n.order-button-second:hover {\n    transform: translateY(-2px);\n    box-shadow: 0 6px 20px rgba(0, 0, 0, 0.25);\n}\n\n.order-button:active {\n    transform: translateY(0);\n    box-shadow: 0 2px 10px rgba(0, 0, 0, 0.2);\n}\n\n.order-button-second::before {\n    content: '';\n    position: absolute;\n    top: 0;\n    left: -100%;\n    width: 100%;\n    height: 100%;\n    background: linear-gradient(135deg, rgba(255, 255, 255, 0.2), rgba(255, 255, 255, 0));\n    transition: left 0.5s ease;\n}\n\n.order-button-second:hover::before {\n    left: 100%;\n}\n\n.order-button {\n            background: linear-gradient(135deg, #6DB4C4, #D4A5B8);\n            color: white;\n            font-size: 18px;\n            font-weight: 600;\n            padding: 16px 32px;\n            border: none;\n            border-radius: 12px;\n            cursor: pointer;\n            text-transform: uppercase;\n            letter-spacing: 1px;\n            box-shadow: 0 4px 15px rgba(0, 0, 0, 0.2);\n            transition: all 0.3s ease;\n            position: relative;\n            overflow: hidden;\n        }\n\n.order-button:hover {\n    transform: translateY(-2px);\n    box-shadow: 0 6px 20px rgba(0, 0, 0, 0.25);\n}\n\n.order-button:active {\n    transform: translateY(0);\n    box-shadow: 0 2px 10px rgba(0, 0, 0, 0.2);\n}\n\n.order-button::before {\n    content: '';\n    position: absolute;\n    top: 0;\n    left: -100%;\n    width: 100%;\n    height: 100%;\n    background: linear-gradient(135deg, rgba(255, 255, 255, 0.2), rgba(255, 255, 255, 0));\n    transition: left 0.5s ease;\n}\n\n.order-button:hover::before {\n    left: 100%;\n}\n.order-button-container-second {\n    /* display: flex; */\n    justify-content: center;\n    margin: 0px 0;\n}\n\n.order-button-container {\n    display: flex;\n    justify-content: center;\n    margin: 0px 0 40px 0;\n}\n\nd-article img {\n    width: 100%!important;\n}\n\n\niframe, .js-plotly-plot {\n    width: 100%!important;\n    margin-bottom: 20px;\n}\n\n.modebar-container {\n    display: none;\n}\n\n#graph-container {\n    display: grid; grid-template-columns: 1fr 1fr; align-items: center;\n}\n\n@media (max-width: 768px) {\n    #graph-container {\n        grid-template-columns: 1fr;\n    }\n}\n\n@media (max-width: 1024px) {\n    #graph-container {\n        grid-template-columns: 1fr;\n    }\n    #graph-all {\n        margin-right: 0px;\n    }\n    #controls {\n        margin-left: 0px;\n    }\n}\n\n.main-plot-container svg {\n    background: transparent !important;\n}\n\n.large-image-background-transparent {\n    margin-left: 0px;\n    margin-right: 0px;\n}\n\n/* Import transformers-specific styles */\n@import url('./transformers-custom.css');"],"sourceRoot":""}]);
+/* Import transformers-specific styles */`, "",{"version":3,"sources":["webpack://./src/style.css"],"names":[],"mappings":"AAAA,iCAAiC;;AAEjC,iFAAiF;AACjF,kBAAkB;AAClB;IACI,kCAAkC;IAClC,wCAAwC;IACxC,0CAA0C;IAC1C,2CAA2C;IAC3C,uBAAuB;AAC3B;;AAEA,+BAA+B;AAC/B;IACI,aAAa;IACb,sBAAsB;IACtB,mBAAmB;IACnB,SAAS,EAAE,8CAA8C;AAC7D;AACA;IACI,kBAAkB;IAClB,eAAe;IACf,YAAY;AAChB;AACA;IACI,kBAAkB;IAClB,gBAAgB;AACpB;;AAEA;IACI,aAAa;IACb,eAAe;IACf,mBAAmB;IACnB,uBAAuB;IACvB,uBAAuB;IACvB,SAAS;AACb;;;AAGA;IACI,aAAa;IACb,mBAAmB;IACnB,sBAAsB;IACtB,SAAS;AACb;;AAEA,kCAAkC;AAClC;IACI,gBAAgB;IAChB,4BAA4B;IAC5B,kBAAkB;IAClB,kBAAkB;IAClB,kBAAkB;IAClB,eAAe;IACf,8CAA8C;IAC9C,aAAa;AACjB;;AAEA,8BAA8B;;AAE9B;IACI,aAAa;IACb,mBAAmB;IACnB,SAAS;AACb;;AAEA;IACI,wBAAwB;IACxB,WAAW;IACX,qCAAqC;IACrC,kBAAkB;IAClB,aAAa;AACjB;;AAEA;IACI,eAAe;IACf,kBAAkB;IAClB,eAAe;AACnB;;AAEA;IACI,wBAAwB;IACxB,gBAAgB;IAChB,WAAW;IACX,YAAY;IACZ,kBAAkB;IAClB,+BAA+B;IAC/B,eAAe;AACnB;;AAEA;IACI,WAAW;IACX,YAAY;IACZ,kBAAkB;IAClB,+BAA+B;IAC/B,eAAe;AACnB;;AAEA,yBAAyB;AACzB;IACI,eAAe;IACf,iBAAiB;AACrB;;AAEA;IACI,gBAAgB;IAChB,mBAAmB;AACvB;;AAEA;IACI,yBAAyB;IACzB,2CAA2C;IAC3C,kBAAkB;IAClB,eAAe;AACnB;AACA;IACI,wBAAwB;AAC5B;;;AAGA;IACI;QACI,wBAAwB;IAC5B;IACA;QACI,wBAAwB;IAC5B;AACJ;;AAEA;EACE,0BAA0B;EAC1B,iBAAiB;EACjB,iBAAiB;EACjB,kBAAkB;AACpB;;AAEA;EACE;IACE,kCAAkC;EACpC;AACF;;AAEA;IACI,eAAe;IACf,kBAAkB;AACtB;;AAEA;IACI,0BAA0B;AAC9B;;AAEA;IACI;QACI,aAAa;QACb,iBAAiB;QACjB,mBAAmB;QACnB,iBAAiB;QACjB,qBAAqB;QACrB,kBAAkB;QAClB,oBAAoB;QACpB,2CAA2C;QAC3C,wBAAwB;QACxB,0BAA0B;QAC1B,uCAAuC;QACvC,kBAAkB;QAClB,0BAA0B;QAC1B,qBAAqB;QACrB,aAAa;IACjB;AACJ;;AAEA;IACI,mBAAmB;AACvB;;AAEA;IACI,iBAAiB;IACjB,gBAAgB;IAChB,WAAW;AACf;;AAEA;IACI,0BAA0B;AAC9B;;AAEA;IACI,wBAAwB;AAC5B;;AAEA;IACI,gBAAgB;IAChB,gBAAgB;IAChB,wBAAwB;IACxB,oCAAoC;AACxC;;AAEA;IACI,aAAa;IACb,aAAa;AACjB;;AAEA;IACI;QACI,yDAAyD;QACzD,iBAAiB;IACrB;;IAEA;QACI,iBAAiB;QACjB,iBAAiB;QACjB,+BAA+B;QAC/B,6BAA6B;QAC7B,uBAAuB;QACvB,iBAAiB;QACjB,eAAe;QACf,kBAAkB;QAClB,iBAAiB;QACjB;;;iDAGyC;QACzC,wBAAwB,EAAE,eAAe;QACzC,gBAAgB;QAChB,SAAS,EAAE,gCAAgC;QAC3C,gBAAgB;QAChB,0BAA0B;QAC1B,qBAAqB;QACrB,oCAAoC;QACpC,aAAa;IACjB;AACJ;;AAEA;IACI,aAAa;IACb,kBAAkB;AACtB;;AAEA;IACI,yBAAyB;IACzB,iBAAiB;AACrB;;AAEA;IACI,yBAAyB;IACzB,mBAAmB;IACnB,qBAAqB;AACzB;;AAEA;IACI,qBAAqB;AACzB;;AAEA;IACI,iBAAiB;AACrB;;AAEA;IACI,oBAAoB;AACxB;;AAEA;IACI,mDAAmD;AACvD;;AAEA;IACI,aAAa;IACb,kBAAkB;AACtB;;;AAGA;IACI,cAAc;IACd,aAAa;IACb,oBAAoB;AACxB;;AAEA;IACI,eAAe;IACf,gBAAgB;AACpB;;AAEA;IACI,kBAAkB;AACtB;;AAEA;IACI,eAAe;AACnB;;AAEA;IACI;QACI,gBAAgB;IACpB;AACJ;;AAEA;;IAEI,qBAAqB;AACzB;;AAEA;IACI,yBAAyB;IACzB,8BAA8B;IAC9B,aAAa;IACb,cAAc,GAAG,4BAA4B;IAC7C,kBAAkB;IAClB,0DAA0D;IAC1D,qBAAqB;AACzB;;AAEA;IACI,SAAS;IACT,cAAc;IACd,gBAAgB;IAChB,cAAc;AAClB;;AAEA;IACI,kBAAkB;IAClB,gBAAgB,GAAG,4BAA4B;IAC/C,cAAc;IACd,gBAAgB;IAChB,kBAAkB;AACtB;;AAEA,0BAA0B;AAC1B;IACI;QACI,yBAAyB;QACzB,0BAA0B;IAC9B;IACA;QACI,cAAc;IAClB;IACA;QACI,cAAc;IAClB;AACJ;;AAEA;IACI,gBAAgB;AACpB;;AAEA;IACI,gBAAgB;IAChB,kBAAkB;IAClB,0BAA0B;IAC1B,kBAAkB;AACtB;;AAEA;IACI,eAAe;AACnB;;AAEA;QACQ,YAAY;QACZ,iBAAiB;QACjB,oBAAoB;QACpB,8BAA8B;QAC9B,+BAA+B;QAC/B,iBAAiB;QACjB,mBAAmB,EAAE,6CAA6C;QAClE,aAAa;QACb,uBAAuB,EAAE,gCAAgC;AACjE;;AAEA;IACI,kBAAkB;IAClB,iBAAiB;IACjB,oBAAoB;IACpB,oCAAoC;IACpC,kBAAkB;IAClB,oBAAoB;IACpB,qCAAqC;IACrC,uBAAuB;IACvB,mBAAmB,EAAE,6CAA6C;IAClE,aAAa;IACb,uBAAuB,EAAE,gCAAgC;AAC7D;;AAEA;IACI,eAAe;IACf,iBAAiB;IACjB,mBAAmB;IACnB,yBAAyB;IACzB,wCAAwC;AAC5C;;AAEA;IACI,oBAAoB;AACxB;;AAEA;IACI,oBAAoB;AACxB;;AAEA;IACI,oBAAoB;AACxB;;AAEA;IACI,iBAAiB;AACrB;;AAEA,yBAAyB;AACzB;IACI,gBAAgB;IAChB,kBAAkB;IAClB,mBAAmB;IACnB,eAAe;IACf,mBAAmB;IACnB,mBAAmB;IACnB,yBAAyB;IACzB,wCAAwC;AAC5C;;;AAGA,0BAA0B;AAC1B;IACI,iBAAiB;IACjB,uBAAuB;IACvB,mBAAmB;IACnB,iBAAiB;IACjB,eAAe;IACf,mBAAmB;IACnB,mBAAmB;IACnB,yBAAyB;IACzB,wCAAwC;AAC5C;;AAEA,gBAAgB;AAChB;IACI,aAAa;IACb,8BAA8B;IAC9B,eAAe;AACnB;;AAEA,gBAAgB;AAChB;IACI,qBAAqB;AACzB;;AAEA,iBAAiB;AACjB;IACI,cAAc;IACd,2BAA2B;IAC3B,iBAAiB;IACjB,gBAAgB;IAChB,cAAc;AAClB;;AAEA,mDAAmD;AACnD;IACI,aAAa;IACb,SAAS;IACT,mBAAmB;AACvB;;AAEA,wBAAwB;AACxB;IACI,OAAO;IACP,WAAW;IACX,mBAAmB;IACnB,kBAAkB;IAClB,gBAAgB;IAChB,aAAa;AACjB;;AAEA;IACI,gBAAgB;IAChB,WAAW;IACX,YAAY;IACZ,mBAAmB;IACnB,kBAAkB;IAClB,eAAe;IACf,iCAAiC;AACrC;;AAEA;IACI,mBAAmB;AACvB;;AAEA,yBAAyB;AACzB;IACI,WAAW;IACX,eAAe;IACf,yBAAyB;IACzB,kBAAkB;IAClB,iBAAiB;IACjB,cAAc;AAClB;;AAEA,mBAAmB;AACnB;IACI,WAAW;IACX,eAAe;IACf,yBAAyB;IACzB,kBAAkB;IAClB,iBAAiB;IACjB,iBAAiB;IACjB,cAAc;IACd,eAAe;AACnB;;AAEA,qBAAqB;AACrB;IACI,aAAa;IACb,cAAc;IACd,oBAAoB;IACpB,yBAAyB;IACzB,kBAAkB;IAClB,eAAe;AACnB;;AAEA,2BAA2B;AAC3B;IACI,qBAAqB;AACzB;;AAEA;IACI,oBAAoB;AACxB;;AAEA,uBAAuB;AACvB;IACI,aAAa;IACb,mBAAmB;IACnB,mBAAmB;AACvB;;AAEA,gCAAgC;AAChC;IACI,gBAAgB;IAChB,kBAAkB;IAClB,aAAa;IACb,mBAAmB;IACnB,yCAAyC;AAC7C;;AAEA;IACI,iBAAiB;IACjB,gBAAgB;IAChB,cAAc;IACd,qBAAqB;AACzB;;AAEA;IACI,iBAAiB;IACjB,gBAAgB;IAChB,cAAc;AAClB;;AAEA,2BAA2B;AAC3B;IACI;QACI,0BAA0B;QAC1B,aAAa;IACjB;;IAEA;QACI,UAAU;IACd;AACJ;;AAEA,iCAAiC;AACjC;IACI,qBAAqB;AACzB;;AAEA;IACI,qBAAqB;IACrB,aAAa;IACb,6CAA6C;AACjD;;AAEA,2BAA2B;AAC3B;IACI,0BAA0B;AAC9B;;AAEA,oCAAoC;AACpC;IACI,yBAAyB;IACzB,gBAAgB;AACpB;;AAEA,8BAA8B;AAC9B;IACI,mBAAmB;IACnB,aAAa;IACb,kBAAkB;IAClB,mBAAmB;AACvB;;AAEA;IACI,mBAAmB;IACnB,aAAa;IACb,kBAAkB;AACtB;;AAEA;YACY,qDAAqD;YACrD,YAAY;YACZ,eAAe;YACf,gBAAgB;YAChB,kBAAkB;YAClB,YAAY;YACZ,mBAAmB;YACnB,eAAe;YACf,yBAAyB;YACzB,mBAAmB;YACnB,yCAAyC;YACzC,yBAAyB;YACzB,kBAAkB;YAClB,gBAAgB;QACpB;AACR;IACI,2BAA2B;IAC3B,0CAA0C;AAC9C;;AAEA;IACI,wBAAwB;IACxB,yCAAyC;AAC7C;;AAEA;IACI,WAAW;IACX,kBAAkB;IAClB,MAAM;IACN,WAAW;IACX,WAAW;IACX,YAAY;IACZ,qFAAqF;IACrF,0BAA0B;AAC9B;;AAEA;IACI,UAAU;AACd;;AAEA;YACY,qDAAqD;YACrD,YAAY;YACZ,eAAe;YACf,gBAAgB;YAChB,kBAAkB;YAClB,YAAY;YACZ,mBAAmB;YACnB,eAAe;YACf,yBAAyB;YACzB,mBAAmB;YACnB,yCAAyC;YACzC,yBAAyB;YACzB,kBAAkB;YAClB,gBAAgB;QACpB;;AAER;IACI,2BAA2B;IAC3B,0CAA0C;AAC9C;;AAEA;IACI,wBAAwB;IACxB,yCAAyC;AAC7C;;AAEA;IACI,WAAW;IACX,kBAAkB;IAClB,MAAM;IACN,WAAW;IACX,WAAW;IACX,YAAY;IACZ,qFAAqF;IACrF,0BAA0B;AAC9B;;AAEA;IACI,UAAU;AACd;AACA;IACI,mBAAmB;IACnB,uBAAuB;IACvB,aAAa;AACjB;;AAEA;IACI,aAAa;IACb,uBAAuB;IACvB,oBAAoB;AACxB;;AAEA;IACI,qBAAqB;AACzB;;;AAGA;IACI,qBAAqB;IACrB,mBAAmB;AACvB;;AAEA;IACI,aAAa;AACjB;;AAEA;IACI,aAAa,EAAE,8BAA8B,EAAE,mBAAmB;AACtE;;AAEA;IACI;QACI,0BAA0B;IAC9B;AACJ;;AAEA;IACI;QACI,0BAA0B;IAC9B;IACA;QACI,iBAAiB;IACrB;IACA;QACI,gBAAgB;IACpB;AACJ;;AAEA;IACI,kCAAkC;AACtC;;AAEA;IACI,gBAAgB;IAChB,iBAAiB;AACrB;;AAEA,wCAAwC","sourcesContent":["/* style.css - Scaling Insanity */\n\n/* Import ultrascale-playbook base styles and add transformers-specific styling */\n/* Define colors */\n:root {\n    --distill-gray: rgb(107, 114, 128);\n    --distill-gray-light: rgb(185, 185, 185);\n    --distill-gray-lighter: rgb(228, 228, 228);\n    --distill-gray-lightest: rgb(245, 245, 245);\n    --distill-blue: #007BFF;\n}\n\n/* Container for the controls */\n[id^=\"plot-\"] {\n    display: flex;\n    flex-direction: column;\n    align-items: center;\n    gap: 15px; /* Adjust the gap between controls as needed */\n}\n[id^=\"plot-\"] figure {\n    margin-bottom: 0px;\n    margin-top: 0px;\n    padding: 0px;\n}\n.plotly_caption {\n    font-style: italic;\n    margin-top: 10px;\n}\n\n.plotly_controls {\n    display: flex;\n    flex-wrap: wrap;\n    flex-direction: row;\n    justify-content: center;\n    align-items: flex-start;\n    gap: 30px;\n}\n\n\n.plotly_input_container {\n    display: flex;\n    align-items: center;\n    flex-direction: column;\n    gap: 10px;\n}\n\n/* Style for the select dropdown */\n.plotly_input_container > select {\n    padding: 2px 4px;\n    /* border: 1px solid #ccc; */\n    line-height: 1.5em;\n    text-align: center;\n    border-radius: 4px;\n    font-size: 12px;\n    background-color: var(--distill-gray-lightest);\n    outline: none;\n}\n\n/* Style for the range input */\n\n.plotly_slider {\n    display: flex;\n    align-items: center;\n    gap: 10px;\n}\n\n.plotly_slider > input[type=\"range\"] {\n    -webkit-appearance: none;\n    height: 2px;\n    background: var(--distill-gray-light);\n    border-radius: 5px;\n    outline: none;\n}\n\n.plotly_slider > span {\n    font-size: 14px;\n    line-height: 1.6em;\n    min-width: 16px;\n}\n\n.plotly_slider > input[type=\"range\"]::-webkit-slider-thumb {\n    -webkit-appearance: none;\n    appearance: none;\n    width: 18px;\n    height: 18px;\n    border-radius: 50%;\n    background: var(--distill-blue);\n    cursor: pointer;\n}\n\n.plotly_slider > input[type=\"range\"]::-moz-range-thumb {\n    width: 18px;\n    height: 18px;\n    border-radius: 50%;\n    background: var(--distill-blue);\n    cursor: pointer;\n}\n\n/* Style for the labels */\n.plotly_input_container > label {\n    font-size: 14px;\n    font-weight: bold;\n}\n\n.main-plot-container {\n    margin-top: 21px;\n    margin-bottom: 35px;\n}\n\n.main-plot-container > figure {\n    display: block !important;\n    /* Let this be handled by graph-container */\n    margin-bottom: 0px;\n    margin-top: 0px;\n}\n.main-plot-container > div {\n    display: none !important;\n}\n\n\n@media (min-width: 768px) {\n    .main-plot-container > figure {\n        display: none !important;\n    }\n    .main-plot-container > div {\n        display: flex !important;\n    }\n}\n\nd-byline .byline {\n  grid-template-columns: 1fr;\n  grid-column: text;\n  font-size: 0.9rem;\n  line-height: 1.8em;\n}\n\n@media (min-width: 768px) {\n  d-byline .byline {\n    grid-template-columns: 5fr 1fr 1fr;\n  }\n}\n\n#title-plot {\n    margin-top: 0px;\n    margin-bottom: 0px;\n}\n\nd-contents > nav a.active {\n    text-decoration: underline;\n}\n\n@media (max-width: 1199px) {\n    d-contents {\n        display: none;\n        background: white;\n        justify-self: start;\n        align-self: start;\n        padding-bottom: 0.5em;\n        margin-bottom: 1em;\n        padding-left: 0.25em;\n        border-bottom: 1px solid rgba(0, 0, 0, 0.1);\n        border-bottom-width: 1px;\n        border-bottom-style: solid;\n        border-bottom-color: rgba(0, 0, 0, 0.1);\n        overflow-y: scroll;\n        height: calc(100vh - 40px);\n        scrollbar-width: none;\n        z-index: -100;\n    }\n}\n\nd-contents a:hover {\n    border-bottom: none;\n}\n\ntoc-title {\n    font-weight: bold;\n    font-size: 1.2em;\n    color: #333;\n}\n\ntoggle-icon {\n    transition: transform 0.3s;\n}\n\ntoggle-icon.collapsed {\n    transform: rotate(90deg);\n}\n\n.toc-content {\n    margin-top: 15px;\n    overflow: hidden;\n    /* max-height: 1000px; */\n    transition: max-height 0.3s ease-out;\n}\n\n.toc-content.collapsed {\n    max-height: 0;\n    margin-top: 0;\n}\n\n@media (min-width: 1200px) {\n    d-article {\n        /* Ensure d-article does not prevent sticky positioning */\n        overflow: visible;\n    }\n\n    d-contents {\n        align-self: start;\n        background: white;\n        grid-column-start: 1 !important;\n        grid-column-end: 4 !important;\n        grid-row: auto / span 6;\n        justify-self: end;\n        margin-top: 0em;\n        padding-right: 3em;\n        padding-left: 2em;\n        /* border-right: 1px solid rgba(0, 0, 0, 0.1);\n        border-right-width: 1px;\n        border-right-style: solid;\n        border-right-color: rgba(0, 0, 0, 0.1); */\n        position: -webkit-sticky; /* For Safari */\n        position: sticky;\n        top: 10px; /* Adjust this value if needed */\n        overflow-y: auto;\n        height: calc(100vh - 40px);\n        scrollbar-width: none;\n        transition: max-height 0.3s ease-out;\n        z-index: -100;\n    }\n}\n\nd-contents nav h3 {\n    margin-top: 0;\n    margin-bottom: 1em;\n}\n\nd-contents nav div div {\n    color: rgba(0, 0, 0, 0.8);\n    font-weight: bold;\n}\n\nd-contents nav a {\n    color: rgba(0, 0, 0, 0.8);\n    border-bottom: none;\n    text-decoration: none;\n}\n\nd-contents li {\n    list-style-type: none;\n}\n\nd-contents ul, d-article d-contents ul {\n    padding-left: 1em;\n}\n\nd-contents nav ul li {\n    margin-bottom: .25em;\n}\n\nd-contents nav a:hover {\n    text-decoration: underline solid rgba(0, 0, 0, 0.6);\n}\n\nd-contents nav ul {\n    margin-top: 0;\n    margin-bottom: 6px;\n}\n\n\nd-contents nav > div {\n    display: block;\n    outline: none;\n    margin-bottom: 0.5em;\n}\n\nd-contents nav > div > a {\n    font-size: 13px;\n    font-weight: 600;\n}\n\nd-article aside {\n    margin-bottom: 1em;\n}\n\nd-article img {\n    max-width: 100%;\n}\n\n@media (min-width: 768px) {\n    d-article aside {\n        margin-bottom: 0;\n    }\n}\n\nd-contents nav > div > a:hover,\nd-contents nav > ul > li > a:hover {\n    text-decoration: none;\n}\n\n.note-box {\n    background-color: #f6f8fa;\n    border-left: 4px solid #444444;\n    padding: 1rem;\n    margin: 1rem 0;  /* Keep this modest margin */\n    border-radius: 6px;\n    /* Add this to ensure the box only takes up needed space */\n    display: inline-block;\n}\n\n.note-box-title {\n    margin: 0;\n    color: #444444;\n    font-weight: 600;\n    font-size: 1em;\n}\n\n.note-box-content {\n    margin-top: 0.5rem;\n    margin-bottom: 0;  /* Ensure no bottom margin */\n    color: #24292f;\n    font-size: 0.9em;\n    line-height: 1.5em;\n}\n\n/* For dark mode support */\n@media (prefers-color-scheme: dark) {\n    .note-box {\n        background-color: #1c1c1c;\n        border-left-color: #888888;\n    }\n    .note-box-title {\n        color: #888888;\n    }\n    .note-box-content {\n        color: #d4d4d4;\n    }\n}\n\nd-article {\n    font-size: 1.0em;\n}\n\n.figure-legend {\n    font-size: 0.9em;\n    font-style: italic;\n    color: var(--distill-gray);\n    line-height: 1.5em;\n}\n\nd-code {\n    font-size: 12px;\n}\n\n.large-image-background {\n        width: 100vw;\n        padding-top: 10px;\n        padding-bottom: 10px;\n        margin-left: calc(-50vw + 50%);\n        margin-right: calc(-50vw + 50%);\n        background: white;\n        height: fit-content; /* This will make it match the image height */\n        display: flex;\n        justify-content: center; /* This will center your image */\n}\n\n.large-image-background-transparent {\n    /* width: 100vw; */\n    padding-top: 10px;\n    padding-bottom: 10px;\n    /* margin-left: calc(-50vw + 50%); */\n    margin-left:-100px;\n    margin-right: -100px;\n    /* margin-right: calc(-50vw + 50%); */\n    /* background: white; */\n    height: fit-content; /* This will make it match the image height */\n    display: flex;\n    justify-content: center; /* This will center your image */\n}\n\n.boxed-image {\n    padding: 0.5rem;\n    background: white;\n    border-radius: 12px;\n    border: 1px solid #e5e7eb;\n    box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);\n}\n\nd-article li {\n    margin-bottom: 0.0em;\n}\n\nd-article ul ul {\n    margin-bottom: 0.0em;\n}\n\nd-article ol ol {\n    margin-bottom: 0.0em;\n}\n\nd-article hr {\n    grid-column: text;\n}\n\n/* Memory visualization */\n#graph-all {\n    min-width: 500px;\n    margin-right: 10px;\n    margin-bottom: 2rem;\n    padding: 0.5rem;\n    background: #f9fafb;\n    border-radius: 12px;\n    border: 1px solid #e5e7eb;\n    box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);\n}\n\n\n/* Main container styles */\n#controls {\n    max-width: 1200px;\n    /* margin: 2rem auto; */\n    margin-bottom: 2rem;\n    margin-left: 10px;\n    padding: 0.6rem;\n    background: #f9fafb;\n    border-radius: 12px;\n    border: 1px solid #e5e7eb;\n    box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);\n}\n\n/* Grid layout */\n#controls {\n    display: grid;\n    grid-template-columns: 1fr 1fr;\n    /* gap: 2rem; */\n}\n\n/* Cell styles */\n.cell {\n    margin-bottom: 0.2rem;\n}\n\n/* Label styles */\nlabel {\n    display: block;\n    /* margin-bottom: 0.5rem; */\n    font-size: 0.8rem;\n    font-weight: 500;\n    color: #374151;\n}\n\n/* Input container for range + number combination */\n.input-container {\n    display: flex;\n    gap: 1rem;\n    align-items: center;\n}\n\n/* Range input styling */\ninput[type=\"range\"] {\n    flex: 1;\n    height: 6px;\n    background: #e5e7eb;\n    border-radius: 3px;\n    appearance: none;\n    outline: none;\n}\n\ninput[type=\"range\"]::-webkit-slider-thumb {\n    appearance: none;\n    width: 16px;\n    height: 16px;\n    background: #3b82f6;\n    border-radius: 50%;\n    cursor: pointer;\n    transition: background 0.15s ease;\n}\n\ninput[type=\"range\"]::-webkit-slider-thumb:hover {\n    background: #2563eb;\n}\n\n/* Number input styling */\ninput[type=\"number\"] {\n    width: 80px;\n    padding: 0.5rem;\n    border: 1px solid #e5e7eb;\n    border-radius: 6px;\n    font-size: 0.9rem;\n    color: #374151;\n}\n\n/* Select styling */\nselect {\n    width: 100%;\n    padding: 0.5rem;\n    border: 1px solid #e5e7eb;\n    border-radius: 6px;\n    background: white;\n    font-size: 0.9rem;\n    color: #374151;\n    cursor: pointer;\n}\n\n/* Checkbox styling */\ninput[type=\"checkbox\"] {\n    width: 1.2rem;\n    height: 1.2rem;\n    margin-right: 0.5rem;\n    border: 2px solid #e5e7eb;\n    border-radius: 4px;\n    cursor: pointer;\n}\n\n/* Column specific styles */\n.column-1 {\n    padding-right: 0.5rem;\n}\n\n.column-2 {\n    padding-left: 0.5rem;\n}\n\n/* Checkbox container */\n.checkbox-container {\n    display: flex;\n    align-items: center;\n    margin-bottom: 1rem;\n}\n\n/* Memory visualization styles */\n.memory-block {\n    background: #fff;\n    border-radius: 8px;\n    padding: 1rem;\n    margin-bottom: 1rem;\n    box-shadow: 0 2px 4px rgba(0, 0, 0, 0.05);\n}\n\n.memory-title {\n    font-size: 1.1rem;\n    font-weight: 500;\n    color: #374151;\n    margin-bottom: 0.5rem;\n}\n\n.memory-value {\n    font-size: 1.5rem;\n    font-weight: 600;\n    color: #3b82f6;\n}\n\n/* Responsive adjustments */\n@media (max-width: 768px) {\n    #controls {\n        grid-template-columns: 1fr;\n        padding: 1rem;\n    }\n\n    .column-1, .column-2 {\n        padding: 0;\n    }\n}\n\n/* Hover states and transitions */\ninput:hover, select:hover {\n    border-color: #3b82f6;\n}\n\ninput:focus, select:focus {\n    border-color: #2563eb;\n    outline: none;\n    box-shadow: 0 0 0 2px rgba(59, 130, 246, 0.1);\n}\n\n/* Add smooth transitions */\ninput, select, button {\n    transition: all 0.15s ease;\n}\n\n/* Preset dropdown special styling */\nselect[name=\"presets\"] {\n    background-color: #f3f4f6;\n    font-weight: 500;\n}\n\n/* Memory graph enhancements */\n.activation-memory {\n    background: #dbeafe;\n    padding: 1rem;\n    border-radius: 8px;\n    margin-bottom: 1rem;\n}\n\n.gradient-memory {\n    background: #ede9fe;\n    padding: 1rem;\n    border-radius: 8px;\n}\n\n.order-button-second {\n            background: linear-gradient(135deg, #6DB4C4, #D4A5B8);\n            color: white;\n            font-size: 18px;\n            font-weight: 600;\n            padding: 20px 20px;\n            border: none;\n            border-radius: 12px;\n            cursor: pointer;\n            text-transform: uppercase;\n            letter-spacing: 1px;\n            box-shadow: 0 4px 15px rgba(0, 0, 0, 0.2);\n            transition: all 0.3s ease;\n            position: relative;\n            overflow: hidden;\n        }\n.order-button-second:hover {\n    transform: translateY(-2px);\n    box-shadow: 0 6px 20px rgba(0, 0, 0, 0.25);\n}\n\n.order-button:active {\n    transform: translateY(0);\n    box-shadow: 0 2px 10px rgba(0, 0, 0, 0.2);\n}\n\n.order-button-second::before {\n    content: '';\n    position: absolute;\n    top: 0;\n    left: -100%;\n    width: 100%;\n    height: 100%;\n    background: linear-gradient(135deg, rgba(255, 255, 255, 0.2), rgba(255, 255, 255, 0));\n    transition: left 0.5s ease;\n}\n\n.order-button-second:hover::before {\n    left: 100%;\n}\n\n.order-button {\n            background: linear-gradient(135deg, #6DB4C4, #D4A5B8);\n            color: white;\n            font-size: 18px;\n            font-weight: 600;\n            padding: 16px 32px;\n            border: none;\n            border-radius: 12px;\n            cursor: pointer;\n            text-transform: uppercase;\n            letter-spacing: 1px;\n            box-shadow: 0 4px 15px rgba(0, 0, 0, 0.2);\n            transition: all 0.3s ease;\n            position: relative;\n            overflow: hidden;\n        }\n\n.order-button:hover {\n    transform: translateY(-2px);\n    box-shadow: 0 6px 20px rgba(0, 0, 0, 0.25);\n}\n\n.order-button:active {\n    transform: translateY(0);\n    box-shadow: 0 2px 10px rgba(0, 0, 0, 0.2);\n}\n\n.order-button::before {\n    content: '';\n    position: absolute;\n    top: 0;\n    left: -100%;\n    width: 100%;\n    height: 100%;\n    background: linear-gradient(135deg, rgba(255, 255, 255, 0.2), rgba(255, 255, 255, 0));\n    transition: left 0.5s ease;\n}\n\n.order-button:hover::before {\n    left: 100%;\n}\n.order-button-container-second {\n    /* display: flex; */\n    justify-content: center;\n    margin: 0px 0;\n}\n\n.order-button-container {\n    display: flex;\n    justify-content: center;\n    margin: 0px 0 40px 0;\n}\n\nd-article img {\n    width: 100%!important;\n}\n\n\niframe, .js-plotly-plot {\n    width: 100%!important;\n    margin-bottom: 20px;\n}\n\n.modebar-container {\n    display: none;\n}\n\n#graph-container {\n    display: grid; grid-template-columns: 1fr 1fr; align-items: center;\n}\n\n@media (max-width: 768px) {\n    #graph-container {\n        grid-template-columns: 1fr;\n    }\n}\n\n@media (max-width: 1024px) {\n    #graph-container {\n        grid-template-columns: 1fr;\n    }\n    #graph-all {\n        margin-right: 0px;\n    }\n    #controls {\n        margin-left: 0px;\n    }\n}\n\n.main-plot-container svg {\n    background: transparent !important;\n}\n\n.large-image-background-transparent {\n    margin-left: 0px;\n    margin-right: 0px;\n}\n\n/* Import transformers-specific styles */\n@import url('./transformers-custom.css');"],"sourceRoot":""}]);
 // Exports
 /* harmony default export */ const __WEBPACK_DEFAULT_EXPORT__ = (___CSS_LOADER_EXPORT___);
 
@@ -1973,11 +1973,11 @@ var update = injectStylesIntoStyleTag_default()(style/* default */.A, options);
        /* harmony default export */ const src_style = (style/* default */.A && style/* default */.A.locals ? style/* default */.A.locals : undefined);
 
 ;// ./src/index.js
-// Main JavaScript file for transformers playthrough
+// Main JavaScript file for Scaling Insanity
 
 
 // Import any additional functionality
-console.log('Transformers Playthrough loaded');
+console.log('Scaling Insanity loaded');
 
 // Add any custom JavaScript functionality here
 document.addEventListener('DOMContentLoaded', function () {

dist/style.css

@@ -1,4 +1,4 @@
-/* style.css - Transformers Playthrough */
+/* style.css - Scaling Insanity */
 
 /* Import ultrascale-playbook base styles and add transformers-specific styling */
 /* Define colors */

src/fragments/tp-plan.html

@@ -1,27 +1,24 @@
+<pre><code class="language-python"># In the model's config (example: ERNIE 4.5-style decoder blocks)
+base_model_tp_plan = {
+    "layers.*.self_attn.q_proj": "colwise",
+    "layers.*.self_attn.k_proj": "colwise",
+    "layers.*.self_attn.v_proj": "colwise",
+    "layers.*.self_attn.o_proj": "rowwise",
+    "layers.*.mlp.gate_proj": "colwise",
+    "layers.*.mlp.up_proj":   "colwise",
+    "layers.*.mlp.down_proj": "rowwise",
+}
 
-    <pre><code class="language-python"># In the model's config (example: ERNIE 4.5-style decoder blocks)
-  base_model_tp_plan = {
-      "layers.*.self_attn.q_proj": "colwise",
-      "layers.*.self_attn.k_proj": "colwise",
-      "layers.*.self_attn.v_proj": "colwise",
-      "layers.*.self_attn.o_proj": "rowwise",
-      "layers.*.mlp.gate_proj": "colwise",
-      "layers.*.mlp.up_proj":   "colwise",
-      "layers.*.mlp.down_proj": "rowwise",
-  }
-  
-  # Runtime
-  import torch
-  from transformers import AutoModelForCausalLM, AutoTokenizer
-  
-  model_id = "your/model-or-local-checkpoint"
-  model = AutoModelForCausalLM.from_pretrained(
-      model_id,
-      dtype=torch.bfloat16,
-      tp_plan=base_model_tp_plan,   # <-- plan defined above
-  )
-  tok = AutoTokenizer.from_pretrained(model_id)
-  inputs = tok("Hello", return_tensors="pt").to(model.device)
-  out = model(**inputs)</code></pre>
+# Runtime
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
 
-  
+model_id = "your/model-or-local-checkpoint"
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    dtype=torch.bfloat16,
+    tp_plan=base_model_tp_plan,   # <-- plan defined above
+)
+tok = AutoTokenizer.from_pretrained(model_id)
+inputs = tok("Hello", return_tensors="pt").to(model.device)
+out = model(**inputs)</code></pre>

src/fragments/warmup_demo.html

@@ -1,247 +1,240 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-    <meta charset="UTF-8">
-    <meta name="viewport" content="width=device-width, initial-scale=1.0">
-    <style>
-        body {
-            font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
-            margin: 0;
-            padding: 20px;
-            background-color: #f5f5f5;
-        }
-
-        .container {
-            max-width: 1200px;
-            margin: 0 auto;
-            background: white;
-            border-radius: 12px;
-            padding: 30px;
-            box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
-        }
-
-        h1 {
-            text-align: center;
-            color: #333;
-            margin-bottom: 10px;
-        }
-
-        .subtitle {
-            text-align: center;
-            color: #666;
-            margin-bottom: 30px;
-            font-size: 16px;
-        }
-
-        .demo-container {
-            display: flex;
-            gap: 40px;
-            margin-bottom: 30px;
-        }
-
-        .side {
-            flex: 1;
-            border: 2px solid #ddd;
-            border-radius: 8px;
-            padding: 20px;
-            background: #fafafa;
-        }
-
-        .side h2 {
-            text-align: center;
-            margin-top: 0;
-            color: #333;
-        }
-
-        .no-warmup h2 {
-            color: #d63384;
-        }
-
-        .with-warmup h2 {
-            color: #198754;
-        }
-
-        .memory-area {
-            height: 400px;
-            border: 2px dashed #ccc;
-            border-radius: 6px;
-            padding: 10px;
-            margin: 20px 0;
-            background: #fff;
-            position: relative;
-            overflow: hidden;
-        }
-
-        .layer-box {
-            width: 80px;
-            height: 30px;
-            border: 2px solid #666;
-            border-radius: 4px;
-            margin: 3px;
-            display: inline-block;
-            position: relative;
-            background: #fff;
-            transition: all 0.3s ease;
-        }
-
-        .layer-box.allocating {
-            background: #e9ecef;
-            border-color: #adb5bd;
-        }
-
-        .layer-box.allocating::after {
-            content: "malloc";
-            position: absolute;
-            top: 50%;
-            left: 50%;
-            transform: translate(-50%, -50%);
-            font-size: 10px;
-            color: #666;
-            font-weight: bold;
-        }
-
-        .layer-box.loaded {
-            background: #d1e7dd;
-            border-color: #198754;
-        }
-
-        .layer-box.loaded::after {
-            content: "data";
-            position: absolute;
-            top: 50%;
-            left: 50%;
-            transform: translate(-50%, -50%);
-            font-size: 10px;
-            color: #198754;
-            font-weight: bold;
-        }
-
-        .warmup-container {
-            width: 100%;
-            height: 60px;
-            border: 3px solid #666;
-            border-radius: 6px;
-            margin-bottom: 20px;
-            background: #fff;
-            position: relative;
-            overflow: hidden;
-        }
-
-        .warmup-container.allocated {
-            border-color: #0d6efd;
-            background: #e7f1ff;
-        }
-
-        .warmup-container::before {
-            content: "Pre-allocated Memory Pool";
-            position: absolute;
-            top: 50%;
-            left: 50%;
-            transform: translate(-50%, -50%);
-            font-size: 14px;
-            color: #666;
-            font-weight: bold;
-            z-index: 1;
-        }
-
-        .warmup-container.allocated::before {
-            color: #0d6efd;
-        }
-
-        .warmup-fill {
-            height: 100%;
-            background: linear-gradient(90deg, #198754, #20c997);
-            width: 0%;
-            transition: width 0.5s ease;
-            border-radius: 3px;
-            position: relative;
-            z-index: 2;
-        }
-
-        .warmup-fill::after {
-            content: "Layer Data Loading";
-            position: absolute;
-            top: 50%;
-            left: 50%;
-            transform: translate(-50%, -50%);
-            font-size: 12px;
-            color: white;
-            font-weight: bold;
-            white-space: nowrap;
-        }
-
-        .timing {
-            text-align: center;
-            font-size: 24px;
-            font-weight: bold;
-            margin: 15px 0;
-            min-height: 30px;
-        }
-
-        .no-warmup .timing {
-            color: #d63384;
-        }
-
-        .with-warmup .timing {
-            color: #198754;
-        }
-
-        .controls {
-            text-align: center;
-            margin: 30px 0;
-        }
-
-        .btn {
-            background: #0d6efd;
-            color: white;
-            border: none;
-            padding: 12px 24px;
-            border-radius: 6px;
-            font-size: 16px;
-            cursor: pointer;
-            margin: 0 10px;
-            transition: background 0.3s ease;
-        }
-
-        .btn:hover {
-            background: #0b5ed7;
-        }
-
-        .btn:disabled {
-            background: #6c757d;
-            cursor: not-allowed;
-        }
-
-
-        .description {
-            background: #f8f9fa;
-            padding: 15px;
-            border-radius: 6px;
-            margin-top: 15px;
-            font-size: 14px;
-            line-height: 1.5;
-        }
-
-        .phase-indicator {
-            font-size: 14px;
-            color: #666;
-            text-align: center;
-            margin-top: 10px;
-            min-height: 20px;
-        }
-
-        .layer-counter {
-            text-align: center;
-            font-size: 16px;
-            color: #495057;
-            margin: 10px 0;
-        }
-    </style>
-</head>
-<body>
+<style>
+    .warmup-demo body {
+        font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
+        margin: 0;
+        padding: 20px;
+        background-color: #f5f5f5;
+    }
+
+    .warmup-demo .container {
+        max-width: 1200px;
+        margin: 0 auto;
+        background: white;
+        border-radius: 12px;
+        padding: 30px;
+        box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
+    }
+
+    .warmup-demo h1 {
+        text-align: center;
+        color: #333;
+        margin-bottom: 10px;
+    }
+
+    .warmup-demo .subtitle {
+        text-align: center;
+        color: #666;
+        margin-bottom: 30px;
+        font-size: 16px;
+    }
+
+    .warmup-demo .demo-container {
+        display: flex;
+        gap: 40px;
+        margin-bottom: 30px;
+    }
+
+    .warmup-demo .side {
+        flex: 1;
+        border: 2px solid #ddd;
+        border-radius: 8px;
+        padding: 20px;
+        background: #fafafa;
+    }
+
+    .warmup-demo .side h2 {
+        text-align: center;
+        margin-top: 0;
+        color: #333;
+    }
+
+    .warmup-demo .no-warmup h2 {
+        color: #d63384;
+    }
+
+    .warmup-demo .with-warmup h2 {
+        color: #198754;
+    }
+
+    .warmup-demo .memory-area {
+        height: 400px;
+        border: 2px dashed #ccc;
+        border-radius: 6px;
+        padding: 10px;
+        margin: 20px 0;
+        background: #fff;
+        position: relative;
+        overflow: hidden;
+    }
+
+    .warmup-demo .layer-box {
+        width: 80px;
+        height: 30px;
+        border: 2px solid #666;
+        border-radius: 4px;
+        margin: 3px;
+        display: inline-block;
+        position: relative;
+        background: #fff;
+        transition: all 0.3s ease;
+    }
+
+    .warmup-demo .layer-box.allocating {
+        background: #e9ecef;
+        border-color: #adb5bd;
+    }
+
+    .warmup-demo .layer-box.allocating::after {
+        content: "malloc";
+        position: absolute;
+        top: 50%;
+        left: 50%;
+        transform: translate(-50%, -50%);
+        font-size: 10px;
+        color: #666;
+        font-weight: bold;
+    }
+
+    .warmup-demo .layer-box.loaded {
+        background: #d1e7dd;
+        border-color: #198754;
+    }
+
+    .warmup-demo .layer-box.loaded::after {
+        content: "data";
+        position: absolute;
+        top: 50%;
+        left: 50%;
+        transform: translate(-50%, -50%);
+        font-size: 10px;
+        color: #198754;
+        font-weight: bold;
+    }
+
+    .warmup-demo .warmup-container {
+        width: 100%;
+        height: 60px;
+        border: 3px solid #666;
+        border-radius: 6px;
+        margin-bottom: 20px;
+        background: #fff;
+        position: relative;
+        overflow: hidden;
+    }
+
+    .warmup-demo .warmup-container.allocated {
+        border-color: #0d6efd;
+        background: #e7f1ff;
+    }
+
+    .warmup-demo .warmup-container::before {
+        content: "Pre-allocated Memory Pool";
+        position: absolute;
+        top: 50%;
+        left: 50%;
+        transform: translate(-50%, -50%);
+        font-size: 14px;
+        color: #666;
+        font-weight: bold;
+        z-index: 1;
+    }
+
+    .warmup-demo .warmup-container.allocated::before {
+        color: #0d6efd;
+    }
+
+    .warmup-demo .warmup-fill {
+        height: 100%;
+        background: linear-gradient(90deg, #198754, #20c997);
+        width: 0%;
+        transition: width 0.5s ease;
+        border-radius: 3px;
+        position: relative;
+        z-index: 2;
+    }
+
+    .warmup-demo .warmup-fill::after {
+        content: "Layer Data Loading";
+        position: absolute;
+        top: 50%;
+        left: 50%;
+        transform: translate(-50%, -50%);
+        font-size: 12px;
+        color: white;
+        font-weight: bold;
+        white-space: nowrap;
+    }
+
+    .warmup-demo .timing {
+        text-align: center;
+        font-size: 24px;
+        font-weight: bold;
+        margin: 15px 0;
+        min-height: 30px;
+    }
+
+    .warmup-demo .no-warmup .timing {
+        color: #d63384;
+    }
+
+    .warmup-demo .with-warmup .timing {
+        color: #198754;
+    }
+
+    .warmup-demo .controls {
+        text-align: center;
+        margin: 30px 0;
+    }
+
+    .warmup-demo .btn {
+        background: #0d6efd;
+        color: white;
+        border: none;
+        padding: 12px 24px;
+        border-radius: 6px;
+        font-size: 16px;
+        cursor: pointer;
+        margin: 0 10px;
+        transition: background 0.3s ease;
+    }
+
+    .warmup-demo .btn:hover {
+        background: #0b5ed7;
+    }
+
+    .warmup-demo .btn:disabled {
+        background: #6c757d;
+        cursor: not-allowed;
+    }
+
+    .warmup-demo .description {
+        background: #f8f9fa;
+        padding: 15px;
+        border-radius: 6px;
+        margin-top: 15px;
+        font-size: 14px;
+        line-height: 1.5;
+    }
+
+    .warmup-demo .phase-indicator {
+        font-size: 14px;
+        color: #666;
+        text-align: center;
+        margin-top: 10px;
+        min-height: 20px;
+    }
+
+    .warmup-demo .layer-counter {
+        text-align: center;
+        font-size: 16px;
+        color: #495057;
+        margin: 10px 0;
+    }
+</style>
+
+<div class="warmup-demo">
     <div class="container">
-        <h1>CUDA Caching Allocator Warmup Visualization</h1>
-        <p class="subtitle">Demonstrating memory allocation patterns during model loading</p>
+        <p class="subtitle">Mem allocation patterns during model loading</p>
 
         <div class="controls">
             <button class="btn" id="startBtn" onclick="startDemo()">Start Animation</button>
@@ -285,137 +278,121 @@
             </div>
         </div>
     </div>
+</div>
+
+<script>
+    let animationSpeed = 1 / 2.4;
+    let isRunning = false;
+    const totalLayers = 10;
+
+    function startDemo() {
+        if (isRunning) return;
+        isRunning = true;
+
+        document.getElementById('startBtn').disabled = true;
+        document.getElementById('resetBtn').disabled = true;
+
+        Promise.all([
+            animateNoWarmup(),
+            animateWithWarmup()
+        ]).then(() => {
+            isRunning = false;
+            document.getElementById('startBtn').disabled = false;
+            document.getElementById('resetBtn').disabled = false;
+        });
+    }
+
+    function resetDemo() {
+        if (isRunning) return;
+
+        document.getElementById('noWarmupArea').innerHTML = '';
+        document.getElementById('warmupLayers').innerHTML = '';
+        document.getElementById('warmupFill').style.width = '0%';
+        document.getElementById('warmupContainer').classList.remove('allocated');
+
+        document.getElementById('noWarmupTime').textContent = '0.00s';
+        document.getElementById('warmupTime').textContent = '0.00s';
+
+        document.getElementById('noWarmupCounter').textContent = 'Layers loaded: 0/10';
+        document.getElementById('warmupCounter').textContent = 'Layers loaded: 0/10';
+
+        document.getElementById('noWarmupPhase').textContent = '';
+        document.getElementById('warmupPhase').textContent = '';
+    }
+
+    async function animateNoWarmup() {
+        const container = document.getElementById('noWarmupArea');
+        const timeEl = document.getElementById('noWarmupTime');
+        const counterEl = document.getElementById('noWarmupCounter');
+        const phaseEl = document.getElementById('noWarmupPhase');
+
+        let currentTime = 0;
+        const baseDelay = 200 / animationSpeed;
+
+        phaseEl.textContent = 'Loading model layers...';
+
+        for (let i = 0; i < totalLayers; i++) {
+            const layerBox = document.createElement('div');
+            layerBox.className = 'layer-box';
+            container.appendChild(layerBox);
+
+            await sleep(baseDelay * 0.3);
+            layerBox.classList.add('allocating');
+            currentTime += 0.08;
+            timeEl.textContent = currentTime.toFixed(2) + 's';
 
-    <script>
-        let animationSpeed = 1 / 2.4; // Slower animation
-        let isRunning = false;
-        const totalLayers = 10;
-
-        function startDemo() {
-            if (isRunning) return;
-            isRunning = true;
-
-            document.getElementById('startBtn').disabled = true;
-            document.getElementById('resetBtn').disabled = true;
-
-            // Start both animations simultaneously
-            Promise.all([
-                animateNoWarmup(),
-                animateWithWarmup()
-            ]).then(() => {
-                isRunning = false;
-                document.getElementById('startBtn').disabled = false;
-                document.getElementById('resetBtn').disabled = false;
-            });
-        }
-
-        function resetDemo() {
-            if (isRunning) return;
-
-            // Clear containers
-            document.getElementById('noWarmupArea').innerHTML = '';
-            document.getElementById('warmupLayers').innerHTML = '';
-            document.getElementById('warmupFill').style.width = '0%';
-            document.getElementById('warmupContainer').classList.remove('allocated');
-
-            // Reset timers
-            document.getElementById('noWarmupTime').textContent = '0.00s';
-            document.getElementById('warmupTime').textContent = '0.00s';
-
-            // Reset counters
-            document.getElementById('noWarmupCounter').textContent = 'Layers loaded: 0/10';
-            document.getElementById('warmupCounter').textContent = 'Layers loaded: 0/10';
+            await sleep(baseDelay * 0.7);
+            layerBox.classList.remove('allocating');
+            layerBox.classList.add('loaded');
+            currentTime += 0.12;
+            timeEl.textContent = currentTime.toFixed(2) + 's';
 
-            // Reset phase indicators
-            document.getElementById('noWarmupPhase').textContent = '';
-            document.getElementById('warmupPhase').textContent = '';
+            counterEl.textContent = `Layers loaded: ${i + 1}/${totalLayers}`;
         }
 
-        async function animateNoWarmup() {
-            const container = document.getElementById('noWarmupArea');
-            const timeEl = document.getElementById('noWarmupTime');
-            const counterEl = document.getElementById('noWarmupCounter');
-            const phaseEl = document.getElementById('noWarmupPhase');
-
-            let currentTime = 0;
-            const baseDelay = 200 / animationSpeed; // Base delay between operations
-
-            phaseEl.textContent = 'Loading model layers...';
-
-            for (let i = 0; i < totalLayers; i++) {
-                // Create layer box
-                const layerBox = document.createElement('div');
-                layerBox.className = 'layer-box';
-                container.appendChild(layerBox);
-
-                // Malloc phase (grey)
-                await sleep(baseDelay * 0.3);
-                layerBox.classList.add('allocating');
-                currentTime += 0.08; // malloc overhead
-                timeEl.textContent = currentTime.toFixed(2) + 's';
-
-                // Data loading phase (green)
-                await sleep(baseDelay * 0.7);
-                layerBox.classList.remove('allocating');
-                layerBox.classList.add('loaded');
-                currentTime += 0.12; // data transfer time
-                timeEl.textContent = currentTime.toFixed(2) + 's';
-
-                // Update counter
-                counterEl.textContent = `Layers loaded: ${i + 1}/${totalLayers}`;
-            }
-
-            phaseEl.textContent = 'Complete!';
-        }
+        phaseEl.textContent = 'Complete!';
+    }
 
-        async function animateWithWarmup() {
-            const container = document.getElementById('warmupLayers');
-            const timeEl = document.getElementById('warmupTime');
-            const counterEl = document.getElementById('warmupCounter');
-            const phaseEl = document.getElementById('warmupPhase');
-            const warmupContainer = document.getElementById('warmupContainer');
-            const warmupFill = document.getElementById('warmupFill');
-
-            let currentTime = 0;
-            const baseDelay = 200 / animationSpeed;
-
-            // Warmup phase
-            phaseEl.textContent = 'Warming up allocator...';
-            await sleep(baseDelay * 2);
-            warmupContainer.classList.add('allocated');
-            currentTime += 0.3; // warmup overhead
-            timeEl.textContent = currentTime.toFixed(2) + 's';
+    async function animateWithWarmup() {
+        const container = document.getElementById('warmupLayers');
+        const timeEl = document.getElementById('warmupTime');
+        const counterEl = document.getElementById('warmupCounter');
+        const phaseEl = document.getElementById('warmupPhase');
+        const warmupContainer = document.getElementById('warmupContainer');
+        const warmupFill = document.getElementById('warmupFill');
 
-            phaseEl.textContent = 'Loading model layers...';
+        let currentTime = 0;
+        const baseDelay = 200 / animationSpeed;
 
-            // Load layers directly into pre-allocated memory
-            for (let i = 0; i < totalLayers; i++) {
-                // Create layer indicator (much smaller since no malloc)
-                const layerBox = document.createElement('div');
-                layerBox.className = 'layer-box loaded';
-                layerBox.style.width = '40px';
-                layerBox.style.height = '20px';
-                container.appendChild(layerBox);
+        phaseEl.textContent = 'Warming up allocator...';
+        await sleep(baseDelay * 2);
+        warmupContainer.classList.add('allocated');
+        currentTime += 0.3;
+        timeEl.textContent = currentTime.toFixed(2) + 's';
 
-                // Update progress bar
-                const progress = ((i + 1) / totalLayers) * 100;
-                warmupFill.style.width = progress + '%';
+        phaseEl.textContent = 'Loading model layers...';
 
-                // Only data transfer time (no malloc overhead)
-                await sleep(baseDelay * 0.5);
-                currentTime += 0.08; // faster due to no malloc
-                timeEl.textContent = currentTime.toFixed(2) + 's';
+        for (let i = 0; i < totalLayers; i++) {
+            const layerBox = document.createElement('div');
+            layerBox.className = 'layer-box loaded';
+            layerBox.style.width = '40px';
+            layerBox.style.height = '20px';
+            container.appendChild(layerBox);
 
-                // Update counter
-                counterEl.textContent = `Layers loaded: ${i + 1}/${totalLayers}`;
-            }
+            const progress = ((i + 1) / totalLayers) * 100;
+            warmupFill.style.width = progress + '%';
 
-            phaseEl.textContent = 'Complete!';
-        }
+            await sleep(baseDelay * 0.5);
+            currentTime += 0.08;
+            timeEl.textContent = currentTime.toFixed(2) + 's';
 
-        function sleep(ms) {
-            return new Promise(resolve => setTimeout(resolve, ms));
+            counterEl.textContent = `Layers loaded: ${i + 1}/${totalLayers}`;
         }
-    </script>
-</body>
-</html>
+
+        phaseEl.textContent = 'Complete!';
+    }
+
+    function sleep(ms) {
+        return new Promise(resolve => setTimeout(resolve, ms));
+    }
+</script>

webpack.config.js

@@ -31,11 +31,19 @@ const loadFragmentsMap = (() => {
                         const dottedPath = 'fragment-' + nameWithoutExt.replace(/\\/g, '-').replace(/\//g, '-').replace(/\./g, '-');
                         const content = fs.readFileSync(filePath, "utf8");
                         // Minify the HTML content using swcMinifyFragment
-                        const minifiedRes = await HtmlMinimizerPlugin.swcMinifyFragment({"tmp.html": content})
-                        if (minifiedRes.errors) {
-                            console.error(minifiedRes.errors)
+                        let minifiedContent;
+                        try {
+                            const minifiedRes = await HtmlMinimizerPlugin.swcMinifyFragment({"tmp.html": content})
+                            if (minifiedRes.errors) {
+                                console.warn("HTML minification warnings:", minifiedRes.errors);
+                                minifiedContent = content; // Use original content if errors
+                            } else {
+                                minifiedContent = minifiedRes.code;
+                            }
+                        } catch (error) {
+                            console.warn(`Failed to minify fragment ${filePath}, using original content:`, error.message);
+                            minifiedContent = content; // Fallback to original content
                         }
-                        const minifiedContent = minifiedRes.code;
                         cachedFragments[dottedPath] = minifiedContent;
                     }
                 }));
@@ -325,10 +333,6 @@ module.exports = {
                 }
             ]
             }),
-            new HtmlMinimizerPlugin({
-                test: /fragments\/.*\.html$/i,
-                minify: HtmlMinimizerPlugin.swcMinifyFragment,
-            })
         ]
     },
 };