upload result to HF dataset

bench/.env.example

@@ -0,0 +1,13 @@
+# Hugging Face Dataset Configuration
+# Repository to push benchmark results to (format: username/dataset-name)
+HF_DATASET_REPO=whitphx/transformersjs-performance-leaderboard-results-dev
+
+# Hugging Face API Token (required for pushing to datasets)
+# Get your token from: https://huggingface.co/settings/tokens
+HF_TOKEN=your_hf_token_here
+
+# Local benchmark results directory
+BENCHMARK_RESULTS_DIR=./benchmark-results
+
+# Server configuration
+PORT=7860

bench/bench/.gitignore

@@ -0,0 +1 @@
+.env

bench/docs/hf-dataset-integration.md

@@ -0,0 +1,215 @@
+# Hugging Face Dataset Integration
+
+The benchmark server can automatically upload results to a Hugging Face Dataset repository for centralized storage and sharing.
+
+## Features
+
+- **Automatic Upload**: Results are automatically pushed to HF Dataset when benchmarks complete
+- **File Structure Preservation**: Uses the same path structure: `{task}/{org}/{model}/{params}.json`
+- **JSON Format**: Results are stored as JSON (not JSONL) for better Dataset compatibility
+- **Overwrite Strategy**: Each configuration gets a single file that is overwritten with the latest result
+- **Error Tracking**: Failed benchmarks are also uploaded to track issues
+
+## Setup
+
+### 1. Create a Hugging Face Dataset
+
+1. Go to https://huggingface.co/new-dataset
+2. Create a new dataset (e.g., `username/transformersjs-benchmark-results`)
+3. Keep it public or private based on your needs
+
+### 2. Get Your HF Token
+
+1. Go to https://huggingface.co/settings/tokens
+2. Create a new token with `write` permissions
+3. Copy the token
+
+### 3. Configure Environment Variables
+
+Create or update `.env` file in the `bench` directory:
+
+```bash
+# Hugging Face Dataset Configuration
+HF_DATASET_REPO=whitphx/transformersjs-performance-leaderboard-results-dev
+HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+
+# Optional: Local storage directory
+BENCHMARK_RESULTS_DIR=./benchmark-results
+
+# Optional: Server port
+PORT=7860
+```
+
+**Important**: Never commit `.env` to git. It's already in `.gitignore`.
+
+## Usage
+
+Once configured, the server will automatically upload results:
+
+```bash
+# Start the server
+npm run server
+
+# You should see:
+# 📤 HF Dataset upload enabled: username/transformersjs-benchmark-results
+```
+
+When benchmarks complete, you'll see:
+
+```
+✅ Completed: abc-123 in 5.2s
+✓ Benchmark abc-123 saved to file
+✓ Uploaded to HF Dataset: feature-extraction/Xenova/all-MiniLM-L6-v2/node_warm_cpu_fp32_b1.json
+```
+
+## File Structure in HF Dataset
+
+The dataset will have the same structure as local storage:
+
+```
+feature-extraction/
+├── Xenova/
+│   ├── all-MiniLM-L6-v2/
+│   │   ├── node_warm_cpu_fp32_b1.json
+│   │   ├── node_warm_webgpu_fp16_b1.json
+│   │   └── web_warm_wasm_b1_chromium.json
+│   └── distilbert-base-uncased/
+│       └── node_warm_cpu_fp32_b1.json
+text-classification/
+└── Xenova/
+    └── distilbert-base-uncased/
+        └── node_warm_cpu_fp32_b1.json
+```
+
+## JSON Format
+
+Each file contains a single benchmark result (not multiple runs):
+
+```json
+{
+  "id": "abc-123-456",
+  "platform": "node",
+  "modelId": "Xenova/all-MiniLM-L6-v2",
+  "task": "feature-extraction",
+  "mode": "warm",
+  "repeats": 3,
+  "dtype": "fp32",
+  "batchSize": 1,
+  "device": "cpu",
+  "timestamp": 1234567890,
+  "status": "completed",
+  "result": {
+    "metrics": { ... },
+    "environment": { ... }
+  }
+}
+```
+
+## Behavior
+
+### Overwriting Results
+
+- Each benchmark configuration maps to a single file
+- New results **overwrite** the existing file
+- Only the **latest** result is kept per configuration
+- This ensures the dataset always has current data
+
+### Local vs Remote Storage
+
+- **Local (JSONL)**: Keeps history of all runs (append-only)
+- **Remote (JSON)**: Keeps only latest result (overwrite)
+
+This dual approach allows:
+- Local: Full history for analysis
+- Remote: Clean, current results for leaderboards
+
+### Failed Benchmarks
+
+Failed benchmarks are also uploaded to track:
+- Which models/configs have issues
+- Error types (memory errors, etc.)
+- Environmental context
+
+Example failed result:
+
+```json
+{
+  "id": "def-456-789",
+  "status": "failed",
+  "error": "Benchmark failed with code 1: ...",
+  "result": {
+    "error": {
+      "type": "memory_error",
+      "message": "Aborted(). Build with -sASSERTIONS for more info.",
+      "stage": "load"
+    },
+    "environment": { ... }
+  }
+}
+```
+
+## Git Commits
+
+Each upload creates a git commit in the dataset with:
+
+```
+Update benchmark: Xenova/all-MiniLM-L6-v2 (node/feature-extraction)
+
+Benchmark ID: abc-123-456
+Status: completed
+Timestamp: 2025-10-13T06:48:57.481Z
+```
+
+## Disabling Upload
+
+To disable HF Dataset upload:
+
+1. Remove `HF_TOKEN` from `.env`, or
+2. Remove both `HF_DATASET_REPO` and `HF_TOKEN`
+
+The server will show:
+
+```
+📤 HF Dataset upload disabled (set HF_DATASET_REPO and HF_TOKEN to enable)
+```
+
+## Error Handling
+
+If HF upload fails:
+- The error is logged but doesn't fail the benchmark
+- Local storage still succeeds
+- You can retry manually or fix configuration
+
+Example error:
+
+```
+✗ Failed to upload benchmark abc-123 to HF Dataset: Authentication failed
+```
+
+## API Endpoint (Future)
+
+Currently uploads happen automatically. In the future, we could add:
+
+```bash
+# Manually trigger upload of a specific result
+POST /api/benchmark/:id/upload
+
+# Re-upload all local results to HF Dataset
+POST /api/benchmarks/sync
+```
+
+## Development vs Production
+
+Use different dataset repositories for development and production:
+
+**Development** (`.env`):
+```bash
+HF_DATASET_REPO=whitphx/transformersjs-performance-leaderboard-results-dev
+```
+
+**Production** (deployed environment):
+```bash
+HF_DATASET_REPO=whitphx/transformersjs-performance-leaderboard-results
+```
+
+This allows testing without polluting production data.

bench/package-lock.json

@@ -9,7 +9,9 @@
       "version": "0.0.2",
       "dependencies": {
         "@hono/node-server": "^1.19.5",
+        "@huggingface/hub": "^2.6.12",
         "@huggingface/transformers": "^3.7.4",
+        "dotenv": "^17.2.3",
         "hono": "^4.9.10",
         "zod": "^4.1.11"
       },
@@ -485,6 +487,24 @@
         "hono": "^4"
       }
     },
+    "node_modules/@huggingface/hub": {
+      "version": "2.6.12",
+      "resolved": "https://registry.npmjs.org/@huggingface/hub/-/hub-2.6.12.tgz",
+      "integrity": "sha512-/AZN2LAtrt4B8S83/Ru4wMorHY4NPwIYXo60SkuD6c/Mr135t1UqffD1vqcqtDYVf0hhLiyVmm1LMU1CXi8iKQ==",
+      "license": "MIT",
+      "dependencies": {
+        "@huggingface/tasks": "^0.19.50"
+      },
+      "bin": {
+        "hfjs": "dist/cli.js"
+      },
+      "engines": {
+        "node": ">=18"
+      },
+      "optionalDependencies": {
+        "cli-progress": "^3.12.0"
+      }
+    },
     "node_modules/@huggingface/jinja": {
       "version": "0.5.1",
       "resolved": "https://registry.npmjs.org/@huggingface/jinja/-/jinja-0.5.1.tgz",
@@ -494,6 +514,12 @@
         "node": ">=18"
       }
     },
+    "node_modules/@huggingface/tasks": {
+      "version": "0.19.50",
+      "resolved": "https://registry.npmjs.org/@huggingface/tasks/-/tasks-0.19.50.tgz",
+      "integrity": "sha512-kHrfiDsJttkuwpdp7PgFiFHaK9rj+COJTIZ+221gk9vdF4B5QySW7hQT4aOGkwwZP+4qbXGhjMNGg/bxOq+LwA==",
+      "license": "MIT"
+    },
     "node_modules/@huggingface/transformers": {
       "version": "3.7.5",
       "resolved": "https://registry.npmjs.org/@huggingface/transformers/-/transformers-3.7.5.tgz",
@@ -1349,6 +1375,16 @@
         "undici-types": "~7.14.0"
       }
     },
+    "node_modules/ansi-regex": {
+      "version": "5.0.1",
+      "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz",
+      "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==",
+      "license": "MIT",
+      "optional": true,
+      "engines": {
+        "node": ">=8"
+      }
+    },
     "node_modules/boolean": {
       "version": "3.2.0",
       "resolved": "https://registry.npmjs.org/boolean/-/boolean-3.2.0.tgz",
@@ -1365,6 +1401,19 @@
         "node": ">=18"
       }
     },
+    "node_modules/cli-progress": {
+      "version": "3.12.0",
+      "resolved": "https://registry.npmjs.org/cli-progress/-/cli-progress-3.12.0.tgz",
+      "integrity": "sha512-tRkV3HJ1ASwm19THiiLIXLO7Im7wlTuKnvkYaTkyoAPefqjNg7W7DHKUlGRxy9vxDvbyCYQkQozvptuMkGCg8A==",
+      "license": "MIT",
+      "optional": true,
+      "dependencies": {
+        "string-width": "^4.2.3"
+      },
+      "engines": {
+        "node": ">=4"
+      }
+    },
     "node_modules/define-data-property": {
       "version": "1.1.4",
       "resolved": "https://registry.npmjs.org/define-data-property/-/define-data-property-1.1.4.tgz",
@@ -1414,6 +1463,25 @@
       "integrity": "sha512-T0NIuQpnTvFDATNuHN5roPwSBG83rFsuO+MXXH9/3N1eFbn4wcPjttvjMLEPWJ0RGUYgQE7cGgS3tNxbqCGM7g==",
       "license": "MIT"
     },
+    "node_modules/dotenv": {
+      "version": "17.2.3",
+      "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-17.2.3.tgz",
+      "integrity": "sha512-JVUnt+DUIzu87TABbhPmNfVdBDt18BLOWjMUFJMSi/Qqg7NTYtabbvSNJGOJ7afbRuv9D/lngizHtP7QyLQ+9w==",
+      "license": "BSD-2-Clause",
+      "engines": {
+        "node": ">=12"
+      },
+      "funding": {
+        "url": "https://dotenvx.com"
+      }
+    },
+    "node_modules/emoji-regex": {
+      "version": "8.0.0",
+      "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz",
+      "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==",
+      "license": "MIT",
+      "optional": true
+    },
     "node_modules/es-define-property": {
       "version": "1.0.1",
       "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz",
@@ -1616,6 +1684,16 @@
         "node": ">=16.9.0"
       }
     },
+    "node_modules/is-fullwidth-code-point": {
+      "version": "3.0.0",
+      "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz",
+      "integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==",
+      "license": "MIT",
+      "optional": true,
+      "engines": {
+        "node": ">=8"
+      }
+    },
     "node_modules/json-stringify-safe": {
       "version": "5.0.1",
       "resolved": "https://registry.npmjs.org/json-stringify-safe/-/json-stringify-safe-5.0.1.tgz",
@@ -2003,6 +2081,34 @@
       "integrity": "sha512-Oo+0REFV59/rz3gfJNKQiBlwfHaSESl1pcGyABQsnnIfWOFt6JNj5gCog2U6MLZ//IGYD+nA8nI+mTShREReaA==",
       "license": "BSD-3-Clause"
     },
+    "node_modules/string-width": {
+      "version": "4.2.3",
+      "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz",
+      "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==",
+      "license": "MIT",
+      "optional": true,
+      "dependencies": {
+        "emoji-regex": "^8.0.0",
+        "is-fullwidth-code-point": "^3.0.0",
+        "strip-ansi": "^6.0.1"
+      },
+      "engines": {
+        "node": ">=8"
+      }
+    },
+    "node_modules/strip-ansi": {
+      "version": "6.0.1",
+      "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz",
+      "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==",
+      "license": "MIT",
+      "optional": true,
+      "dependencies": {
+        "ansi-regex": "^5.0.1"
+      },
+      "engines": {
+        "node": ">=8"
+      }
+    },
     "node_modules/tar": {
       "version": "7.5.1",
       "resolved": "https://registry.npmjs.org/tar/-/tar-7.5.1.tgz",

bench/package.json

@@ -16,7 +16,9 @@
   },
   "dependencies": {
     "@hono/node-server": "^1.19.5",
+    "@huggingface/hub": "^2.6.12",
     "@huggingface/transformers": "^3.7.4",
+    "dotenv": "^17.2.3",
     "hono": "^4.9.10",
     "zod": "^4.1.11"
   },

bench/src/server/hf-dataset.ts

@@ -0,0 +1,121 @@
+/**
+ * Hugging Face Dataset Integration
+ *
+ * Uploads benchmark results to a Hugging Face Dataset repository.
+ * - Preserves file path structure: {task}/{org}/{model}/{params}.json
+ * - Uses JSON format (not JSONL)
+ * - Overwrites existing files instead of appending
+ */
+
+import { uploadFile, listFiles } from "@huggingface/hub";
+import { generateBenchmarkPath, type BenchmarkSettings } from "../core/benchmark-id.js";
+import type { QueuedBenchmark } from "./queue.js";
+
+export interface HFDatasetConfig {
+  repo: string;
+  token: string;
+}
+
+export class HFDatasetUploader {
+  private config: HFDatasetConfig | null = null;
+
+  constructor(config?: HFDatasetConfig) {
+    if (config && config.repo && config.token) {
+      this.config = config;
+    }
+  }
+
+  /**
+   * Check if HF Dataset upload is enabled
+   */
+  isEnabled(): boolean {
+    return this.config !== null;
+  }
+
+  /**
+   * Get the HF Dataset file path for a benchmark
+   */
+  private getHFFilePath(benchmark: QueuedBenchmark): string {
+    const settings: BenchmarkSettings = {
+      platform: benchmark.platform,
+      modelId: benchmark.modelId,
+      task: benchmark.task,
+      mode: benchmark.mode,
+      device: benchmark.device,
+      dtype: benchmark.dtype,
+      batchSize: benchmark.batchSize,
+      browser: benchmark.browser,
+      headed: benchmark.headed,
+    };
+
+    const { fullPath } = generateBenchmarkPath(settings);
+    // Replace .jsonl extension with .json
+    return fullPath.replace(/\.jsonl$/, ".json");
+  }
+
+  /**
+   * Upload a benchmark result to HF Dataset
+   * Overwrites the file if it already exists
+   */
+  async uploadResult(benchmark: QueuedBenchmark): Promise<void> {
+    if (!this.config) {
+      throw new Error("HF Dataset upload is not configured");
+    }
+
+    const filePath = this.getHFFilePath(benchmark);
+
+    // Convert benchmark to JSON string
+    const content = JSON.stringify(benchmark, null, 2);
+    const blob = new Blob([content], { type: "application/json" });
+
+    try {
+      // Upload file to HF Dataset (overwrites if exists)
+      await uploadFile({
+        repo: {
+          type: "dataset",
+          name: this.config.repo,
+        },
+        credentials: { accessToken: this.config.token },
+        file: {
+          path: filePath,
+          content: blob,
+        },
+        commitTitle: `Update benchmark: ${benchmark.modelId} (${benchmark.platform}/${benchmark.task})`,
+        commitDescription: `Benchmark ID: ${benchmark.id}\nStatus: ${benchmark.status}\nTimestamp: ${new Date(benchmark.timestamp).toISOString()}`,
+      });
+
+      console.log(`✓ Uploaded to HF Dataset: ${filePath}`);
+    } catch (error: any) {
+      console.error(`✗ Failed to upload to HF Dataset: ${filePath}`, error.message);
+      throw error;
+    }
+  }
+
+  /**
+   * List all files in the HF Dataset
+   */
+  async listAllFiles(): Promise<string[]> {
+    if (!this.config) {
+      throw new Error("HF Dataset upload is not configured");
+    }
+
+    try {
+      const files = [];
+      for await (const file of listFiles({
+        repo: {
+          type: "dataset",
+          name: this.config.repo,
+        },
+        credentials: { accessToken: this.config.token },
+      })) {
+        if (file.path.endsWith(".json")) {
+          files.push(file.path);
+        }
+      }
+      return files;
+    } catch (error: any) {
+      console.error("✗ Failed to list files from HF Dataset", error.message);
+      throw error;
+    }
+  }
+}

bench/src/server/index.ts

@@ -3,17 +3,38 @@ import { cors } from "hono/cors";
 import { serve } from "@hono/node-server";
 import { BenchmarkQueue, BenchmarkRequest } from "./queue.js";
 import { BenchmarkStorage } from "./storage.js";
+import { HFDatasetUploader } from "./hf-dataset.js";
 import { randomUUID } from "crypto";
 import { z } from "zod";
+import { config as dotenvConfig } from "dotenv";
+
+// Load environment variables
+dotenvConfig();
 
 const app = new Hono();
 const queue = new BenchmarkQueue();
 const storage = new BenchmarkStorage();
 
+// Initialize HF Dataset uploader if configured
+const hfUploader = new HFDatasetUploader(
+  process.env.HF_DATASET_REPO && process.env.HF_TOKEN
+    ? {
+        repo: process.env.HF_DATASET_REPO,
+        token: process.env.HF_TOKEN,
+      }
+    : undefined
+);
+
+if (hfUploader.isEnabled()) {
+  console.log(`📤 HF Dataset upload enabled: ${process.env.HF_DATASET_REPO}`);
+} else {
+  console.log("📤 HF Dataset upload disabled (set HF_DATASET_REPO and HF_TOKEN to enable)");
+}
+
 // Enable CORS for development
 app.use("/*", cors());
 
-// Store completed benchmarks to file
+// Store completed benchmarks to file and upload to HF Dataset
 queue.on("completed", async (benchmark) => {
   try {
     await storage.appendResult(benchmark);
@@ -21,6 +42,16 @@ queue.on("completed", async (benchmark) => {
   } catch (error) {
     console.error(`✗ Failed to save benchmark ${benchmark.id}:`, error);
   }
+
+  // Upload to HF Dataset if enabled
+  if (hfUploader.isEnabled()) {
+    try {
+      await hfUploader.uploadResult(benchmark);
+    } catch (error) {
+      console.error(`✗ Failed to upload benchmark ${benchmark.id} to HF Dataset:`, error);
+      // Don't fail the whole operation if HF upload fails
+    }
+  }
 });
 
 queue.on("failed", async (benchmark) => {
@@ -30,6 +61,15 @@ queue.on("failed", async (benchmark) => {
   } catch (error) {
     console.error(`✗ Failed to save failed benchmark ${benchmark.id}:`, error);
   }
+
+  // Also upload failed benchmarks to HF Dataset if enabled (for tracking failures)
+  if (hfUploader.isEnabled()) {
+    try {
+      await hfUploader.uploadResult(benchmark);
+    } catch (error) {
+      console.error(`✗ Failed to upload failed benchmark ${benchmark.id} to HF Dataset:`, error);
+    }
+  }
 });
 
 // Log queue events