README.md

metadata

title: Transformers.js Benchmark Server
emoji: 🚀
colorFrom: blue
colorTo: green
sdk: docker
pinned: false

Transformers.js Benchmark Server

A REST API server for running and managing Transformers.js benchmarks on both Node.js and browser (via Playwright) platforms.

Features

• Queue-based benchmark execution: Submit benchmarks via REST API and process them sequentially
• Multi-platform support: Run benchmarks on Node.js or in browsers (via Playwright)
• Result persistence: Store benchmark results in JSONL format
• Validation: Request validation using Zod schemas
• CLI client: Command-line interface for interacting with the server

API Endpoints

Submit Benchmark

POST /api/benchmark
Content-Type: application/json

{
  "platform": "node",          # "node" or "web"
  "modelId": "Xenova/all-MiniLM-L6-v2",
  "task": "feature-extraction",
  "mode": "warm",              # "warm" or "cold"
  "repeats": 3,
  "dtype": "fp32",             # fp32, fp16, q8, int8, uint8, q4, bnb4, q4f16
  "batchSize": 1,
  "device": "webgpu",          # For web: "webgpu" or "wasm"
  "browser": "chromium",       # For web: "chromium", "firefox", "webkit"
  "headed": false
}

Get Benchmark Result

GET /api/benchmark/:id

List All Benchmarks

GET /api/benchmarks

Queue Status

GET /api/queue

Clear Results

DELETE /api/benchmarks

Architecture

.
├── bench/          # Benchmark server and execution logic
│   ├── src/
│   │   ├── core/      # Shared types and utilities
│   │   ├── node/      # Node.js benchmark runner
│   │   ├── web/       # Browser benchmark runner (Playwright)
│   │   └── server/    # REST API server (Hono)
│   └── package.json
├── client/         # CLI client for the server
│   ├── src/
│   │   └── index.ts   # Yargs-based CLI
│   └── package.json
└── Dockerfile

Development

Running locally

1. Install dependencies:

cd bench && npm install
cd ../client && npm install

2. Install Playwright browsers:

cd bench && npm run bench:install

3. Start the server:

cd bench && npm run server

4. Use the CLI client:

cd client && npm run cli -- submit Xenova/all-MiniLM-L6-v2 feature-extraction --wait

Deployment

This server is designed to run on Hugging Face Spaces using Docker. The Dockerfile includes all necessary dependencies including Playwright browsers for running web-based benchmarks.

License

MIT