README.md

metadata

title: TuRTLe Leaderboard
emoji: 🐢
colorFrom: gray
colorTo: green
sdk: gradio
app_file: app.py
pinned: true
license: apache-2.0
short_description: A Unified Evaluation of LLMs for RTL Generation.
sdk_version: 5.39.0

Quick Introduction

Prerequisites

• Python 3.11 or higher (required by the project)
• uv for managing dependencies (optional but recommended)

Installing uv (optional)

On macOS and Linux:

curl -LsSf https://astral.sh/uv/install.sh | sh

Install dependencies:

uv sync

# or using regular python
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Deploy Locally

$ uv run app.py # or python3 app.py
* Running on local URL:  http://127.0.0.1:7860
* To create a public link, set `share=True` in `launch()`.

Then on localhost http://127.0.0.1:7860 you should have the leaderboard running

Add new models

If you are from outside of HPAI you must directly modify the results/results_icarus.json and results/results_verilator.json files.

If you are from HPAI, you can add your model onto our shared .csv file of results and follow these steps:

1. Modify the config/model_metadata.py file, MODELS dictionary to include a new entry for your model

   For example, if we wish to include the classic GPT2 model, we would add the following metadata:

   MODELS = {
       ...
   
       "GPT2": ModelMetadata(
           "https://huggingface.co/openai-community/gpt2",  # model url
           0.13,  # params (in B)
           "Coding",  # model type: "General", "Coding", or "RTL-Specific"
           "V1",  # release of the TuRTLe Leaderboard: "V1", "V2", or "V3"
           "Dense"  # model architecture: "Dense" or "Reasoning"
       ),
   }
   

2. Parse the CSV files onto JSON, which is what the Leaderboard will take as ground truth

   $ uv run -m results.parse results/results_v3_mlcad_icarus.csv # will generate results/results_v3_mlcad_icarus.json
   $ uv run -m results.parse results/results_v3_mlcad_verilator.csv # will generate results/results_v3_mlcad_verilator.json
   

   The application is hardcoded to look for results_icarus.json and results_verilator.json. Rename the files you just created:

   $ mv results/results_v3_mlcad_icarus.json results/results_icarus.json
   $ mv results/results_v3_mlcad_verilator.json results/results_verilator.json
   

3. Compute the aggregated scores

   This will generate the corresponding aggregated_scores files that the leaderboard uses for some of its views.

   $ uv run results/compute_agg_results.py results/results_v3_mlcad_icarus.csv
   $ uv run results/compute_agg_results.py results/results_v3_mlcad_verilator.csv
   

   This will create aggregated_scores_v3_mlcad_icarus.csv and aggregated_scores_v3_mlcad_verilator.csv. Rename them to what the application expects:

   $ mv results/aggregated_scores_v3_mlcad_icarus.csv results/aggregated_scores_icarus.csv
   $ mv results/aggregated_scores_v3_mlcad_verilator.csv results/aggregated_scores_verilator.csv
   

License

This project is licensed under the Apache License 2.0.
See the LICENSE and NOTICE files for more details.