Upload README.md with huggingface_hub

README.md

@@ -1,12 +1,12 @@
 
 ---
 
+library_name: transformers
+license: llama3.2
 base_model: meta-llama/Llama-3.2-3B-Instruct
-library_name: sft
 datasets:
 - lianghsun/tw-emergency-medicine-bench
 - lianghsun/tw-legal-nlp
-- lianghsun/tw-structured-law-article
 - lianghsun/tw-legal-synthetic-qa
 - lianghsun/tw-law-article-qa
 - lianghsun/tw-judgment-qa
@@ -16,7 +16,13 @@ tags:
 - TW
 - Taiwan
 - ROC
-license: llama3.2
+- llama-factory
+- full
+- generated_from_trainer
+model-index:
+- name: train_2024-10-17
+  results: []
+new_version: lianghsun/Llama-3.2-Taiwan-Legal-3B-Instruct
 language:
 - zh
 pipeline_tag: text-generation
@@ -32,13 +38,24 @@ This is quantized version of [lianghsun/Llama-3.2-Taiwan-Legal-3B-Instruct](http
 # Original Model Card
 
 
-# Model Card for Model lianghsun/Llama-3.2-Taiwan-Legal-3B-Instruct
-![Training Status](https://img.shields.io/badge/training-in%20progress-orange) ![Epoch Progress](https://img.shields.io/badge/epoch-10%25-yellow) ![Welcome Feedback](https://img.shields.io/badge/welcome-feedback-brightgreen)
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
 
+# Model Card for Model lianghsun/Llama-3.2-Taiwan-Legal-3B-Instruct
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/618dc56cbc345ca7bf95f3cd/W6-UDg0_cNm4WJVlR9tiD.png)
+
 基於 [meta-llama/Llama-3.2-3B-Instruct](meta-llama/Llama-3.2-3B-Instruct) 模型，透過中華民國台灣法律條文及判決書等相關資料集進行微調。
 
+## Model Update History
+
+| Update Date  | Model Version         | Key Changes                         |
+|--------------|-----------------------|-------------------------------------|
+| 2024-10-17   | v1.1.0                | Experimental fine-tuning on v1.0.0 with added legal code data from the Republic of China (Taiwan) |
+| 2024-10-10   | v1.0.0                | Full model training completed, but missing legal code data for the Republic of China (Taiwan) |
+| 2024-09-27   | v0.1.0                | Model v0.1.0 released, but training was interrupted after 3 epochs due to lack of compute resources |
+
+
 ## Model Details
 
 ### Model Description
@@ -63,16 +80,19 @@ This is quantized version of [lianghsun/Llama-3.2-Taiwan-Legal-3B-Instruct](http
 ### Direct Use
 
 <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
 此模型可以直接用於理解和生成繁體中文法律文本，適合需要處理台灣法律相關問題的應用場景。模型預設的指令和回應能夠有效提供法律資訊、釐清法律條文、並生成符合法律專業的回應。其直接使用範圍包括但不限於法律資訊查詢、法律文本摘要、和基本的法條對話。
 
 ### Downstream Use
 
 <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
 經過微調後，該模型可用於更具體的法律任務，如自動判決書分析、法律實體識別（NER）、法規編號轉換，以及法律合規審查輔助。此模型可以無縫集成至法律數據科學應用或法律技術（LegalTech）系統中，幫助法律專業人士或企業提升工作效率。
 
 ### Out-of-Scope Use
 
 <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
 該模型並不適用於非法律相關領域的生成任務，且不應用於進行可能涉及誤導或錯誤的法律建議，尤其是在未經專業審查的情況下。避免將模型用於未經授權或非法用途，如生成具爭議性或具偏見的法律建議。
 
 ## Bias, Risks, and Limitations
@@ -101,6 +121,8 @@ This is quantized version of [lianghsun/Llama-3.2-Taiwan-Legal-3B-Instruct](http
 
 ## How to Get Started with the Model
 
+<!-- Use the code below to get started with the model. -->
+
 ### Using vLLM
 
 要使用 [vLLM Docker image](https://docs.vllm.ai/en/latest/serving/deploying_with_docker.html) 來啟動此模型，您可以按照以下操作：
@@ -113,60 +135,54 @@ docker run --runtime nvidia --gpus all \
     vllm/vllm-openai:latest \
     --model lianghsun/Llama-3.2-Taiwan-Legal-3B-Instruct
 ```
-
 ## Training Details
 
-### Training Data
+### Training Data (for v1.1.0)
 
 - [lianghsun/tw-legal-nlp](https://huggingface.co/datasets/lianghsun/tw-legal-nlp)
-- [lianghsun/tw-structured-law-article](https://huggingface.co/datasets/lianghsun/tw-structured-law-article)
 - [lianghsun/tw-legal-synthetic-qa](https://huggingface.co/datasets/lianghsun/tw-legal-synthetic-qa)
 - [lianghsun/tw-law-article-qa](https://huggingface.co/datasets/lianghsun/tw-law-article-qa)
 - [lianghsun/tw-judgment-qa](https://huggingface.co/datasets/lianghsun/tw-judgment-qa)
 - [lianghsun/tw-bar-examination-2020-chat](https://huggingface.co/datasets/lianghsun/tw-bar-examination-2020-chat)
 - [lianghsun/tw-emergency-medicine-bench](https://huggingface.co/datasets/lianghsun/tw-emergency-medicine-bench)
 
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+### Training procedure
 
 #### Preprocessing
 
 無。基本上我們並沒有針對 [meta-llama/Llama-3.2-3B-Instruct](meta-llama/Llama-3.2-3B-Instruct) 做任何的預訓練或更改其模型架構；Tokenizer 也是採用原生所提供的。
 
-#### Training Hyperparameters
-
-- **Training regime**: bf16 mixed precision
-- **Learning rate**: 5e-06
-- **Batch size**: 6 (per device)
-- **Epochs**: 10 *(Note: 由於算力成本考量，在 `epoch: 0.78` 就停止訓練)*
-- **Gradient accumulation steps**: 8
-- **Cutoff length**: 2048
-- **Scheduler**: cosine
-- **Optimizer**: adamw_torch
-- **Max gradient norm**: 1.0
-- **Warmup steps**: 100
-- **Logging steps**: 5
-- **Save steps**: 1000
-- **Max samples**: 1,500,000
-
-#### Speeds, Sizes, Times
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-*Note: 由於算力成本考量，在 `epoch: 0.78` 就停止訓練，故以下資訊會有部份缺陷及不具參考價值*
-
-- **Duration**: 6h 12m 13s
-- **Train runtime**: 22,333 seconds
-- **Train samples per second**: `nan`
-- **Train steps per second**: `nan`
-- **Total training FLOPs**: `nan`
-- **Train loss**: `nan` (final loss: 0.3377)
+#### Training hyperparameters (for v1.1.0)
+
+The following hyperparameters were used during training:
+
+- **learning_rate:** 0.0004378 (value at epoch 3.9)
+- **train_batch_size:** 12
+- **eval_batch_size:** Not specified
+- **seed:** Not specified
+- **distributed_type:** single-GPU
+- **num_devices:** 1
+- **gradient_accumulation_steps:** 512
+- **total_train_batch_size:** 6144 (train_batch_size * gradient_accumulation_steps)
+- **optimizer:** AdamW
+- **lr_scheduler_type:** cosine
+- **lr_scheduler_warmup_steps:** 100
+- **num_epochs:** 15
+- **grad_norm:** 0.0899 (value at epoch 3.9)
+- **global_step:** 645
+
+### Speeds, Sizes, Times (for v1.1.0)
+
+- **Duration**: 92h 27m 40s
+- **Train runtime**: 92h 27m 40s
+- **Train samples per second**: Not directly available
+- **Train steps per second**: Approximately 0.002 steps/s
+- **Total training FLOPs**: Not directly provided
+- **Train loss**: 0.0512 (at epoch 3.9)
 
 ## Evaluation
 
 <!-- This section describes the evaluation protocols and provides the results. -->
-**Note**: ..(WIP)..
 
 ### Testing Data, Factors & Metrics
 
@@ -198,6 +214,8 @@ docker run --runtime nvidia --gpus all \
 
 ## Model Examination
 
+<!-- Relevant interpretability work for the model goes here -->
+
 ### 法條回覆
 
 **Note**: ..(WIP)..
@@ -210,14 +228,13 @@ docker run --runtime nvidia --gpus all \
 
 **Note**: ..(WIP)..
 
+## Environmental Impact (for v1.1.0)
 
-## Environmental Impact
-
-- **Hardware Type:** 8 x NVIDIA A100 40GB
-- **Hours used:** 6.03 hours 
-- **Cloud Provider:** Google Cloud Platform
-- **Compute Region:** us-central1-c
-- **Carbon Emitted:** `0.86 kgCO$_2$eq`                        
+- **Hardware Type:** 1 x NVIDIA H100 NVL 80GB
+- **Hours used:** 92h 27m 40s
+- **Cloud Provider:** N/A
+- **Compute Region:** N/A
+- **Carbon Emitted:** N/A                        
 
 ## Technical Specifications
 
@@ -227,9 +244,9 @@ docker run --runtime nvidia --gpus all \
 
 ### Compute Infrastructure
 
-#### Hardware
+#### Hardware (for v1.1.0)
 
-- 8 x NVIDIA A100 40GB
+- 1 x NVIDIA H100 NVL 80GB
 
 #### Software
 
@@ -241,6 +258,7 @@ docker run --runtime nvidia --gpus all \
 
 ## Glossary
 
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
 無。
 
 ## More Information
@@ -248,8 +266,6 @@ docker run --runtime nvidia --gpus all \
 ### 算力
 儘管我們已準備了許多關於中華民國台灣法律領域的資料集，但由於算力資源有限，**無法將所有資料集進行完整訓練**（是的，我們並沒有將全部資料集都進行訓練，僅取出被認為最基礎的法律文本），導致模型尚未達到最佳表現。因此，目前的 checkpoint 是基於有限資源的版本。如果您有意願贊助算力，歡迎與我聯繫。我相信，若能將更多已準備但尚未納入訓練的法律語料進行微調，該模型將能達到繁體中文法律領域的最佳表現。
 
-**另外**，和 [lianghsun/Llama-3.2-Taiwan-Legal-1B-Instruct](https://huggingface.co/lianghsun/Llama-3.2-Taiwan-Legal-1B-Instruct) 相較之下，又因為算力成本考量， [lianghsun/Llama-3.2-Taiwan-Legal-3B-Instruct](https://huggingface.co/lianghsun/Llama-3.2-Taiwan-Legal-3B-Instruct) 未訓練到 1 epoch，所以在表現上又更加不如預期。
-
 ### 持績更新
 此模型如有進一步資源，將會不定期更新。
 
@@ -263,4 +279,7 @@ docker run --runtime nvidia --gpus all \
 
 ### Framework versions
 
-- PEFT 0.12.0
+- Transformers 4.45.2
+- Pytorch 2.4.1+cu121
+- Datasets 2.21.0
+- Tokenizers 0.20.0