Julian Bilcke
commited on
Commit
·
11ef19f
1
Parent(s):
e020146
temporary readme
Browse files- README.md +1 -81
- README_WIP.md +97 -0
README.md
CHANGED
|
@@ -13,84 +13,4 @@ short_description: All-in-one tool for AI video training
|
|
| 13 |
|
| 14 |
# 🎥 Video Model Studio (VMS)
|
| 15 |
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
VMS is an all-in-one tool to train LoRA models for various open-source AI video models:
|
| 19 |
-
|
| 20 |
-
- Data collection from various sources
|
| 21 |
-
- Splitting videos into short single camera shots
|
| 22 |
-
- Automatic captioning
|
| 23 |
-
- Training HunyuanVideo or LTX-Video
|
| 24 |
-
|
| 25 |
-
## Similar projects
|
| 26 |
-
|
| 27 |
-
I wasn't aware of it when I started this project,
|
| 28 |
-
but there is also this: https://github.com/alisson-anjos/diffusion-pipe-ui
|
| 29 |
-
|
| 30 |
-
## Installation
|
| 31 |
-
|
| 32 |
-
VMS is built on top of Finetrainers and Gradio, and designed to run as a Hugging Face Space (but you can deploy it elsewhere if you want to).
|
| 33 |
-
|
| 34 |
-
### Full installation at Hugging Face
|
| 35 |
-
|
| 36 |
-
Easy peasy: create a Space (make sure to use the `Gradio` type/template), and push the repo. No Docker needed!
|
| 37 |
-
|
| 38 |
-
### Dev mode on Hugging Face
|
| 39 |
-
|
| 40 |
-
Enable dev mode in the space, then open VSCode in local or remote and run:
|
| 41 |
-
|
| 42 |
-
```
|
| 43 |
-
pip install -r requirements.txt
|
| 44 |
-
```
|
| 45 |
-
|
| 46 |
-
As this is not automatic, then click on "Restart" in the space dev mode UI widget.
|
| 47 |
-
|
| 48 |
-
### Full installation somewhere else
|
| 49 |
-
|
| 50 |
-
I haven't tested it, but you can try to provided Dockerfile
|
| 51 |
-
|
| 52 |
-
### Full installation in local
|
| 53 |
-
|
| 54 |
-
the full installation requires:
|
| 55 |
-
- Linux
|
| 56 |
-
- CUDA 12
|
| 57 |
-
- Python 3.10
|
| 58 |
-
|
| 59 |
-
This is because of flash attention, which is defined in the `requirements.txt` using an URL to download a prebuilt wheel (python bindings for a native library)
|
| 60 |
-
|
| 61 |
-
```bash
|
| 62 |
-
./setup.sh
|
| 63 |
-
```
|
| 64 |
-
|
| 65 |
-
### Degraded installation in local
|
| 66 |
-
|
| 67 |
-
If you cannot meet the requirements, you can:
|
| 68 |
-
|
| 69 |
-
- solution 1: fix requirements.txt to use another prebuilt wheel
|
| 70 |
-
- solution 2: manually build/install flash attention
|
| 71 |
-
- solution 3: don't use clip captioning
|
| 72 |
-
|
| 73 |
-
Here is how to do solution 3:
|
| 74 |
-
```bash
|
| 75 |
-
./setup_no_captions.sh
|
| 76 |
-
```
|
| 77 |
-
|
| 78 |
-
## Run
|
| 79 |
-
|
| 80 |
-
### Running the Gradio app
|
| 81 |
-
|
| 82 |
-
Note: please make sure you properly define the environment variables for `STORAGE_PATH` (eg. `/data/`) and `HF_HOME` (eg. `/data/huggingface/`)
|
| 83 |
-
|
| 84 |
-
```bash
|
| 85 |
-
python app.py
|
| 86 |
-
```
|
| 87 |
-
|
| 88 |
-
### Running locally
|
| 89 |
-
|
| 90 |
-
See above remarks about the environment variable.
|
| 91 |
-
|
| 92 |
-
By default `run.sh` will store stuff in `.data/` (located inside the current working directory):
|
| 93 |
-
|
| 94 |
-
```bash
|
| 95 |
-
./run.sh
|
| 96 |
-
```
|
|
|
|
| 13 |
|
| 14 |
# 🎥 Video Model Studio (VMS)
|
| 15 |
|
| 16 |
+
This project is a work in progress, not all features are working yet (there are some issue with the automatic captioning).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
README_WIP.md
ADDED
|
@@ -0,0 +1,97 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
README_WIP.md
|
| 2 |
+
---
|
| 3 |
+
title: Video Model Studio
|
| 4 |
+
emoji: 🎥
|
| 5 |
+
colorFrom: gray
|
| 6 |
+
colorTo: gray
|
| 7 |
+
sdk: gradio
|
| 8 |
+
sdk_version: 5.15.0
|
| 9 |
+
app_file: app.py
|
| 10 |
+
pinned: true
|
| 11 |
+
license: apache-2.0
|
| 12 |
+
short_description: All-in-one tool for AI video training
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# 🎥 Video Model Studio (VMS)
|
| 16 |
+
|
| 17 |
+
## Presentation
|
| 18 |
+
|
| 19 |
+
VMS is an all-in-one tool to train LoRA models for various open-source AI video models:
|
| 20 |
+
|
| 21 |
+
- Data collection from various sources
|
| 22 |
+
- Splitting videos into short single camera shots
|
| 23 |
+
- Automatic captioning
|
| 24 |
+
- Training HunyuanVideo or LTX-Video
|
| 25 |
+
|
| 26 |
+
## Similar projects
|
| 27 |
+
|
| 28 |
+
I wasn't aware of it when I started this project,
|
| 29 |
+
but there is also this: https://github.com/alisson-anjos/diffusion-pipe-ui
|
| 30 |
+
|
| 31 |
+
## Installation
|
| 32 |
+
|
| 33 |
+
VMS is built on top of Finetrainers and Gradio, and designed to run as a Hugging Face Space (but you can deploy it elsewhere if you want to).
|
| 34 |
+
|
| 35 |
+
### Full installation at Hugging Face
|
| 36 |
+
|
| 37 |
+
Easy peasy: create a Space (make sure to use the `Gradio` type/template), and push the repo. No Docker needed!
|
| 38 |
+
|
| 39 |
+
### Dev mode on Hugging Face
|
| 40 |
+
|
| 41 |
+
Enable dev mode in the space, then open VSCode in local or remote and run:
|
| 42 |
+
|
| 43 |
+
```
|
| 44 |
+
pip install -r requirements.txt
|
| 45 |
+
```
|
| 46 |
+
|
| 47 |
+
As this is not automatic, then click on "Restart" in the space dev mode UI widget.
|
| 48 |
+
|
| 49 |
+
### Full installation somewhere else
|
| 50 |
+
|
| 51 |
+
I haven't tested it, but you can try to provided Dockerfile
|
| 52 |
+
|
| 53 |
+
### Full installation in local
|
| 54 |
+
|
| 55 |
+
the full installation requires:
|
| 56 |
+
- Linux
|
| 57 |
+
- CUDA 12
|
| 58 |
+
- Python 3.10
|
| 59 |
+
|
| 60 |
+
This is because of flash attention, which is defined in the `requirements.txt` using an URL to download a prebuilt wheel (python bindings for a native library)
|
| 61 |
+
|
| 62 |
+
```bash
|
| 63 |
+
./setup.sh
|
| 64 |
+
```
|
| 65 |
+
|
| 66 |
+
### Degraded installation in local
|
| 67 |
+
|
| 68 |
+
If you cannot meet the requirements, you can:
|
| 69 |
+
|
| 70 |
+
- solution 1: fix requirements.txt to use another prebuilt wheel
|
| 71 |
+
- solution 2: manually build/install flash attention
|
| 72 |
+
- solution 3: don't use clip captioning
|
| 73 |
+
|
| 74 |
+
Here is how to do solution 3:
|
| 75 |
+
```bash
|
| 76 |
+
./setup_no_captions.sh
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
+
## Run
|
| 80 |
+
|
| 81 |
+
### Running the Gradio app
|
| 82 |
+
|
| 83 |
+
Note: please make sure you properly define the environment variables for `STORAGE_PATH` (eg. `/data/`) and `HF_HOME` (eg. `/data/huggingface/`)
|
| 84 |
+
|
| 85 |
+
```bash
|
| 86 |
+
python app.py
|
| 87 |
+
```
|
| 88 |
+
|
| 89 |
+
### Running locally
|
| 90 |
+
|
| 91 |
+
See above remarks about the environment variable.
|
| 92 |
+
|
| 93 |
+
By default `run.sh` will store stuff in `.data/` (located inside the current working directory):
|
| 94 |
+
|
| 95 |
+
```bash
|
| 96 |
+
./run.sh
|
| 97 |
+
```
|