Merge branch 'main' of https://huggingface.co/Alpha-VLLM/Lumina-Next-SFT-diffusers into main

README.md

@@ -27,7 +27,7 @@ Our generative model has `Next-DiT` as the backbone, the text encoder is the `Ge
 
 ## 📰 News
 
-- [2024-06-21] 🎉🎉🎉 We have supported diffusers to load the `Lumina-Next-SFT` model. https://huggingface.co/Alpha-VLLM/Lumina-Next-SFT-diffusers
+- [2024-06-23] 🎉🎉🎉 We have supported diffusers to load the `Lumina-Next-SFT` model. https://huggingface.co/Alpha-VLLM/Lumina-Next-SFT-diffusers
 
 - [2024-06-08] 🎉🎉🎉 We have released the `Lumina-Next-SFT` model.
 
@@ -43,180 +43,54 @@ More checkpoints of our model will be released soon~
 
 | Resolution | Next-DiT Parameter| Text Encoder | Prediction | Download URL  |
 | ---------- | ----------------------- | ------------ | -----------|-------------- |
-| 1024       | 2B             |    [Gemma-2B](https://huggingface.co/google/gemma-2b)  |   Rectified Flow | [hugging face](https://huggingface.co/Alpha-VLLM/Lumina-Next-SFT) |
+| 1024  | 2B  | [Gemma-2B](https://huggingface.co/google/gemma-2b)  | Rectified Flow | [hugging face](https://huggingface.co/Alpha-VLLM/Lumina-Next-SFT-diffusers) |
 
 ## Installation
 
-Before installation, ensure that you have a working ``nvcc``
-
-```bash
-# The command should work and show the same version number as in our case. (12.1 in our case).
-nvcc --version
-```
+### 1. Create a conda environment and install PyTorch
 
-On some outdated distros (e.g., CentOS 7), you may also want to check that a late enough version of
-``gcc`` is available
+Note: You may want to adjust the CUDA version [according to your driver version](https://docs.nvidia.com/deploy/cuda-compatibility/#default-to-minor-version).
 
 ```bash
-# The command should work and show a version of at least 6.0.
-# If not, consult distro-specific tutorials to obtain a newer version or build manually.
-gcc --version
+conda create -n Lumina_T2X -y
+conda activate Lumina_T2X
+conda install python=3.11 pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia -y
 ```
 
-Downloading Lumina-T2X repo from GitHub:
+### 2. Install dependencies
 
 ```bash
-git clone https://github.com/Alpha-VLLM/Lumina-T2X
+pip install diffusers huggingface_hub
 ```
 
-### 1. Create a conda environment and install PyTorch
-
-Note: You may want to adjust the CUDA version [according to your driver version](https://docs.nvidia.com/deploy/cuda-compatibility/#default-to-minor-version).
-
-  ```bash
-  conda create -n Lumina_T2X -y
-  conda activate Lumina_T2X
-  conda install python=3.11 pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia -y
-  ```
-
-### 2. Install dependencies
-
-  ```bash
-  pip install diffusers fairscale accelerate tensorboard transformers gradio torchdiffeq click
-  ```
-
-  or you can use
-
-  ```bash
-  cd lumina_next_t2i
-  pip install -r requirements.txt
-  ```
-
 ### 3. Install ``flash-attn``
 
-  ```bash
-  pip install flash-attn --no-build-isolation
-  ```
-
-### 4. Install [nvidia apex](https://github.com/nvidia/apex) (optional)
-
->[!Warning]
-> While Apex can improve efficiency, it is *not* a must to make Lumina-T2X work.
->
-> Note that Lumina-T2X works smoothly with either:
-> + Apex not installed at all; OR
-> + Apex successfully installed with CUDA and C++ extensions.
->
-> However, it will fail when:
-> + A Python-only build of Apex is installed.
-> 
-> If the error `No module named 'fused_layer_norm_cuda'` appears, it typically means you are using a Python-only build of Apex. To resolve this, please run `pip uninstall apex`, and Lumina-T2X should then function correctly.
-
-You can clone the repo and install following the official guidelines (note that we expect a full
-build, i.e., with CUDA and C++ extensions)
-
 ```bash
-pip install ninja
-git clone https://github.com/NVIDIA/apex
-cd apex
-# if pip >= 23.1 (ref: https://pip.pypa.io/en/stable/news/#v23-1) which supports multiple `--config-settings` with the same key... 
-pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./
-# otherwise
-pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./
+pip install flash-attn --no-build-isolation
 ```
 
 ## Inference
 
-To ensure that our generative model is ready to use right out of the box, we provide a user-friendly CLI program and a locally deployable Web Demo site.
-
-### CLI
-
-1. Install Lumina-Next-T2I
-
-```bash
-pip install -e .
-```
 
-2. Prepare the pre-trained model
+1. Prepare the pre-trained model
 
 ⭐⭐ (Recommended) you can use huggingface_cli to download our model:
 
 ```bash
-huggingface-cli download --resume-download Alpha-VLLM/Lumina-Next-SFT --local-dir /path/to/ckpt
+huggingface-cli download --resume-download Alpha-VLLM/Lumina-Next-SFT-diffusers --local-dir /path/to/ckpt
 ```
 
-or using git for cloning the model you want to use:
+2. Run with demo code:
 
-```bash
-git clone https://huggingface.co/Alpha-VLLM/Lumina-Next-T2I
-``` 
-
-1. Setting your personal inference configuration
-
-Update your own personal inference settings to generate different styles of images, checking `config/infer/config.yaml` for detailed settings. Detailed config structure:
-
-> `/path/to/ckpt` should be a directory containing `consolidated*.pth` and `model_args.pth`
-
-```yaml
-- settings:
-
-  model:
-    ckpt: ""
-    ckpt_lm: ""
-    token: ""
-
-  transport:
-    path_type: "Linear"             # option: ["Linear", "GVP", "VP"]
-    prediction: "velocity"          # option: ["velocity", "score", "noise"]
-    loss_weight: "velocity"         # option: [None, "velocity", "likelihood"]
-    sample_eps: 0.1
-    train_eps: 0.2
-
-  ode:
-    atol: 1e-6                      # Absolute tolerance
-    rtol: 1e-3                      # Relative tolerance
-    reverse: false                  # option: true or false
-    likelihood: false               # option: true or false
-
-  infer:
-      resolution: "1024x1024"       # option: ["1024x1024", "512x2048", "2048x512", "(Extrapolation) 1664x1664", "(Extrapolation) 1024x2048", "(Extrapolation) 2048x1024"]
-      num_sampling_steps: 60        # range: 1-1000
-      cfg_scale: 4.                 # range: 1-20
-      solver: "euler"               # option: ["euler", "dopri5", "dopri8"]
-      t_shift: 4                    # range: 1-20 (int only)
-      scaling_method: "Time-aware"  # option: ["Time-aware", "None"]
-      scale_watershed: 0.3          # range: 0.0-1.0
-      proportional_attn: true       # option: true or false
-      seed: 0                       # rnage: any number
-```
+```python
+from diffusers import LuminaText2ImgPipeline
+import torch
 
-1. Run with CLI
+pipeline = LuminaText2ImgPipeline.from_pretrained("/path/to/ckpt/Lumina-Next-SFT-diffusers", torch_dtype=torch.bfloat16).to("cuda")
 
-inference command:
-```bash
-lumina_next infer -c <config_path> <caption_here> <output_dir>
-```
+# or you can download the model using code directly
+# pipeline = LuminaText2ImgPipeline.from_pretrained("Alpha-VLLM/Lumina-Next-SFT-diffusers", torch_dtype=torch.bfloat16).to("cuda")
 
-e.g. Demo command:
-
-```bash
-cd lumina_next_t2i
-lumina_next infer -c "config/infer/settings.yaml" "a snowman of ..." "./outputs"
+image = pipeline(prompt="Upper body of a young woman in a Victorian-era outfit with brass goggles and leather straps. "
+                        "Background shows an industrial revolution cityscape with smoky skies and tall, metal structures").images[0]
 ```
-
-### Web Demo
-
-To host a local gradio demo for interactive inference, run the following command:
-
-```bash
-# `/path/to/ckpt` should be a directory containing `consolidated*.pth` and `model_args.pth`
-
-# default
-python -u demo.py --ckpt "/path/to/ckpt"
-
-# the demo by default uses bf16 precision. to switch to fp32:
-python -u demo.py --ckpt "/path/to/ckpt" --precision fp32 
-
-# use ema model
-python -u demo.py --ckpt "/path/to/ckpt" --ema
-```