Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,10 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/samples/eligen_example_1.png filter=lfs diff=lfs merge=lfs -text
+assets/samples/eligen_example_2.png filter=lfs diff=lfs merge=lfs -text
+assets/samples/eligen_example_3.png filter=lfs diff=lfs merge=lfs -text
+assets/samples/eligen_example_4.png filter=lfs diff=lfs merge=lfs -text
+assets/samples/eligen_example_5.png filter=lfs diff=lfs merge=lfs -text
+assets/samples/poster.png filter=lfs diff=lfs merge=lfs -text
+assets/title.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,78 @@
+---
+license: apache-2.0
+---
+# Qwen-Image Precise Region Control Model
+
+![](./assets/title.png)
+
+## Model Introduction
+
+This model is a precise region control model trained based on [Qwen-Image](https://www.modelscope.cn/models/Qwen/Qwen-Image), with a LoRA architecture. It enables control over the position and shape of each entity by taking as input both textual descriptions and regional conditions (mask maps) for each entity. The training framework is built on [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio), using the dataset [DiffSynth-Studio/EliGenTrainSet](https://www.modelscope.cn/datasets/DiffSynth-Studio/EliGenTrainSet).
+
+## Results Demonstration
+
+|Entity Control Condition|Generated Image|
+|-|-|
+|![eligen_example_1_0](./assets/samples/poster_region.png)|![eligen_example_1_mask_0](./assets/samples/poster.png)|
+|![eligen_example_1_0](./assets/samples/eligen_example_1_mask.png)|![eligen_example_1_mask_0](./assets/samples/eligen_example_1.png)|
+|![eligen_example_1_0](./assets/samples/eligen_example_2_mask.png)|![eligen_example_1_mask_0](./assets/samples/eligen_example_2.png)|
+|![eligen_example_1_0](./assets/samples/eligen_example_3_mask.png)|![eligen_example_1_mask_0](./assets/samples/eligen_example_3.png)|
+|![eligen_example_1_0](./assets/samples/eligen_example_4_mask.png)|![eligen_example_1_mask_0](./assets/samples/eligen_example_4.png)|
+|![eligen_example_1_0](./assets/samples/eligen_example_5_mask.png)|![eligen_example_1_mask_0](./assets/samples/eligen_example_5.png)|
+
+## Inference Code
+```
+git clone https://github.com/modelscope/DiffSynth-Studio.git  
+cd DiffSynth-Studio
+pip install -e .
+```
+
+```python
+from diffsynth.pipelines.qwen_image import QwenImagePipeline, ModelConfig
+from modelscope import dataset_snapshot_download, snapshot_download
+import torch
+from PIL import Image
+```
+
+```python
+pipe = QwenImagePipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="cuda",
+    model_configs=[
+        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"),
+        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="text_encoder/model*.safetensors"),
+        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"),
+    ],
+    tokenizer_config=ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="tokenizer/"),
+)
+snapshot_download("DiffSynth-Studio/Qwen-Image-EliGen", local_dir="models/DiffSynth-Studio/Qwen-Image-EliGen", allow_file_pattern="model.safetensors")
+pipe.load_lora(pipe.dit, "models/DiffSynth-Studio/Qwen-Image-EliGen/model.safetensors")
+
+global_prompt = "Poster for the Qwen-Image-EliGen Magic Café, featuring two magical coffees—one emitting flames and the other emitting ice spikes—against a light blue misty background, with text reading 'Qwen-Image-EliGen Magic Café' and 'New Arrival'"
+entity_prompts = ["A red magic coffee with flames rising from the cup", 
+                  "A red magic coffee surrounded by ice spikes", 
+                  "Text: 'New Arrival'", 
+                  "Text: 'Qwen-Image-EliGen Magic Café'"]
+
+dataset_snapshot_download(dataset_id="DiffSynth-Studio/examples_in_diffsynth", local_dir="./", allow_file_pattern=f"data/examples/eligen/qwen-image/example_6/*.png")
+masks = [Image.open(f"./data/examples/eligen/qwen-image/example_6/{i}.png").convert('RGB').resize((1328, 1328)) for i in range(len(entity_prompts))]
+
+image = pipe(
+    prompt=global_prompt,
+    seed=0,
+    eligen_entity_prompts=entity_prompts,
+    eligen_entity_masks=masks,
+)
+image.save("image.jpg")
+```
+
+## Citation
+If you find our work helpful, please consider citing our research:
+```
+@article{zhang2025eligen,
+  title={Eligen: Entity-level Controlled Image Generation with Regional Attention},
+  author={Zhang, Hong and Duan, Zhongjie and Wang, Xingjun and Chen, Yingda and Zhang, Yu},
+  journal={arXiv preprint arXiv:2501.01097},
+  year={2025}
+}
+```

README_from_modelscope.md

@@ -0,0 +1,107 @@
+---
+frameworks:
+- Pytorch
+license: Apache License 2.0
+tasks:
+- text-to-image-synthesis
+
+#model-type:
+##如 gpt、phi、llama、chatglm、baichuan 等
+#- gpt
+
+#domain:
+##如 nlp、cv、audio、multi-modal
+#- nlp
+
+#language:
+##语言代码列表 https://help.aliyun.com/document_detail/215387.html?spm=a2c4g.11186623.0.0.9f8d7467kni6Aa
+#- cn
+
+#metrics:
+##如 CIDEr、Blue、ROUGE 等
+#- CIDEr
+
+#tags:
+##各种自定义，包括 pretrained、fine-tuned、instruction-tuned、RL-tuned 等训练方法和其他
+#- pretrained
+
+#tools:
+##如 vllm、fastchat、llamacpp、AdaSeq 等
+#- vllm
+base_model:
+  - Qwen/Qwen-Image
+base_model_relation: adapter
+new_version: DiffSynth-Studio/Qwen-Image-EliGen-V2
+---
+# Qwen-Image 精确分区控制模型
+
+![](./assets/title.png)
+
+## 模型介绍
+
+本模型是基于 [Qwen-Image](https://www.modelscope.cn/models/Qwen/Qwen-Image) 训练的精确分区控制模型，模型结构为 LoRA，可以通过输入每个实体的文本和区域条件（蒙版图）来控制每个实体的位置和形状。训练框架基于 [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) 构建，采用的数据集是 [DiffSynth-Studio/EliGenTrainSet](https://www.modelscope.cn/datasets/DiffSynth-Studio/EliGenTrainSet)。
+
+
+## 效果展示
+
+|实体控制条件|生成图|
+|-|-|
+|![eligen_example_1_0](./assets/samples/poster_region.png)|![eligen_example_1_mask_0](./assets/samples/poster.png)|
+|![eligen_example_1_0](./assets/samples/eligen_example_1_mask.png)|![eligen_example_1_mask_0](./assets/samples/eligen_example_1.png)|
+|![eligen_example_1_0](./assets/samples/eligen_example_2_mask.png)|![eligen_example_1_mask_0](./assets/samples/eligen_example_2.png)|
+|![eligen_example_1_0](./assets/samples/eligen_example_3_mask.png)|![eligen_example_1_mask_0](./assets/samples/eligen_example_3.png)|
+|![eligen_example_1_0](./assets/samples/eligen_example_4_mask.png)|![eligen_example_1_mask_0](./assets/samples/eligen_example_4.png)|
+|![eligen_example_1_0](./assets/samples/eligen_example_5_mask.png)|![eligen_example_1_mask_0](./assets/samples/eligen_example_5.png)|
+
+
+## 推理代码
+```
+git clone https://github.com/modelscope/DiffSynth-Studio.git  
+cd DiffSynth-Studio
+pip install -e .
+```
+
+```python
+from diffsynth.pipelines.qwen_image import QwenImagePipeline, ModelConfig
+from modelscope import dataset_snapshot_download, snapshot_download
+import torch
+from PIL import Image
+
+
+pipe = QwenImagePipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="cuda",
+    model_configs=[
+        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"),
+        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="text_encoder/model*.safetensors"),
+        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"),
+    ],
+    tokenizer_config=ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="tokenizer/"),
+)
+snapshot_download("DiffSynth-Studio/Qwen-Image-EliGen", local_dir="models/DiffSynth-Studio/Qwen-Image-EliGen", allow_file_pattern="model.safetensors")
+pipe.load_lora(pipe.dit, "models/DiffSynth-Studio/Qwen-Image-EliGen/model.safetensors")
+
+global_prompt =  "Qwen-Image-EliGen魔法咖啡厅的宣传海报，主体是两杯魔法咖啡，一杯冒着火焰，一杯冒着冰锥，背景是浅蓝色水雾，海报写着“Qwen-Image-EliGen魔法咖啡厅”、“新品上市”"
+entity_prompts = ["一杯红色魔法咖啡，杯中火焰燃烧", "一杯红色魔法咖啡，杯中冰锥环绕", "字：“新品上市”", "字：“Qwen-Image-EliGen魔法咖啡厅”"]
+
+dataset_snapshot_download(dataset_id="DiffSynth-Studio/examples_in_diffsynth", local_dir="./", allow_file_pattern=f"data/examples/eligen/qwen-image/example_6/*.png")
+masks = [Image.open(f"./data/examples/eligen/qwen-image/example_6/{i}.png").convert('RGB').resize((1328, 1328)) for i in range(len(entity_prompts))]
+
+image = pipe(
+    prompt=global_prompt,
+    seed=0,
+    eligen_entity_prompts=entity_prompts,
+    eligen_entity_masks=masks,
+)
+image.save("image.jpg")
+```
+## 引用
+如果您觉得我们的工作对您有所帮助，欢迎引用我们的成果。
+```
+@article{zhang2025eligen,
+  title={Eligen: Entity-level controlled image generation with regional attention},
+  author={Zhang, Hong and Duan, Zhongjie and Wang, Xingjun and Chen, Yingda and Zhang, Yu},
+  journal={arXiv preprint arXiv:2501.01097},
+  year={2025}
+}
+```

configuration.json

@@ -0,0 +1 @@
+{"framework":"Pytorch","task":"text-to-image-synthesis"}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ac6571af480d75fc344fe8faf4b8f655e8c7809c473aaea439af6f6c8957a3b
+size 472047184