Upload folder using huggingface_hub

README.md

@@ -62,6 +62,8 @@ InternVL 2.0 is a multimodal large language model series, featuring models of va
 | MathVista<sub>testmini</sub> |      63.8       |       67.7       |     63.7      |         65.5         |
 |  OpenCompass<sub>avg</sub>   |      69.9       |       67.9       |     69.7      |         71.0         |
 
+- For more details and evaluation reproduction, please refer to our [Evaluation Guide](https://internvl.readthedocs.io/en/latest/internvl2.0/evaluation.html).
+
 - We simultaneously use InternVL and VLMEvalKit repositories for model evaluation. Specifically, the results reported for DocVQA, ChartQA, InfoVQA, TextVQA, MME, AI2D, MMBench, CCBench, MMVet, and SEED-Image were tested using the InternVL repository. OCRBench, RealWorldQA, HallBench, and MathVista were evaluated using the VLMEvalKit.
 
 - For MMMU, we report both the original scores (left side: evaluated using the InternVL codebase for InternVL series models, and sourced from technical reports or webpages for other models) and the VLMEvalKit scores (right side: collected from the OpenCompass leaderboard).
@@ -321,7 +323,7 @@ tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast
 
 # set the max number of tiles in `max_num`
 pixel_values = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()
-generation_config = dict(max_new_tokens=1024, do_sample=False)
+generation_config = dict(max_new_tokens=1024, do_sample=True)
 
 # pure-text conversation (纯文本对话)
 question = 'Hello, who are you?'
@@ -473,30 +475,28 @@ for new_text in streamer:
 
 ## Finetune
 
-SWIFT from ModelScope community has supported the fine-tuning (Image/Video) of InternVL, please check [this link](https://github.com/modelscope/swift/blob/main/docs/source_en/Multi-Modal/internvl-best-practice.md) for more details.
+Many repositories now support fine-tuning of the InternVL series models, including [InternVL](https://github.com/OpenGVLab/InternVL), [SWIFT](https://github.com/modelscope/ms-swift), [XTurner](https://github.com/InternLM/xtuner), and others. Please refer to their documentation for more details on fine-tuning.
 
 ## Deployment
 
 ### LMDeploy
 
-#### Service
-
-To deploy InternVL2 as an API, please configure the chat template config first. Create the following JSON file `chat_template.json`.
+LMDeploy is a toolkit for compressing, deploying, and serving LLM, developed by the MMRazor and MMDeploy teams.
 
-```json
-{
-    "model_name":"internvl-internlm2",
-    "meta_instruction":"我是书生·万象，英文名是InternVL，是由上海人工智能实验室、清华大学及多家合作单位联合开发的多模态大语言模型。",
-    "stop_words":["<|im_start|>", "<|im_end|>"]
-}
+```sh
+pip install lmdeploy==0.5.3
 ```
 
+LMDeploy abstracts the complex inference process of multi-modal Vision-Language Models (VLM) into an easy-to-use pipeline, similar to the Large Language Model (LLM) inference pipeline.
+
+#### Service
+
 LMDeploy's `api_server` enables models to be easily packed into services with a single command. The provided RESTful APIs are compatible with OpenAI's interfaces. Below are an example of service startup:
 
 > **⚠️ Warning**: Please make sure to install Flash Attention; otherwise, using `--tp` will cause errors.
 
 ```shell
-CUDA_VISIBLE_DEVICES=0,1,2,3 lmdeploy serve api_server OpenGVLab/InternVL2-Llama3-76B --backend turbomind --server-port 23333 --chat-template chat_template.json --tp 4
+CUDA_VISIBLE_DEVICES=0,1,2,3 lmdeploy serve api_server OpenGVLab/InternVL2-Llama3-76B --backend turbomind --server-port 23333 --tp 4
 ```
 
 To use the OpenAI-style interface, you need to install OpenAI:
@@ -533,14 +533,6 @@ response = client.chat.completions.create(
 print(response)
 ```
 
-### vLLM
-
-TODO
-
-### Ollama
-
-TODO
-
 ## License
 
 This project is released under the MIT license, while Llama3 is licensed under the Llama 3 Community License.
@@ -613,6 +605,8 @@ InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模
 | MathVista<sub>testmini</sub> |      63.8       |       67.7       |     63.7      |         65.5         |
 |  OpenCompass<sub>avg</sub>   |      69.9       |       67.9       |     69.7      |         71.0         |
 
+- 关于更多的细节以及评测复现，请看我们的[评测指南](https://internvl.readthedocs.io/en/latest/internvl2.0/evaluation.html)。
+
 - 我们同时使用 InternVL 和 VLMEvalKit 仓库进行模型评估。具体来说，DocVQA、ChartQA、InfoVQA、TextVQA、MME、AI2D、MMBench、CCBench、MMVet 和 SEED-Image 的结果是使用 InternVL 仓库测试的。OCRBench、RealWorldQA、HallBench 和 MathVista 是使用 VLMEvalKit 进行评估的。
 
 - 对于MMMU，我们报告了原始分数（左侧：InternVL系列模型使用InternVL代码库评测，其他模型的分数来自其技术报告或网页）和VLMEvalKit分数（右侧：从OpenCompass排行榜收集）。
@@ -671,30 +665,28 @@ InternVL 2.0 是一个多模态大语言模型系列，包含各种规模的模
 
 ## 微调
 
-来自ModelScope社区的SWIFT已经���持对InternVL进行微调（图像/视频），详情请查看[此链接](https://github.com/modelscope/swift/blob/main/docs/source_en/Multi-Modal/internvl-best-practice.md)。
+许多仓库现在都支持 InternVL 系列模型的微调，包括 [InternVL](https://github.com/OpenGVLab/InternVL)、[SWIFT](https://github.com/modelscope/ms-swift)、[XTurner](https://github.com/InternLM/xtuner) 等。请参阅它们的文档以获取更多微调细节。
 
 ## 部署
 
 ### LMDeploy
 
-#### API部署
-
-为了将InternVL2部署成API，请先配置聊天模板配置文件。创建如下的 JSON 文件 `chat_template.json`。
+LMDeploy 是由 MMRazor 和 MMDeploy 团队开发的用于压缩、部署和服务大语言模型（LLM）的工具包。
 
-```json
-{
-    "model_name":"internvl-internlm2",
-    "meta_instruction":"我是书生·万象，英文名是InternVL，是由上海人工智能实验室、清华大学及多家合作单位联合开发的多模态大语言模型。",
-    "stop_words":["<|im_start|>", "<|im_end|>"]
-}
+```sh
+pip install lmdeploy==0.5.3
 ```
 
+LMDeploy 将多模态视觉-语言模型（VLM）的复杂推理过程抽象为一个易于使用的管道，类似于大语言模型（LLM）的推理管道。
+
+#### API部署
+
 LMDeploy 的 `api_server` 使模型能够通过一个命令轻松打包成服务。提供的 RESTful API 与 OpenAI 的接口兼容。以下是服务启动的示例：
 
 > **⚠️ 注意**: 请务必安装Flash Attention; 否则，使用`——tp`将存在异常。
 
 ```shell
-CUDA_VISIBLE_DEVICES=0,1,2,3 lmdeploy serve api_server OpenGVLab/InternVL2-Llama3-76B --backend turbomind --server-port 23333 --chat-template chat_template.json --tp 4
+CUDA_VISIBLE_DEVICES=0,1,2,3 lmdeploy serve api_server OpenGVLab/InternVL2-Llama3-76B --backend turbomind --server-port 23333 --tp 4
 ```
 
 为了使用OpenAI风格的API接口，您需要安装OpenAI:
@@ -731,14 +723,6 @@ response = client.chat.completions.create(
 print(response)
 ```
 
-### vLLM
-
-TODO
-
-### Ollama
-
-TODO
-
 ## 开源许可证
 
 该项目采用 MIT 许可证发布，而 LLama3 则采用 Llama 3 Community License 许可证。

modeling_intern_vit.py

@@ -20,18 +20,12 @@ from transformers.utils import logging
 from .configuration_intern_vit import InternVisionConfig
 
 try:
-    try:  # v1
-        from flash_attn.flash_attn_interface import \
-            flash_attn_unpadded_qkvpacked_func
-    except:  # v2
-        from flash_attn.flash_attn_interface import \
-            flash_attn_varlen_qkvpacked_func as flash_attn_unpadded_qkvpacked_func
-
     from flash_attn.bert_padding import pad_input, unpad_input
-
+    from flash_attn.flash_attn_interface import \
+        flash_attn_varlen_qkvpacked_func
     has_flash_attn = True
 except:
-    print('FlashAttention is not installed.')
+    print('FlashAttention2 is not installed.')
     has_flash_attn = False
 
 logger = logging.get_logger(__name__)
@@ -74,7 +68,7 @@ class FlashAttention(nn.Module):
                 max_s = seqlen
                 cu_seqlens = torch.arange(0, (batch_size + 1) * seqlen, step=seqlen, dtype=torch.int32,
                                           device=qkv.device)
-                output = flash_attn_unpadded_qkvpacked_func(
+                output = flash_attn_varlen_qkvpacked_func(
                     qkv, cu_seqlens, max_s, self.dropout_p if self.training else 0.0,
                     softmax_scale=self.softmax_scale, causal=causal
                 )
@@ -84,7 +78,7 @@ class FlashAttention(nn.Module):
                 x = rearrange(qkv, 'b s three h d -> b s (three h d)')
                 x_unpad, indices, cu_seqlens, max_s = unpad_input(x, key_padding_mask)
                 x_unpad = rearrange(x_unpad, 'nnz (three h d) -> nnz three h d', three=3, h=nheads)
-                output_unpad = flash_attn_unpadded_qkvpacked_func(
+                output_unpad = flash_attn_varlen_qkvpacked_func(
                     x_unpad, cu_seqlens, max_s, self.dropout_p if self.training else 0.0,
                     softmax_scale=self.softmax_scale, causal=causal
                 )
@@ -93,7 +87,7 @@ class FlashAttention(nn.Module):
                                    'b s (h d) -> b s h d', h=nheads)
         else:
             assert max_s is not None
-            output = flash_attn_unpadded_qkvpacked_func(
+            output = flash_attn_varlen_qkvpacked_func(
                 qkv, cu_seqlens, max_s, self.dropout_p if self.training else 0.0,
                 softmax_scale=self.softmax_scale, causal=causal
             )