Skip to content

[CVPR2024 Highlight][VideoChatGPT] ChatGPT with video understanding! And many more supported LMs such as miniGPT4, StableLM, and MOSS.

License

Notifications You must be signed in to change notification settings

OpenGVLab/Ask-Anything

Repository files navigation

🦜 Ask-Anything [Paper]

Open in OpenXLab | | | |
Open in Spaces [VideoChat-7B-8Bit] End2End ChatBOT for video and image.

[VideoChat2-7B]End2End ChatBOT for video and image.

中文 README 及 中文交流群 | Paper

🚀: We update video_chat by instruction tuning for video & image chatting now! Find its details here. We release instruction data at InternVideo. The old version of video_chat moved to video_chat_with_chatGPT.

⭐️: We are also working on a updated version, stay tuned!

english.mp4
intro.mp4

🔥 Updates

  • 2024/06/25: We release the branch of videochat2 using vllm, speed up the inference of videochat2.

  • 2024/06/19: 🎉🎉 Our VideoChat2 achieves the best performances among the open-sourced VideoLLMs on MLVU, a multi-task long video understanding benchmark.

  • 2024/06/13: Fix some bug and give testing scripts/

  • 2024/06/07: 🔥🔥🔥 We release VideoChat2_HD, which is fine-tuned with high-resolution data and is capable of handling more diverse tasks. It showcases better performance on different benchmarks, especially for detailed captioning. Furthermore, it achieves 54.8% on Video-MME, the best score among 7B MLLMs. Have a try! 🏃🏻‍♀️🏃🏻

  • 2024/06/06: We release VideoChat2_phi3, a faster model with robust performaces.

  • 2024/05/22: We release VideoChat2_mistral, which shows better capacity on diverse tasks (60.4% on MVBench, 78.6% on NExT-QA, 63.8% on STAR, 46.4% on TVQA, 54.4% on EgoSchema-full and 80.5% on IntentQA). More details have been updated in the paper.

  • 2024/04/05 MVBench is selected as Poster (Highlight)!

  • 2024/2/27 MVBench is accepted by CVPR2024.

  • 2023/11/29 VideoChat2 and MVBench are released.

  • 2023/05/11 End-to-end VideoChat and its technical report.

    • VideoChat: Instruction tuning for video chatting (also supports image one).
    • Paper: We present how we craft VideoChat with two versions (via text and embed) along with some discussions on its background, applications, and more.
  • 2023/04/25 Watch videos longer than one minute with chatGPT

  • 2023/04/21 Chat with MOSS

  • 2023/04/20: Chat with StableLM

  • 2023/04/19: Code release & Online Demo

🔨 Getting Started

Build video chat with:

📄 Citation

If you find this project useful in your research, please consider cite:

@article{2023videochat,
  title={VideoChat: Chat-Centric Video Understanding},
  author={Li, Kunchang and He, Yinan and Wang, Yi and Li, Yizhuo and Wang, Wenhai and Luo, Ping and Wang, Yali and Wang, Limin and Qiao, Yu},
  journal={arXiv preprint arXiv:2305.06355},
  year={2023}
}

⏳ Ongoing

Our team constantly studies general video understanding and long-term video reasoning:

  • Strong video foundation model.
  • Video-text dataset and video reasoning benchmark.
  • Video-language system with LLMs.
  • Artificial Intelligence Generated Content (AIGC) for Video.
  • ...

🌤️ Discussion Group

If you have any questions during the trial, running or deployment, feel free to join our WeChat group discussion! If you have any ideas or suggestions for the project, you are also welcome to join our WeChat group discussion!

image image

We are hiring researchers, engineers and interns in General Vision Group, Shanghai AI Lab. If you are interested in working with us, please contact Yi Wang ([email protected]).