qiuyuu commited on
Commit
5c33dd4
Β·
verified Β·
1 Parent(s): 525acef

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +136 -0
README.md CHANGED
@@ -1,3 +1,139 @@
1
  ---
2
  license: cc-by-nc-sa-4.0
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: cc-by-nc-sa-4.0
3
+ language:
4
+ - en
5
+ pipeline_tag: image-to-video
6
+ tags:
7
+ - World Model
8
  ---
9
+
10
+ <div align="center">
11
+ <img src="assets/teaser.png">
12
+
13
+ <h1>Infinite Worlds with Versatile Interactions</h1>
14
+
15
+ Robbyant Team
16
+
17
+ </div>
18
+
19
+
20
+ <div align="center">
21
+
22
+ [![Page](https://img.shields.io/badge/%F0%9F%8C%90%20Project%20Page-Demo-00bfff)](https://technology.robbyant.com/lingbot-world-v2)
23
+ [![Tech Report](https://img.shields.io/static/v1?label=Paper&message=PDF&color=red&logo=arxiv)](https://arxiv.org/abs/xxx.xxx)
24
+ [![Model](https://img.shields.io/static/v1?label=%F0%9F%A4%97%20Model&message=HuggingFace&color=yellow)](https://huggingface.co/collections/robbyant/lingbot-world-v2)
25
+ [![Model](https://img.shields.io/static/v1?label=%F0%9F%A4%96%20Model&message=ModelScope&color=purple)](https://modelscope.cn/collections/Robbyant/LingBot-World-V2)
26
+ [![License](https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-green)](LICENSE.txt)
27
+
28
+
29
+ </div>
30
+
31
+ -----
32
+
33
+ We present **LingBot-World 2.0** (also known as **LingBot-World-Infinity**), an advanced iteration of [LingBot-World](https://technology.robbyant.com/lingbot-world) featuring four distinct upgrades.
34
+ - **Unbounded Interaction Horizon**: Our model achieves an unbounded interaction horizon while maintaining consistent output quality, benefiting from a carefully crafted causal pretraining paradigm.
35
+ - **Rapid Response Time**: Through distilling a real-time variant from the base model, our system guarantees rapid response time, sufficient to drive 720p video streams at 60 fps.
36
+ - **Highly Diverse Interactive Elements**: Compared to the previous version, this update introduces highly diverse interactive elements, comprising a broader spectrum of actions (*e.g.*, attacking, archery, spell-casting, and shooting) alongside a richer variety of text-driven events.
37
+ - **Agentic Harness**: We pioneer the integration of an agentic harness within the domain of world modeling, wherein a pilot agent is tasked with planning and executing character behaviors, while a director agent is responsible for synthesizing novel environmental elements as the scene progresses.
38
+
39
+
40
+ ## πŸš€ Try it now
41
+ The real-time version of LingBot-World-Infinity is available on two platforms. We thank [Reactor](https://www.reactor.inc/lingbot-world-v2) and [LingGuang](https://www.lingguang.com/support) for their support:
42
+ - **International (Web)**: Experience it on [Reactor](https://www.reactor.inc/lingbot-world-v2).
43
+ - **Domestic (Mobile)**: Experience it on [LingGuang](https://www.lingguang.com/support).
44
+
45
+ > **Note:** Reactor and LingGuang provide a convenient way to try LingBot-World-Infinity in real time, but some functions are missing. In our official setup, the model runs at full capability. To experience our official demo, join us at [WAIC 2026](https://waica2026.worldaic.com.cn/).
46
+
47
+ ## πŸ”₯ News
48
+ - Jul. 9, 2026: πŸŽ‰ We release the technical report, inference code, and models for LingBot-World-Infinity.
49
+
50
+ ## πŸ“‹ TODO
51
+ - [x] Release the causal-fast inference code and model of the 14B model
52
+ - [ ] Release the causal-pretrained model of the 14B model
53
+ - [ ] Release the bidirectional model of the 14B model
54
+ - [ ] Release the causal-fast and causal-pretrained models of the 1.3B model
55
+
56
+ ## βš™οΈ Quick Start
57
+ This codebase is built upon [Wan2.2](https://github.com/Wan-Video/Wan2.2). Please refer to their documentation for installation instructions.
58
+ ### Installation
59
+ Clone the repo:
60
+ ```sh
61
+ git clone https://github.com/robbyant/lingbot-world-v2.git
62
+ cd lingbot-world-v2
63
+ ```
64
+ Install dependencies:
65
+ ```sh
66
+ # Ensure torch >= 2.4.0
67
+ pip install -r requirements.txt
68
+ ```
69
+ Install [`flash_attn`](https://github.com/Dao-AILab/flash-attention):
70
+ ```sh
71
+ pip install flash-attn --no-build-isolation
72
+ ```
73
+ ### Model Download
74
+
75
+ | Model | Model Type | Model Size | Download Links |
76
+ | :--- | :--- | :--- | :--- |
77
+ | **lingbot-world-v2-14b-causal-fast** | causal-fast | 14B | πŸ€— [HuggingFace](https://huggingface.co/robbyant/lingbot-world-v2-14b-causal-fast) πŸ€– [ModelScope](https://www.modelscope.cn/models/Robbyant/lingbot-world-v2-14b-causal-fast) |
78
+ | **lingbot-world-v2-14b-causal-pretrain** | causal-pretrain | 14B | TODO |
79
+
80
+ Download models using huggingface-cli:
81
+ ```sh
82
+ pip install "huggingface_hub[cli]"
83
+ huggingface-cli download robbyant/lingbot-world-v2-14b-causal-fast --local-dir ./lingbot-world-v2-14b-causal-fast
84
+ ```
85
+ Download models using modelscope-cli:
86
+ ```sh
87
+ pip install modelscope
88
+ modelscope download robbyant/lingbot-world-v2-14b-causal-fast --local_dir ./lingbot-world-v2-14b-causal-fast
89
+ ```
90
+
91
+ ### Inference
92
+
93
+ We provide `generate.py` for causal inference with KV caching, which processes video frames chunk-by-chunk instead of all at once.
94
+ <!-- The `--infer_mode` flag selects the inference mode:
95
+
96
+ | infer_mode | Model | Sampling |
97
+ | :--- | :--- | :--- |
98
+ | `causal_fast` (default) | Distilled few-step model (`LingBot-World-Fast`) | 4 steps per chunk, no CFG |
99
+ | `causal_pretrain` | Pretrained causal model | 40 steps per chunk with CFG | -->
100
+
101
+ - `causal_fast` β€” 480P, multi-GPU:
102
+ ``` sh
103
+ torchrun --nproc_per_node=8 generate.py --task i2v-A14B --size 480*832 --ckpt_dir lingbot-world-v2-14b-causal-fast --image examples/03/image.jpg --action_path examples/03 --dit_fsdp --t5_fsdp --ulysses_size 8 --frame_num 361 --local_attn_size 18 --sink_size 6 --prompt "A serene lakeside scene with a lone tree standing in calm water, surrounded by distant snow-capped mountains under a bright blue sky with drifting white clouds β€” gentle ripples reflect the tree and sky, creating a tranquil, meditative atmosphere."
104
+ ```
105
+
106
+ <!-- - `causal_pretrain` β€” 480P, multi-GPU:
107
+ ``` sh
108
+ torchrun --nproc_per_node=8 generate.py --task i2v-A14B --infer_mode causal_pretrain --size 480*832 --ckpt_dir lingbot-world-v2-14b-causal-pretrain --image examples/03/image.jpg --action_path examples/03 --dit_fsdp --t5_fsdp --ulysses_size 8 --frame_num 81 --prompt "A serene lakeside scene with a lone tree standing in calm water, surrounded by distant snow-capped mountains under a bright blue sky with drifting white clouds β€” gentle ripples reflect the tree and sky, creating a tranquil, meditative atmosphere."
109
+ ``` -->
110
+
111
+ You can also use the provided `run_fast.sh` script:
112
+ ``` sh
113
+ bash run_fast.sh <weights_dir> <frame_num>
114
+ # e.g. bash run_fast.sh lingbot-world-v2-14b-causal-fast 361
115
+ ```
116
+
117
+ ### Deployment
118
+ We do NOT plan to release our deployment code. If you would like to deploy our model yourself, please refer to the LingBot-World deployment in [SGLang](https://docs.sglang.io/cookbook/diffusion/LingBot-World/LingBot-World-2.0) or [flashdreams](https://github.com/NVIDIA/flashdreams).
119
+
120
+ ## πŸ“š Related Projects
121
+ - [LingBot-World](https://github.com/robbyant/lingbot-world)
122
+
123
+ ## πŸ“œ License
124
+ This project is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0). The project is available for non-commercial use only: you may share and adapt it with proper attribution, but derivative works must be distributed under the same license. Please refer to the [LICENSE file](LICENSE.txt) for the full text, including details on rights and restrictions.
125
+
126
+ ## ✨ Acknowledgement
127
+ We would like to express our gratitude to the Wan Team for open-sourcing their code and models. Their contributions have been instrumental to the development of this project.
128
+
129
+ ## πŸ“– Citation
130
+ If you find this work useful for your research, please cite our paper:
131
+
132
+ ```
133
+ @article{lingbot-world-v2,
134
+ title={Infinite Worlds with Versatile Interactions},
135
+ author={Zelin Gao and Qiuyu Wang and Jiapeng Zhu and Jingye Chen and Zichen Liu and Qingyan Bai and Jiahao Wang and Yufeng Yuan and Hanlin Wang and Yichong Lu and Ka Leong Cheng and Haojie Zhang and Jian Gao and Tianrui Feng and Yuzheng Liu and Yao Yao and Yinghao Xu and Xing Zhu and Yujun Shen and Hao Ouyang},
136
+ journal={arXiv preprint arXiv:xxx.xxx},
137
+ year={2026}
138
+ }
139
+ ```