Eloi Alonso (@EloiAlonso1) / X

Eloi Alonso

73 posts

Eloi Alonso

@EloiAlonso1

Co-Founder @gen_intuition. PhD in world models @UNIGE_en.

Geneva

Joined February 2021

Pinned
Eloi Alonso
@EloiAlonso1
Oct 11, 2024
Ever wanted to play Counter-Strike in a neural network? These videos show people playing (with keyboard & mouse) in 💎 DIAMOND's diffusion world model, trained to simulate the game Counter-Strike: Global Offensive. 💻 Download and play it yourself → github.com/eloialonso/dia… 🧵
00:00
748K
Eloi Alonso
@EloiAlonso1
May 24, 2024
We introduce DIAMOND (DIffusion As a Model Of eNvironment Dreams), a reinforcement learning agent trained entirely in a diffusion world model. 💎 Paper: arxiv.org/abs/2405.12399 Code: github.com/eloialonso/dia… (1/n) 🧵👇
GIF
135K
Eloi Alonso
@EloiAlonso1
Oct 11, 2024
Replying to @EloiAlonso1
We were very impressed by the recent DOOM #GameNGen from @Google, which inspired us to apply DIAMOND to CS:GO (first made public for Atari in May arxiv.org/pdf/2405.12399). Our CS:GO world model runs at ~10 FPS on a standard gaming GPU (RTX 3090). (2/n)
00:00
47K
Eloi Alonso
@EloiAlonso1
Oct 11, 2024
Replying to @EloiAlonso1
However, scaling would not solve everything, such as unexpected behaviors due to the limited memory of our model. This limitation interestingly enables the model to compose multiple jumps in a row by generalizing the effect of a jump on the geometry of a scene! (5/n)
00:00
72K
Eloi Alonso
@EloiAlonso1
Oct 11, 2024
Replying to @EloiAlonso1
As a comparison to #GameNGen, our model was trained on only 0.5% of the number of frames, with 1 GPU (compared to 128 TPUs). And our code, model and data are completely open-source! You can play it on your local machine. github.com/eloialonso/dia… (3/n)
GitHub - eloialonso/diamond: DIAMOND (DIffusion As a Model Of eNvironment Dreams) is a reinforcem...
From github.com
23K
Eloi Alonso
@EloiAlonso1
Oct 11, 2024
Replying to @EloiAlonso1
There are still many limitations of our model. We’re sure you can find more yourself by interacting with the model. However, we expect the world model would continue to improve by scaling up data and compute, given our dataset only amounts to 87h of gameplay. (4/n)
00:00
32K
Eloi Alonso
@EloiAlonso1
Oct 11, 2024
Replying to @EloiAlonso1
💎 DIAMOND is a Spotlight at NeurIPS 2024, so come and chat with @AdamJelley2 and me in Vancouver 🇨🇦! 🎬 Project page: diamond-wm.github.io 💻 Code and Playable World Models: github.com/eloialonso/dia… 📝 Paper: arxiv.org/abs/2405.12399 (8/n)
11K
Eloi Alonso
@EloiAlonso1
Oct 11, 2024
Replying to @EloiAlonso1
For more details on how our world model works, check out our first thread on 💎 DIAMOND for Atari: x.com/EloiAlonso1/st… (6/n)
GIF
Eloi Alonso
@EloiAlonso1
May 24, 2024
We introduce DIAMOND (DIffusion As a Model Of eNvironment Dreams), a reinforcement learning agent trained entirely in a diffusion world model. 💎 Paper: arxiv.org/abs/2405.12399 Code: github.com/eloialonso/dia… (1/n) 🧵👇
17K
Eloi Alonso
@EloiAlonso1
Oct 11, 2024
Replying to @EloiAlonso1
We scaled up our original Atari setup, with three main changes: - No RL: fixed dataset of gameplay. - Two stages: dynamics prediction in low resolution, before upsampling with a second model, reducing training cost. - Stochastic sampling: improves generation quality. (7/n)
00:00
10K
Eloi Alonso
@EloiAlonso1
Oct 12, 2024
Crazy to see this running online <12h after our initial post! x.com/EloiAlonso1/st… Code & model here: github.com/eloialonso/dia…
fal
@fal
Oct 12, 2024
world models are here! @fal is the best platform for real-time audio and video inference. RT for access 😇 incredible work by @EloiAlonso1 to open source this model
00:00
7.4K
Eloi Alonso
@EloiAlonso1
Dec 11, 2024
Try 💎 DIAMOND’s Counter Strike world model directly in your browser! → next.journee.ai/xyz-diamond ← How long can you stay in distribution? Can you beat @stephliemnguyen’s 1000 frames? @AdamJelley2 and I are at NeurIPS! Poster #6306, Friday 11am-2pm, West Ballroom
00:00
9.1K
Eloi Alonso
@EloiAlonso1
Oct 11, 2024
Replying to @EloiAlonso1
Special thanks to our resident CSGO expert @stephliemnguyen for testing our world model, @Tea_Pearce for dataset expertise, and to our collaborators @micheli_vincent, Anssi Kanervisto, @AmosStorkey, @smdvlin, @katjahofmann, and @francoisfleuret! (9/9)
8.9K
Eloi Alonso
@EloiAlonso1
May 24, 2024
Replying to @EloiAlonso1
How does it work? We train a diffusion model to predict the next frame of the game. The diffusion model takes into account the agent’s action and the previous frames to simulate the environment response. (2/n)
GIF
3.3K
Eloi Alonso
@EloiAlonso1
May 24, 2024
Replying to @EloiAlonso1
Go to github.com/eloialonso/dia… to try our playable world models! git clone [email protected]:eloialonso/diamond.git cd diamond conda create -n diamond python=3.10 conda activate diamond pip install -r requirements.txt python src/play.py --pretrained (9/n)
GitHub - eloialonso/diamond: DIAMOND (DIffusion As a Model Of eNvironment Dreams) is a reinforcem...
From github.com
5.1K