You use GPUs everyday, but do you (actually) know how they work?
GPU-Puzzles (v0.1) - 14 short puzzles in Python with a visual debugger. No background required. Do puzzles, learn CUDA.
Link: github.com/srush/GPU-Puzz…
Sasha Rush
8,683 posts
Sasha Rush
@srush_nlp
Researcher at Cursor
youtube.com/@srush_nlp
- Some personal news: I recently joined Cursor. Cursor is a small, ambitious team, and they’ve created my favorite AI systems. We’re now building frontier RL models at scale in real-world coding environments. Excited for how good coding is going to be.
00:00 - I'm advising a PhD student at Mistral and, really not sure how I feel about their LateX template.
- 1/ Spent the last couple weeks in quarantine obsessively coding a website for Virtual ICLR with @hen_str. We wanted to build something that was fun to browse, async first, and feels alive.
GIF - For some reason, I needed xkcd formatting to convince my brain what Large means.
- Introducing COLM (colmweb.org) the Conference on Language Modeling. A new research venue dedicated to the theory, practice, and applications of language models. Submissions: March 15 (it's pronounced "collum" 🕊️)
- These tutorial slides on "High Perf NLP" are really impressive. Every slide is current to the minute. Amazing set of diagrams. gabrielilharco.com/publications/E… (@gabriel_ilharco @Tim_Dettmers @IuliaTurc @kentonctlee Felipe Ferreira Cesar Ilharco)
- Mamba apparently was rejected !? (openreview.net/forum?id=AL1fq…) Honestly I don't even understand. If this gets rejected, what chance do us 🤡 s have.
- Do you (really) know PyTorch 🔥? Try out my Tensor Puzzles 🧩. 16 mini-puzzles for those ready to take off the stackoverflow training wheels. github.com/srush/Tensor-P…
- This talk by Angela Fan on Llama2 is so good. 30 min, she just tells you all the things.
- If you know Torch, I think you can code for GPU now with OpenAI's Triton language. We made some puzzles to help you rewire your brain. Starts easy, but gets quickly to fun modern models like FlashAttention and GPT-Q. Good luck! github.com/srush/Triton-P…
GIF - There are like 200 people in the world who were in charge of training 50B+ param LLMs. They all talk about it like they survived a tornado. "And then the door flew off, but the roof held!"
