TorchCode README

The project's README from github.com/duoan/TorchCode. As one markdown file: README.md.

title: TorchCode emoji: 🔥 colorFrom: red colorTo: yellow sdk: docker app_port: 7860 pinned: false

🔥 TorchCode

Crack the PyTorch interview.

Practice implementing operators and architectures from scratch — the exact skills top ML teams test for.

An interactive coding platform, but for tensors. Self-hosted. Jupyter-based. Instant feedback.

PyTorch Jupyter Docker Python License: MIT

GitHub stars GitHub Container Registry Hugging Face Spaces Problems GPU

Star History Chart

🎯 Why TorchCode?

Top companies (Meta, Google DeepMind, OpenAI, etc.) expect ML engineers to implement core operations from memory on a whiteboard. Reading papers isn't enough — you need to write softmax, LayerNorm, MultiHeadAttention, and full Transformer blocks code.

TorchCode gives you a structured practice environment with:

Feature
🧩 41 curated problems The most frequently asked PyTorch interview topics
⚖️ Automated judge Correctness checks, gradient verification, and timing
🎨 Instant feedback Colored pass/fail per test case, just like competitive programming
💡 Hints when stuck Nudges without full spoilers
📖 Reference solutions Study optimal implementations after your attempt
📊 Progress tracking What you've solved, best times, and attempt counts
🔄 One-click reset Toolbar button to reset any notebook back to its blank template — practice the same problem as many times as you want
Open In Colab Open in Colab Every notebook has an "Open in Colab" badge + toolbar button — run problems in Google Colab with zero setup

No cloud. No signup. No GPU needed. Just make run — or try it instantly on Hugging Face.

🚀 Quick Start

Option 0 — Try it online (zero install)

Launch on Hugging Face Spaces — opens a full JupyterLab environment in your browser. Nothing to install.

Or open any problem directly in Google Colab — every notebook has an Open In Colab badge.

Option 0b — Use the judge in Colab (pip)

In Google Colab, install the judge from PyPI so you can run check(...) without cloning the repo:

!pip install torch-judge

Then in a notebook cell:

from torch_judge import check, status, hint, reset_progress
status()           # list all problems and your progress
check("relu")      # run tests for the "relu" task
hint("relu")       # show a hint

Option 1 — Pull the pre-built image (fastest)

docker run -p 8888:8888 -e PORT=8888 ghcr.io/duoan/torchcode:latest

If the registry image is unavailable for your platform, use Option 2 instead. This is the common path on Apple Silicon / arm64.

Option 2 — Build locally

make run

make run will try the prebuilt image first and automatically fall back to a local build when needed.

Open http://localhost:8888 — that's it. Works with both Docker and Podman (auto-detected).

Option 3 — Standalone Web UI (Next.js + FastAPI)

For a modern, standalone coding experience with an integrated IDE and dual-pane layout:

  1. Start Backend (FastAPI):
    pip install -r api/requirements.txt
python -m uvicorn api.main:app --port 8000 --reload
  2. Start Frontend (Next.js):
    cd web
npm install
npm run dev
  3. Open http://localhost:3000 in your browser.

TorchCode UI Preview

📋 Problem Set

Frequency: 🔥 = very likely in interviews, ⭐ = commonly asked, 💡 = emerging / differentiator

🧱 Fundamentals — "Implement X from scratch"

The bread and butter of ML coding interviews. You'll be asked to write these without torch.nn.

# Problem What You'll Implement Difficulty Freq Key Concepts
1 ReLU Open In Colab relu(x) Easy 🔥 Activation functions, element-wise ops
2 Softmax Open In Colab my_softmax(x, dim) Easy 🔥 Numerical stability, exp/log tricks
16 Cross-Entropy Loss Open In Colab cross_entropy_loss(logits, targets) Easy 🔥 Log-softmax, logsumexp trick
17 Dropout Open In Colab MyDropout (nn.Module) Easy 🔥 Train/eval mode, inverted scaling
18 Embedding Open In Colab MyEmbedding (nn.Module) Easy 🔥 Lookup table, weight[indices]
19 GELU Open In Colab my_gelu(x) Easy ⭐ Gaussian error linear unit, torch.erf
20 Kaiming Init Open In Colab kaiming_init(weight) Easy ⭐ std = sqrt(2/fan_in), variance scaling
21 Gradient Clipping Open In Colab clip_grad_norm(params, max_norm) Easy ⭐ Norm-based clipping, direction preservation
31 Gradient Accumulation Open In Colab accumulated_step(model, opt, ...) Easy 💡 Micro-batching, loss scaling
40 Linear Regression Open In Colab LinearRegression (3 methods) Medium 🔥 Normal equation, GD from scratch, nn.Linear
3 Linear Layer Open In Colab SimpleLinear (nn.Module) Medium 🔥 y = xW^T + b, Kaiming init, nn.Parameter
4 LayerNorm Open In Colab my_layer_norm(x, γ, β) Medium 🔥 Normalization, running stats, affine transform
7 BatchNorm Open In Colab my_batch_norm(x, γ, β) Medium ⭐ Batch vs layer statistics, train/eval behavior
8 RMSNorm Open In Colab rms_norm(x, weight) Medium ⭐ LLaMA-style norm, simpler than LayerNorm
15 SwiGLU MLP Open In Colab SwiGLUMLP (nn.Module) Medium ⭐ Gated FFN, SiLU(gate) * up, LLaMA/Mistral-style
22 Conv2d Open In Colab my_conv2d(x, weight, ...) Medium 🔥 Convolution, unfold, stride/padding

🧠 Attention Mechanisms — The heart of modern ML interviews

If you're interviewing for any role touching LLMs or Transformers, expect at least one of these.

# Problem What You'll Implement Difficulty Freq Key Concepts
23 Cross-Attention Open In Colab MultiHeadCrossAttention (nn.Module) Medium ⭐ Encoder-decoder, Q from decoder, K/V from encoder
5 Scaled Dot-Product Attention Open In Colab scaled_dot_product_attention(Q, K, V) Hard 🔥 softmax(QK^T/√d_k)V, the foundation of everything
6 Multi-Head Attention Open In Colab MultiHeadAttention (nn.Module) Hard 🔥 Parallel heads, split/concat, projection matrices
9 Causal Self-Attention Open In Colab causal_attention(Q, K, V) Hard 🔥 Autoregressive masking with -inf, GPT-style
10 Grouped Query Attention Open In Colab GroupQueryAttention (nn.Module) Hard ⭐ GQA (LLaMA 2), KV sharing across heads
11 Sliding Window Attention Open In Colab sliding_window_attention(Q, K, V, w) Hard ⭐ Mistral-style local attention, O(n·w) complexity
12 Linear Attention Open In Colab linear_attention(Q, K, V) Hard 💡 Kernel trick, φ(Q)(φ(K)^TV), O(n·d²)
14 KV Cache Attention Open In Colab KVCacheAttention (nn.Module) Hard 🔥 Incremental decoding, cache K/V, prefill vs decode
24 RoPE Open In Colab apply_rope(q, k) Hard 🔥 Rotary position embedding, relative position via rotation
25 Flash Attention Open In Colab flash_attention(Q, K, V, block_size) Hard 💡 Tiled attention, online softmax, memory-efficient

🏗️ Architecture & Adaptation — Put it all together

# Problem What You'll Implement Difficulty Freq Key Concepts
26 LoRA Open In Colab LoRALinear (nn.Module) Medium ⭐ Low-rank adaptation, frozen base + BA update
27 ViT Patch Embedding <a

TorchCode (duoan/TorchCode)

Searching Database…

4.7K ·GitHub
TorchCode
这是什么？TorchCode 是一个 LeetCode 风格的 PyTorch 练习平台：它提供 41 道精选编程题，让你在 Jupyter 笔记本中从零实现机器学习算子和架构（softmax、attention、GPT-2 块等），并配有自动评测器，即时检查正确性、梯度和耗时。
这是我要导入的库，还是我要运行的独立平台？它是一个基于 Jupyter 笔记本构建、可自托管的编程练习应用，不是加进你自己项目的库——你要么通过 Docker 在本地运行，要么通过托管的 JupyterLab 环境使用。
怎么让它跑起来？最简单的方式是启动 Hugging Face Spaces 实例，在浏览器里获得完整的 JupyterLab，零安装；或者在 Google Colab 中直接打开任意题目笔记本。要自托管，运行 make run，它会拉取预构建的 Docker 镜像，必要时回退到本地构建。
安装上有什么坑需要提前知道吗？预构建的 Docker 镜像可能并非在所有平台上都可用，Apple Silicon / arm64 上就是常见情况——这时你需要回退到在本地构建镜像，而不是直接拉取。
它真正面向的是谁？它面向正在备战顶级 AI 实验室面试的机器学习工程师，这类面试要求能凭记忆实现 attention、归一化层等核心 PyTorch 算子。
一共有多少道题？都覆盖什么内容？41 道精选题目分为三组：基础（ReLU、Softmax、LayerNorm、Conv2d、线性回归）、注意力机制（Multi-Head、Causal、GQA、RoPE、Flash Attention），以及架构与适配（LoRA、ViT Patch Embedding 等）。
如果我卡住了，它会提供提示吗？会，它提供的提示只做点拨而不泄露完整解法；参考解会在你自己尝试之后开放，供你研究。
自动评测器到底检查什么？自动评测器会运行正确性检查、验证梯度并测量耗时，按测试用例给出彩色的通过/失败反馈，类似竞赛评测机。
如果我把题目搞砸了，可以重做吗？可以，每个笔记本的工具栏里都有一个一键重置按钮，能把题目恢复成空白模板，想练多少次都可以。
What are similar projects?LLMs-from-scratch, DeepLearning, llama3-from-scratch, autumn-recruitment, nn-zero-to-hero, manga-image-translator, happy-llm, apachecn-algo-zh
Which topics is it in?coding-interview, leetcode, attention-mechanism, lora, interview, interview-preparation, neural-network, jupyter notebook, transformer, docker-compose, pytorch, learning
Creator
duoan
Size
0.6 MB
Stars
4.7K
Updated
CommitJul 31
Age
7mo
Commits · 90 Days