Hybrid Mamba-2 / MoE / Attention decoder — 52 layers, sequenced exactly as in
config.json. Rotate: drag · Zoom: scroll · Inspect: hover / click a layer.
Layer types — click to toggle
Parameter accounting
—
total parameters
—
active per token
—
sparsity — every token runs all Mamba & attention layers, but only 6 of 128 routed experts (+1 shared) per MoE layer
Controls
Router — pick a token
Real weights. The router puts each token's hidden state through a linear layer,
takes a sigmoid per expert (not softmax), adds a learned per-expert bias for selection only
(aux-loss-free load balancing), keeps the top-6, renormalizes the unbiased scores
(norm_topk_prob), and scales by routed_scaling_factor = 2.5 — so the six weights always
sum to 2.50. The shared expert is added unweighted. Data captured from an actual forward pass on this machine
(hook on mixer.gate).
Capture ran on the model's PyTorch fallback path — the fused mamba-ssm/causal-conv1d kernels have no build for
this stack, and the gated RMSNorm was a torch reimplementation of the kernel's documented semantics, not
numerically diffed against it. bf16 numerics can flip near-tie top-6 picks versus a kernels build, so treat
expert identities as representative rather than bit-exact; weight magnitudes and the routing behavior shown
are unaffected.
Pick a different MoE slab in the stack to see that layer's routing.
Note the prompt contains the token 2 twice — at layer 1 the two occurrences share 2 of 6 experts,
at layer 27 just 1, at layer 51 none: the router reads the contextual hidden state, not the token.
Selected component
Nothing selected
Hover a layer for a quick readout, or click to pin its explanation here.
Click an MoE slab to open its 128-expert grid.
token flow: bottom → top auto-rotating — drag to take over
What am I looking at?
This is a 3D map of an AI language model — NVIDIA's Nemotron-3-Nano —
showing how it turns a sentence into a prediction. The tall stack is 52
processing layers. A word enters at the bottom and is transformed layer by
layer on its way to the top.
Two things make this model clever:
It's efficient. The model has 31.6 billion internal parameters,
but it only uses about 3.6 billion for any single word — roughly 9× less
work than using all of them. It does this by waking up only 6 of its 128
"expert" sub-networks per word.
It understands context. Press Router demo, then tap the
token 2 — it appears twice in the example sentence ("The derivative
of x^2 is 2x."). The model sends it to different experts each time,
because the surrounding words are different. It's reacting to context, not
looking up a fixed answer.
Try this: press Forward pass to watch a word travel up through
the layers, then press Router demo to see which experts each word wakes
up. Hover or tap any block to learn what it does.