google/gemma-4-E2B-it
Original5.1Bparameters
Fixture36.3Mparameters
Layer stacks3compared
Findings0to review
✓ Same architecture, smaller
Every module class, layer kind and checked property of the original is in the fixture.
Model tester (advisory)
The model's own small test config in its ModelTester (53 literal values, 31 set by the model's own tester, the rest generic base-tester defaults). Testers are usually representative but not always, so these notes are for review, not rules.
The fixture covers every per-layer kind the tester sets, and shrinks every size it does.
Layer stacks
One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.
model.vision_tower.encoder.layers
Original · 16
Fixture · 2
| Kind | Original | Fixture | |
|---|---|---|---|
| VisionEncoderLayer | 16 | 2 | ✓ kept |
model.language_model.layers
Original · 35
Fixture · 4
| Kind | Original | Fixture | |
|---|---|---|---|
| sliding_attention | 28 | 2 | ✓ kept |
| full_attention | 7 | 2 | ✓ kept |
model.audio_tower.layers
Original · 12
Fixture · 2
| Kind | Original | Fixture | |
|---|---|---|---|
| AudioLayer | 12 | 2 | ✓ kept |
Where the parameters are
Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.
Original
Fixture
| Component | Original | Share | Fixture | Share |
|---|---|---|---|---|
| model.language_model | 2.8B | 54.2% | 33.6M | 92.5% |
| model.language_model.layers | 1.9B | 36.5% | 766.8K | 2.1% |
| model.audio_tower.layers | 302.2M | 5.9% | 198.4K | 0.5% |
| model.vision_tower.encoder.layers | 151.0M | 3.0% | 82.5K | 0.2% |
| model.vision_tower | 16.3M | 0.3% | 1.4M | 3.7% |
| model.audio_tower | 2.7M | 0.1% | 203.6K | 0.6% |
| model.embed_audio | 2.4M | 0.0% | 98.3K | 0.3% |
| 1 others | 1.2M | 0.0% | 4.1K | 0.0% |
Preserved properties
| Config | Property | Original | Fixture | |
|---|---|---|---|---|
| model | model_type | gemma4 | gemma4 | ✓ kept |
| model | tie_word_embeddings | True | True | ✓ kept |
| text_config | attention style | MQA (8 heads, 1 KV) | MQA (4 heads, 1 KV) | ✓ kept |
| text_config | model_type | gemma4_text | gemma4_text | ✓ kept |
| text_config | vocab_size | 262144 | 262144 | ✓ kept |
| text_config | tie_word_embeddings | True | True | ✓ kept |
| text_config | max_position_embeddings | 131072 | 131072 | ✓ kept |
| vision_config | attention style | MHA (12 heads, 12 KV) | MHA (4 heads, 4 KV) | ✓ kept |
| vision_config | RoPE type | axial | axial | ✓ kept |
| vision_config | model_type | gemma4_vision | gemma4_vision | ✓ kept |
| vision_config | max_position_embeddings | 131072 | 131072 | ✓ kept |
| vision_config | patch_size | 16 | 16 | ✓ kept |
| audio_config | model_type | gemma4_audio | gemma4_audio | ✓ kept |
| audio_config | hidden_act | silu | silu | ✓ kept |
Module classes
Instances of each class in the built model (prefix Gemma4 omitted).
| Class | Original | Fixture |
|---|---|---|
| Linear | 526 | 75 |
| RMSNorm | 464 | 64 |
| ClippableLinear | 232 | 34 |
| GELUTanh | 86 | 10 |
| SiLUActivation | 36 | 6 |
| TextAttention | 35 | 4 |
| TextDecoderLayer | 35 | 4 |
| TextMLP | 35 | 4 |
| AudioFeedForward | 24 | 4 |
| VisionAttention | 16 | 2 |
| VisionEncoderLayer | 16 | 2 |
| VisionMLP | 16 | 2 |
| AudioAttention | 12 | 2 |
| AudioCausalConv1d | 12 | 2 |
| AudioLayer | 12 | 2 |
| AudioLightConv1d | 12 | 2 |
| ModuleList | 3 | 3 |
| Conv2d | 2 | 2 |
| AudioSubSampleConvProjectionLayer | 2 | 2 |
| MultimodalEmbedder | 2 | 2 |
| TextScaledWordEmbedding | 2 | 2 |
| LayerNorm | 2 | 2 |
| ReLU | 2 | 2 |
| AudioModel | 1 | 1 |
| AudioRelPositionalEncoding | 1 | 1 |
| AudioSubSampleConvProjection | 1 | 1 |
| ForConditionalGeneration | 1 | 1 |
| Model | 1 | 1 |
| TextModel | 1 | 1 |
| TextRotaryEmbedding | 1 | 1 |
| VisionEncoder | 1 | 1 |
| VisionModel | 1 | 1 |
| VisionPatchEmbedder | 1 | 1 |
| VisionPooler | 1 | 1 |
| VisionRotaryEmbedding | 1 | 1 |
Config changes
26 fields changed, 101 unchanged.
| Field | Original | Fixture |
|---|---|---|
| audio_config.hidden_size | 1024 | 64 |
| audio_config.num_attention_heads | 8 | 4 |
| audio_config.num_hidden_layers | 12 | 2 |
| text_config.head_dim | 256 | 16 |
| text_config.hidden_size | 1536 | 64 |
| text_config.hidden_size_per_layer_input | 256 | 16 |
| text_config.intermediate_size | 6144 | 128 |
| text_config.layer_types | 35 entries: 28× sliding_attention, 7× full_attention | ["sliding_attention", "full_attention", "sliding_attention", "full_attention"] |
| text_config.num_attention_heads | 8 | 4 |
| text_config.num_hidden_layers | 35 | 4 |
| text_config.num_kv_shared_layers | 20 | 2 |
| text_config.per_layer_config.04 | {"head_dim": 512} | — |
| text_config.per_layer_config.09 | {"head_dim": 512} | — |
| text_config.per_layer_config.1 | — | {"head_dim": 512} |
| text_config.per_layer_config.14 | {"head_dim": 512} | — |
| text_config.per_layer_config.19 | {"head_dim": 512} | — |
| text_config.per_layer_config.24 | {"head_dim": 512} | — |
| text_config.per_layer_config.29 | {"head_dim": 512} | — |
| text_config.per_layer_config.3 | — | {"head_dim": 512} |
| text_config.per_layer_config.34 | {"head_dim": 512} | — |
| vision_config.head_dim | 64 | 16 |
| vision_config.hidden_size | 768 | 64 |
| vision_config.intermediate_size | 3072 | 128 |
| vision_config.num_attention_heads | 12 | 4 |
| vision_config.num_hidden_layers | 16 | 2 |
| vision_config.num_key_value_heads | 12 | 4 |