zai-org/GLM-5.3-Flash

huggingface-mic/GLM-5.3-Flash-mic@53f3e5bd10a6 ← zai-org/GLM-5.3-Flash@46a050bec130 · Glm5NextForConditionalGeneration · random weights · built before the shrink rule (no recorded recipe)

Original313.9Bparameters

Fixture40.7Mparameters

Layer stacks2compared

Findings1to review

⚠ To review

One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.

Original · 24

Fixture · 2

Kind	Original	Fixture
VisionBlock	24	2	✓ kept

Original · 45

Fixture · 3

Kind	Original	Fixture
linear_attention, Conv1d, TextForgetGate …	3	1	✓ kept
is_causal=True, TextAttention, TextExperts …	11	1	✓ kept
linear_attention, Conv1d, TextExperts …	31	1	✓ kept

Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.

Original

Fixture

Component	Original	Share	Fixture	Share
model.language_model.layers	312.1B	99.4%	811.0K	2.0%
model.language_model	634.4M	0.2%	19.8M	48.7%
lm_head	634.4M	0.2%	19.8M	48.7%
model.visual.blocks	403.0M	0.1%	83.4K	0.2%
model.visual	160.6M	0.1%	174.1K	0.4%

Config	Property	Original	Fixture
model	model_type	glm5_next	glm5_next	✓ kept
model	tie_word_embeddings	False	False	✓ kept
text_config	attention style	MHA (64 heads, 64 KV)	MHA (4 heads, 4 KV)	✓ kept
text_config	MoE routing	top-8 of 288 experts	top-2 of 8 experts	✓ kept
text_config	model_type	glm5_next_text	glm5_next_text	✓ kept
text_config	vocab_size	154880	154880	✓ kept
text_config	tie_word_embeddings	False	False	✓ kept
text_config	hidden_act	silu	silu	✓ kept
text_config	max_position_embeddings	1048576	1048576	✓ kept
vision_config	RoPE type	axial	axial	✓ kept
vision_config	model_type	glm5_next_vision	glm5_next_vision	✓ kept
vision_config	hidden_act	silu	silu	✓ kept
vision_config	image_size	448	56	⚠ changed
vision_config	patch_size	14	14	✓ kept

Instances of each class in the built model (prefix Glm5Next omitted).

33 fields changed, 86 unchanged.

Field	Original	Fixture
quantization_config	{"activation_scheme": "dynamic", "fmt": "e4m3", "modules_to_not_convert": ["a…	—
text_config.first_k_dense_replace	3	1
text_config.hidden_size	4096	128
text_config.index_head_dim	128	32
text_config.index_n_heads	32	4
text_config.index_topk	2048	64
text_config.indexer_types	45 entries: 45× full	["full", "full", "full"]
text_config.intermediate_size	12288	128
text_config.kv_lora_rank	512	64
text_config.layer_types	45 entries: 34× linear_attention, 11× indexed_attention	["linear_attention", "indexed_attention", "linear_attention"]
text_config.linear_attn_config.full_attn_layers	11 entries	[1]
text_config.linear_attn_config.head_dim	128	32
text_config.linear_attn_config.kda_layers	34 entries	[0, 2]
text_config.linear_attn_config.num_heads	64	4
text_config.linear_head_dim	128	32
text_config.linear_num_heads	64	4
text_config.mlp_layer_types	45 entries: 42× sparse, 3× dense	["dense", "sparse", "sparse"]
text_config.moe_intermediate_size	2048	64
text_config.n_routed_experts	288	8
text_config.num_attention_heads	64	4
text_config.num_experts_per_tok	8	2
text_config.num_hidden_layers	45	3
text_config.num_key_value_heads	64	4
text_config.q_lora_rank	1536	64
text_config.qk_nope_head_dim	256	32
text_config.v_head_dim	256	32
vision_config.depth	24	2
vision_config.hidden_size	1024	64
vision_config.image_size	448	56
vision_config.intermediate_size	4096	128
vision_config.num_heads	16	4
vision_config.out_hidden_size	4096	128
vision_config.projection_intermediate_size	10240	128