← MIC coverage of Transformers model types

Qwen/Qwen3.5-35B-A3B

huggingface-mic/Qwen3.5-35B-A3B-mic@ed07422cd836 ← Qwen/Qwen3.5-35B-A3B@59d61f3ce65a · Qwen3_5MoeForConditionalGeneration · random weights

CI: no failure on main in 30 days · MIC failures dashboard

Original35.1Bparameters

Fixture33.3Mparameters

Layer stacks2compared

Findings0to review

✓ Same architecture, smaller

Every module class, layer kind and checked property of the original is in the fixture.

Model tester (advisory)

The model's own small test config in its ModelTester (54 literal values, 34 set by the model's own tester, the rest generic base-tester defaults). Testers are usually representative but not always, so these notes are for review, not rules.

The fixture covers every per-layer kind the tester sets, and shrinks every size it does.

Layer stacks

One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.

model.visual.blocks

Original · 27

Fixture · 27

Kind	Original	Fixture
VisionBlock	27	27	✓ kept

model.language_model.layers

Original · 40

Fixture · 2

Kind	Original	Fixture
linear_attention, Conv1d, GatedDeltaNet …	30	1	✓ kept
is_causal=True, Attention	10	1	✓ kept

Where the parameters are

Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.

Original

Fixture

Component	Original	Share	Fixture	Share
model.language_model.layers	33.6B	95.8%	255.0K	0.8%
model.language_model	508.6M	1.4%	15.9M	47.8%
lm_head	508.6M	1.4%	15.9M	47.8%
model.visual.blocks	411.5M	1.2%	903.7K	2.7%
model.visual	35.1M	0.1%	328.2K	1.0%

Preserved properties

Config	Property	Original	Fixture
model	model_type	qwen3_5_moe	qwen3_5_moe	✓ kept
model	tie_word_embeddings	False	False	✓ kept
text_config	attention style	GQA (16 heads, 2 KV)	GQA (4 heads, 2 KV)	✓ kept
text_config	MoE routing	top-8 of 256 experts	top-2 of 4 experts	✓ kept
text_config	RoPE type	default	default	✓ kept
text_config	model_type	qwen3_5_moe_text	qwen3_5_moe_text	✓ kept
text_config	vocab_size	248320	248320	✓ kept
text_config	tie_word_embeddings	False	False	✓ kept
text_config	hidden_act	silu	silu	✓ kept
text_config	max_position_embeddings	262144	262144	✓ kept
vision_config	RoPE type	axial	axial	✓ kept
vision_config	model_type	qwen3_5_moe_vision	qwen3_5_moe_vision	✓ kept
vision_config	hidden_act	gelu_pytorch_tanh	gelu_pytorch_tanh	✓ kept
vision_config	patch_size	16	16	✓ kept

Module classes

Instances of each class in the built model (prefix Qwen3_5Moe omitted).

Class	Original	Fixture
Linear	461	128
RMSNorm	101	7
SiLUActivation	80	4
LayerNorm	55	55
DecoderLayer	40	2
Experts	40	2
MLP	40	2
SparseMoeBlock	40	2
TopKRouter	40	2
Conv1d	30	1
GatedDeltaNet	30	1
RMSNormGated	30	1
GELUTanh	27	27
VisionAttention	27	27
VisionBlock	27	27
VisionMLP	27	27
Attention	10	1
Embedding	2	2
ModuleList	2	2
Conv3d	1	1
GELU	1	1
ForConditionalGeneration	1	1
Model	1	1
TextModel	1	1
TextRotaryEmbedding	1	1
VisionModel	1	1
VisionPatchEmbed	1	1
VisionPatchMerger	1	1
VisionRotaryEmbedding	1	1

Config changes

16 fields changed, 66 unchanged.

Field	Original	Fixture
text_config.head_dim	256	16
text_config.hidden_size	2048	64
text_config.layer_types	40 entries: 30× linear_attention, 10× full_attention	["linear_attention", "full_attention"]
text_config.linear_key_head_dim	128	16
text_config.linear_value_head_dim	128	16
text_config.moe_intermediate_size	512	32
text_config.num_attention_heads	16	4
text_config.num_experts	256	4
text_config.num_experts_per_tok	8	2
text_config.num_hidden_layers	40	2
text_config.rope_parameters	{"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "defa…	{"mrope_interleaved": true, "mrope_section": [3, 3, 2], "partial_rotary_facto…
text_config.shared_expert_intermediate_size	512	128
vision_config.hidden_size	1152	64
vision_config.intermediate_size	4304	128
vision_config.num_heads	16	4
vision_config.out_hidden_size	2048	64