← MIC coverage of Transformers model types

Qwen/Qwen2.5-Omni-7B

huggingface-mic/qwen2_5_omni-mic@72e87592273c ← Qwen/Qwen2.5-Omni-7B@ae9e1690543f · Qwen2_5OmniForConditionalGeneration · random weights · built before the shrink rule (no recorded recipe)

CI: failing now on main (28 failed checks in 30 days) · MIC failures dashboard

Original10.7Bparameters

Fixture170.4Mparameters

Layer stacks8compared

Findings0to review

✓ Same architecture, smaller

Every module class, layer kind and checked property of the original is in the fixture.

Layer stacks

One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.

thinker.model.audio_tower.layers

Original · 32

Fixture · 2

Kind	Original	Fixture
AudioEncoderLayer	32	2	✓ kept

thinker.model.visual.blocks

Original · 32

Fixture · 32

Kind	Original	Fixture
VisionBlock	32	32	✓ kept

thinker.model.language_model.layers

Original · 28

Fixture · 2

Kind	Original	Fixture
DecoderLayer	28	2	✓ kept

talker.model.layers

Original · 24

Fixture · 2

Kind	Original	Fixture
DecoderLayer	24	2	✓ kept

token2wav.code2wav_dit_model.input_embed.spk_encoder.blocks

Original · 4

Fixture · 4

Kind	Original	Fixture
no ModuleList	1	1	✓ kept
ModuleList, Res2NetBlock, Sigmoid …	3	3	✓ kept

token2wav.code2wav_dit_model.transformer_blocks

Original · 22

Fixture · 2

Kind	Original	Fixture
DiTDecoderLayer	22	2	✓ kept

token2wav.code2wav_bigvgan_model.ups

Original · 6

Fixture · 6

Kind	Original	Fixture
ModuleList	6	6	✓ kept

token2wav.code2wav_bigvgan_model.resblocks

Original · 18

Fixture · 18

Kind	Original	Fixture
AMPBlock	18	18	✓ kept

Where the parameters are

Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.

Original

Fixture

Component	Original	Share	Fixture	Share
thinker.model.language_model.layers	6.5B	60.8%	74.2K	0.0%
talker.model.layers	1.3B	12.2%	74.2K	0.0%
thinker.model.visual.blocks	630.5M	5.9%	1.3M	0.8%
thinker.model.audio_tower.layers	629.6M	5.9%	66.8K	0.0%
thinker.model	601.1M	5.6%	10.2M	6.0%
thinker.lm_head	545.0M	5.1%	9.7M	5.7%
token2wav.code2wav_dit_model.transformer_blocks	323.3M	3.0%	116.4K	0.1%
8 others	166.9M	1.6%	148.8M	87.3%

Preserved properties

Config	Property	Original	Fixture
model	model_type	qwen2_5_omni	qwen2_5_omni	✓ kept
thinker_config	model_type	qwen2_5_omni_thinker	qwen2_5_omni_thinker	✓ kept
thinker_config	tie_word_embeddings	False	False	✓ kept
thinker_config.audio_config	model_type	qwen2_5_omni_audio_encoder	qwen2_5_omni_audio_encoder	✓ kept
thinker_config.audio_config	tie_word_embeddings	True	True	✓ kept
thinker_config.audio_config	num_mel_bins	128	128	✓ kept
thinker_config.vision_config	RoPE type	axial	axial	✓ kept
thinker_config.vision_config	model_type	qwen2_5_omni_vision_encoder	qwen2_5_omni_vision_encoder	✓ kept
thinker_config.vision_config	tie_word_embeddings	True	True	✓ kept
thinker_config.vision_config	hidden_act	silu	silu	✓ kept
thinker_config.vision_config	patch_size	14	14	✓ kept
thinker_config.text_config	attention style	GQA (28 heads, 4 KV)	GQA (4 heads, 2 KV)	✓ kept
thinker_config.text_config	RoPE type	default	default	✓ kept
thinker_config.text_config	model_type	qwen2_5_omni_text	qwen2_5_omni_text	✓ kept
thinker_config.text_config	vocab_size	152064	152064	✓ kept
thinker_config.text_config	tie_word_embeddings	False	False	✓ kept
thinker_config.text_config	hidden_act	silu	silu	✓ kept
thinker_config.text_config	max_position_embeddings	32768	32768	✓ kept
talker_config	attention style	GQA (12 heads, 4 KV)	GQA (4 heads, 2 KV)	✓ kept
talker_config	RoPE type	default	default	✓ kept
talker_config	model_type	qwen2_5_omni_talker	qwen2_5_omni_talker	✓ kept
talker_config	vocab_size	8448	8448	✓ kept
talker_config	tie_word_embeddings	False	False	✓ kept
talker_config	hidden_act	silu	silu	✓ kept
talker_config	max_position_embeddings	32768	32768	✓ kept
token2wav_config	model_type	qwen2_5_omni_token2wav	qwen2_5_omni_token2wav	✓ kept
token2wav_config.dit_config	RoPE type	default	default	✓ kept
token2wav_config.dit_config	model_type	qwen2_5_omni_dit	qwen2_5_omni_dit	✓ kept
token2wav_config.dit_config	tie_word_embeddings	True	True	✓ kept
token2wav_config.dit_config	max_position_embeddings	32768	32768	✓ kept
token2wav_config.bigvgan_config	model_type	qwen2_5_omni_bigvgan	qwen2_5_omni_bigvgan	✓ kept
token2wav_config.bigvgan_config	tie_word_embeddings	True	True	✓ kept

Module classes

Instances of each class in the built model (prefix Qwen2_5Omni omitted).

Class	Original	Fixture
Linear	945	289
RMSNorm	171	75
Conv1d	132	132
ModuleList	116	76
LayerNorm	110	10
AntiAliasedActivation1d	109	109
DownSample1d	109	109
SnakeBeta	109	109
UpSample1d	109	109
SiLUActivation	84	36
Qwen2MLP	52	4
Attention	52	4
DecoderLayer	52	4
Dropout	44	4
GELUActivation	32	2
AudioAttention	32	2
AudioEncoderLayer	32	2
MLP	32	32
VisionAttention	32	32
VisionBlock	32	32
SiLU	24	4
GELU	23	3
DiTAttention	22	2
DiTDecoderLayer	22	2
DiTMLP	22	2
Qwen2_5_OmniAdaLayerNormZero	22	2
AMPBlock	18	18
ReLU	15	15
TimeDelayNetBlock	12	12
ConvTranspose1d	6	6
Embedding	4	4
Res2NetBlock	3	3
Sigmoid	3	3
SqueezeExcitationBlock	3	3
SqueezeExcitationRes2NetBlock	3	3
RotaryEmbedding	2	2
AttentiveStatisticsPooling	1	1
AvgPool1d	1	1
Conv3d	1	1
DiTCodecEmbedding	1	1
DiTInputEmbedding	1	1
DiTTimestepEmbedding	1	1
ECAPA_TimeDelayNet	1	1
AudioEncoder	1	1
DiTRotaryEmbedding	1	1
ForConditionalGeneration	1	1
PatchMerger	1	1
TalkerForConditionalGeneration	1	1
TalkerModel	1	1
ThinkerForConditionalGeneration	1	1
ThinkerModel	1	1
ThinkerTextModel	1	1
Token2WavBigVGANModel	1	1
Token2WavDiTModel	1	1
Token2WavModel	1	1
VisionEncoder	1	1
VisionRotaryEmbedding	1	1
Qwen2_5_OmniAdaLayerNormZero_Final	1	1
Qwen2_5_VisionPatchEmbed	1	1
Sequential	1	1
SinusPositionEmbedding	1	1
SinusoidsPositionEmbedding	1	1
Tanh	1	1

Config changes

30 fields changed, 281 unchanged.

Field	Original	Fixture
talker_config.head_dim	128	16
talker_config.hidden_size	896	64
talker_config.intermediate_size	18944	128
talker_config.layer_types	24 entries: 24× full_attention	["full_attention", "full_attention"]
talker_config.num_attention_heads	12	4
talker_config.num_hidden_layers	24	2
talker_config.num_key_value_heads	4	2
talker_config.rope_parameters	{"mrope_section": [16, 24, 24], "rope_type": "default", "type": "default", "r…	{"mrope_section": [2, 3, 3], "rope_theta": 1000000.0, "rope_type": "default",…
thinker_config.audio_config.d_model	1280	64
thinker_config.audio_config.encoder_attention_heads	20	4
thinker_config.audio_config.encoder_ffn_dim	5120	128
thinker_config.audio_config.encoder_layers	32	2
thinker_config.text_config.hidden_size	3584	64
thinker_config.text_config.intermediate_size	18944	128
thinker_config.text_config.layer_types	28 entries: 28× full_attention	["full_attention", "full_attention"]
thinker_config.text_config.num_attention_heads	28	4
thinker_config.text_config.num_hidden_layers	28	2
thinker_config.text_config.num_key_value_heads	4	2
thinker_config.text_config.rope_parameters	{"mrope_section": [16, 24, 24], "rope_type": "default", "type": "default", "r…	{"mrope_section": [2, 3, 3], "rope_theta": 1000000.0, "rope_type": "default",…
thinker_config.vision_config.embed_dim	1280	64
thinker_config.vision_config.hidden_size	1280	64
thinker_config.vision_config.intermediate_size	3420	128
thinker_config.vision_config.num_heads	16	4
thinker_config.vision_config.out_hidden_size	3584	64
token2wav_config.dit_config.dim	1024	64
token2wav_config.dit_config.emb_dim	512	64
token2wav_config.dit_config.head_dim	64	16
token2wav_config.dit_config.hidden_size	1024	64
token2wav_config.dit_config.num_attention_heads	16	4
token2wav_config.dit_config.num_hidden_layers	22	2