← MIC coverage of Transformers model types

Salesforce/blip2-opt-2.7b

huggingface-mic/blip2-opt-2.7b-mic@35cfe69ce3e1 ← Salesforce/blip2-opt-2.7b@59a1ef6c1e51 · Blip2ForConditionalGeneration · random weights

CI: no failure on main in 30 days · MIC failures dashboard

Original3.7Bparameters

Fixture3.6Mparameters

Layer stacks3compared

Findings0to review

✓ Same architecture, smaller

Every module class, layer kind and checked property of the original is in the fixture.

Model tester (advisory)

The model's own small test config in its ModelTester (40 literal values, 40 set by the model's own tester, the rest generic base-tester defaults). Testers are usually representative but not always, so these notes are for review, not rules.

The fixture covers every per-layer kind the tester sets, and shrinks every size it does.

Layer stacks

One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.

vision_model.encoder.layers

Original · 39

Fixture · 2

Kind	Original	Fixture
EncoderLayer	39	2	✓ kept

qformer.encoder.layer

Original · 12

Fixture · 2

Kind	Original	Fixture
QFormerLayer	12	2	✓ kept

language_model.model.decoder.layers

Original · 32

Fixture · 2

Kind	Original	Fixture
OPTDecoderLayer	32	2	✓ kept

Where the parameters are

Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.

Original

Fixture

Component	Original	Share	Fixture	Share
language_model.model.decoder.layers	2.5B	67.2%	66.9K	1.8%
vision_model.encoder.layers	984.8M	26.3%	66.9K	1.8%
language_model.model	134.0M	3.6%	3.4M	92.3%
qformer.encoder.layer	105.1M	2.8%	83.7K	2.3%
language_projection	2.0M	0.1%	4.2K	0.1%
vision_model.embeddings	1.2M	0.0%	54.2K	1.5%
query_tokens	24.6K	0.0%	2.0K	0.1%
2 others	4.4K	0.0%	256	0.0%

Preserved properties

Config	Property	Original	Fixture
model	model_type	blip-2	blip-2	✓ kept
vision_config	model_type	blip_2_vision_model	blip_2_vision_model	✓ kept
vision_config	hidden_act	gelu	gelu	✓ kept
vision_config	image_size	224	224	✓ kept
vision_config	patch_size	14	14	✓ kept
vision_config	num_channels	3	3	✓ kept
qformer_config	model_type	blip_2_qformer	blip_2_qformer	✓ kept
qformer_config	vocab_size	30522	30522	✓ kept
qformer_config	hidden_act	gelu	gelu	✓ kept
qformer_config	max_position_embeddings	512	512	✓ kept
text_config	model_type	opt	opt	✓ kept
text_config	vocab_size	50304	50304	✓ kept
text_config	tie_word_embeddings	True	True	✓ kept
text_config	max_position_embeddings	2048	2048	✓ kept

Module classes

Instances of each class in the built model (prefix Blip2 omitted).

Class	Original	Fixture
Linear	446	38
LayerNorm	175	16
GELUActivation	51	4
Attention	39	2
EncoderLayer	39	2
MLP	39	2
OPTAttention	32	2
OPTDecoderLayer	32	2
ReLU	32	2
Dropout	31	6
QFormerAttention	18	3
QFormerMultiHeadAttention	18	3
QFormerSelfOutput	18	3
QFormerIntermediate	12	2
QFormerLayer	12	2
QFormerOutput	12	2
ModuleList	3	3
Encoder	1	1
ForConditionalGeneration	1	1
QFormerEncoder	1	1
QFormerModel	1	1
VisionEmbeddings	1	1
VisionModel	1	1
Conv2d	1	1
Embedding	1	1
OPTDecoder	1	1
OPTForCausalLM	1	1
OPTLearnedPositionalEmbedding	1	1
OPTModel	1	1

Config changes

16 fields changed, 80 unchanged.

Field	Original	Fixture
qformer_config.encoder_hidden_size	1408	64
qformer_config.hidden_size	768	64
qformer_config.intermediate_size	3072	128
qformer_config.num_attention_heads	12	4
qformer_config.num_hidden_layers	12	2
text_config.ffn_dim	10240	128
text_config.hidden_size	2560	64
text_config.num_attention_heads	32	4
text_config.num_hidden_layers	32	2
text_config.word_embed_proj_dim	2560	64
vision_config.hidden_size	1408	64
vision_config.initializer_range	1e-10	0.02
vision_config.intermediate_size	6144	128
vision_config.num_attention_heads	16	4
vision_config.num_hidden_layers	39	2
vision_config.projection_dim	512	32