deepmind/language-perceiver

huggingface-mic/language-perceiver-mic@6c70b7e085e8 ← deepmind/language-perceiver@47a1ba43d819 · PerceiverForMaskedLM · random weights · built before the shrink rule (no recorded recipe)

Original201.1Mparameters

Fixture194.1Mparameters

Layer stacks1compared

Findings1to review

⚠ To review

One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.

Original · 26

Fixture · 26

Kind	Original	Fixture
Layer	26	26	✓ kept

Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.

Original

Fixture

Component	Original	Share	Fixture	Share
perceiver.encoder.self_attends	187.7M	93.3%	187.7M	96.7%
perceiver.encoder	6.4M	3.2%	5.4M	2.8%
perceiver.decoder	4.9M	2.4%	572.9K	0.3%
perceiver.input_preprocessor	1.8M	0.9%	147.8K	0.1%
perceiver.embeddings	327.7K	0.2%	327.7K	0.2%
embedding_decoder	262	0.0%	262	0.0%

Config	Property	Original	Fixture
model	model_type	perceiver	perceiver	✓ kept
model	vocab_size	262	262	✓ kept
model	hidden_act	gelu	gelu	✓ kept
model	max_position_embeddings	2048	2048	✓ kept
model	image_size	56	56	✓ kept

Instances of each class in the built model (prefix Perceiver omitted).

Class	Original	Fixture
Linear	168	168
LayerNorm	58	58
Dropout	28	28
GELUActivation	28	28
Identity	28	28
Attention	28	28
Layer	28	28
MLP	28	28
SelfAttention	28	28
SelfOutput	28	28
Embedding	2	2
ModuleList	1	1
BasicDecoder	1	1
EmbeddingDecoder	1	1
Embeddings	1	1
Encoder	1	1
ForMaskedLM	1	1
Model	1	1
TextPreprocessor	1	1
TrainablePositionEncoding	1	1

1 fields changed, 35 unchanged.

Field	Original	Fixture
d_model	768	64