patrickvonplaten/bert2bert-cnn_dailymail-fp16

Original247.4Mparameters

Fixture4.2Mparameters

Layer stacks2compared

Findings0to review

✓ Same architecture, smaller

Every module class, layer kind and checked property of the original is in the fixture.

One square per layer, colored by kind: the layer's module classes and type attributes. Hover a square for its index and kind.

Original · 12

Fixture · 2

Kind	Original	Fixture
BertLayer	12	2	✓ kept

Original · 12

Fixture · 2

Kind	Original	Fixture
BertLayer	12	2	✓ kept

Share of each model's parameters per component. Vocabulary and patch sizes are kept, so embeddings usually take a larger share of the fixture.

Original

Fixture

Component	Original	Share	Fixture	Share
decoder.bert.encoder.layer	113.4M	45.9%	100.5K	2.4%
encoder.encoder.layer	85.1M	34.4%	66.9K	1.6%
encoder.embeddings	23.8M	9.6%	2.0M	47.5%
decoder.bert	23.8M	9.6%	2.0M	47.5%
decoder.cls	622.6K	0.3%	34.8K	0.8%
encoder.pooler	590.6K	0.2%	4.2K	0.1%

Config	Property	Original	Fixture
model	model_type	encoder-decoder	encoder-decoder	✓ kept
encoder	model_type	bert	bert	✓ kept
encoder	vocab_size	30522	30522	✓ kept
encoder	tie_word_embeddings	True	True	✓ kept
encoder	hidden_act	gelu	gelu	✓ kept
encoder	max_position_embeddings	512	512	✓ kept
decoder	model_type	bert	bert	✓ kept
decoder	vocab_size	30522	30522	✓ kept
decoder	tie_word_embeddings	True	True	✓ kept
decoder	hidden_act	gelu	gelu	✓ kept
decoder	max_position_embeddings	512	512	✓ kept

Instances of each class in the built model (prefix EncoderDecoder omitted).

Class	Original	Fixture
Linear	195	35
Dropout	98	18
LayerNorm	63	13
BertAttention	36	6
BertSelfOutput	36	6
GELUActivation	25	5
BertIntermediate	24	4
BertLayer	24	4
BertOutput	24	4
BertSelfAttention	24	4
BertCrossAttention	12	2
Embedding	6	6
BertEmbeddings	2	2
BertEncoder	2	2
BertModel	2	2
ModuleList	2	2
BertLMHeadModel	1	1
BertLMPredictionHead	1	1
BertOnlyMLMHead	1	1
BertPooler	1	1
BertPredictionHeadTransform	1	1
Model	1	1
Tanh	1	1

9 fields changed, 79 unchanged.