To calculate the number of parameters in a BERT model, let's break down each part of the architecture:

1. Embedding Layer

The embedding layer consists of:

• Token Embeddings: Since the vocabulary size is $40,000$ and the embedding dimension is $768$, the number of parameters for token embeddings will be: $\text{Token Embedding Params} = 40,000 \times 768 = 30,720,000$

• Position Embeddings: BERT uses positional embeddings for a sequence of up to $512$ tokens, and the embedding dimension is again $768$: $\text{Position Embedding Params} = 512 \times 768 = 393,216$

• Segment Embeddings: BERT typically uses two segment embeddings (for sentence A and sentence B in NSP tasks), so: $\text{Segment Embedding Params} = 2 \times 768 = 1,536$

Total Embedding Parameters: $\text{Total Embedding Params} = 30,720,000 + 393,216 + 1,536 = 31,114,752$

2. Transformer Layers (8 Layers)

Each Transformer layer consists of:

1. Multi-head Attention:

   • In each layer, we have 8 attention heads, each with a query, key, and value matrix. Each of these matrices transforms the 768-dimensional input to $768/8 = 96$-dimensional outputs (one per head). So for each head, the number of parameters is $768 \times 96$, and there are 3 such matrices (query, key, value) for each head.
   • Additionally, there is an output projection matrix to map back to 768 dimensions.

   Therefore, the number of parameters for multi-head attention per layer is: $\text{Attention Params} = 3 \times (768 \times 96) \times 8 + (768 \times 768)$ $\text{Attention Params} = 3 \times 768 \times 96 \times 8 + 768 \times 768 = 1,769,472$

2. Feed-Forward Network (FFN):

   • The FFN consists of two linear transformations: from 768 dimensions to 3072 dimensions (the hidden layer), and then back from 3072 to 768 dimensions. $\text{FFN Params} = (768 \times 3072) + (3072 \times 768) = 2,362,368 + 2,362,368 = 4,724,736$

3. Layer Norm and Other Parameters: Typically, Layer Norm would add a small number of parameters, but since we're ignoring bias terms and other small contributions, we'll focus on the main weights.

Total Parameters per Transformer Layer: $\text{Total per Layer} = 1,769,472 (\text{Attention}) + 4,724,736 (\text{FFN}) = 6,494,208$

Since there are 8 layers, the total number of parameters for all layers is: $\text{Total Transformer Layer Params} = 6,494,208 \times 8 = 51,953,664$

3. Total Number of Parameters

Now, summing up the parameters from the embedding layer and the transformer layers:

$\text{Total Params} = 31,114,752 (\text{Embeddings}) + 51,953,664 (\text{Transformer Layers})$ $\text{Total Params} = 83,068,416$

Thus, the total number of parameters in the BERT model is approximately 83 million.

---

Let me know if you'd like to explore any particular details or have further questions.

Here are 5 related questions:

1. How does increasing the number of layers affect the total number of parameters?
2. What role does the dimensionality of the hidden states play in the model's capacity?
3. How does the size of the vocabulary affect the embedding layer size?
4. What impact do attention heads have on multi-head attention's parameter count?
5. How can we reduce the number of parameters without changing the model structure?

Tip: Increasing hidden dimensions significantly increases the parameter count, especially in the feed-forward networks.