Part 1: What are the different types Of Attention Mechanisms? | Glue Labs

Part 1: What are the different types Of Attention Mechanisms?

Attention mechanisms differ based on where the particular attention mechanism or model finds its application. Another distinction is the areas or relevant parts of the input sequence where the model focuses and places its attention.

The following are the types of attention mechanisms:

1. Generalized Attention
2. Self-Attention
3. Multi-Head attention
4. Additive Attention
5. Global Attention

Let’s take a closer look at these types.

1. Generalized Attention

When a sequence of words or an image is fed to a generalized attention model, it verifies each element of the input sequence and compares it against the output sequence.

So, each iteration involves the mechanism's encoder capturing the input sequence and comparing it with each element of the decoder's sequence.

From the comparison scores, the mechanism then selects the words or parts of the image that it needs to pay attention to.

2. Self-Attention

The self-attention mechanism is also sometimes referred to as the intra-attention mechanism.

It is so-called because it picks up particular parts at different positions in the input sequence and over time it computes an initial composition of the output sequence.

It does not take into consideration the output sequence as there is no manual data entry procedure where the prediction of the output sequence is assisted in any way.

In our next post (part 2), we'll take this discussion forward, discussing the different types of attention mechanisms. So stay tuned!