Transformers

Understanding Transformers: The Magic Behind Modern AI
AI
Deep Learning
NLP
Study
Author

Mahmut Osmanovic

Published

July 22, 2024

1.0 | Understanding Transformers: The Magic Behind Modern AI

In the world of artificial intelligence (AI), there’s a revolutionary tool called “transformers” that’s making waves. Whether it’s translating languages, writing text, or even generating art, transformers are at the heart of many cutting-edge technologies. But what exactly are transformers, and how do they work? Let’s dive in and explore this fascinating topic.

2.0 | What Are Transformers?

Transformers are a type of AI model designed to understand and generate human language. They were introduced in a groundbreaking paper titled “Attention is All You Need” by Vaswani et al. in 2017. Unlike previous models that processed data sequentially (one word after another), transformers can process all words in a sentence simultaneously. This parallel processing capability makes them incredibly powerful and efficient.

3.0 | How Do Transformers Work?

To grasp how transformers work, let’s imagine a magical friend who loves reading stories. This friend has a unique ability: they don’t read stories one word at a time. Instead, they look at the whole sentence at once, which helps them understand the context and meaning much better.

When our magical friend reads a story, they:

  1. Start with the first word and understand its meaning.
  2. Add the next word and immediately look at both words together to see how they fit.
  3. Continue adding words one by one, each time looking at the entire sentence they’ve read so far to ensure it makes sense.

This is essentially how transformers work. They start with the first word, then add the next word, and each time they consider all the words in the sentence to understand their relationships and meanings. This method allows transformers to grasp the context and generate coherent and contextually accurate text.

4.0 | The Role of Self-Attention

The real magic behind transformers lies in their self-attention mechanism. Self-attention allows the model to focus on different parts of the input sentence with varying intensity, depending on their relevance. For instance, in the sentence “The cat sat on the mat,” the model can focus more on the relationship between “cat” and “sat” to understand the action taking place.

5.0 | Mathematical Overview of Self-Attention

Let’s break down how self-attention works mathematically:

  1. Input Representation: Each word in a sentence is first converted into a numerical vector, typically using an embedding layer. Suppose we have a sentence with \(n\) words, represented as \(X=[x_1, x_2, \ldots, x_n]\), where \(x_i\) is the vector for the \(i\)-th word.

  2. Query, Key, and Value Vectors: For each word, we create three vectors: Query (\(Q\)), Key (\(K\)), and Value (\(V\)). These vectors are obtained by multiplying the input vector \(x_i\) with three different weight matrices \(W^Q\), \(W^K\), and \(W^V\):

    • \(Q\) = \(XW^{Q}\), \(\quad K\) = \(XW^K\), \(\quad V\) = \(XW^V\)

  3. Calculating Attention Scores: The attention score for each word pair is calculated by taking the dot product of the Query vector of one word with the Key vector of another word. This gives a matrix of scores \(S\):

    • \(S\) = \(QK^T\)

  4. Scaled Dot-Product Attention: To make the scores more stable, we scale them by the square root of the dimension of the Key vectors \(d_k\), and then apply a softmax function to get the attention weights \(A\):

    • \(A = \text{softmax}\left(\frac{S}{\sqrt{d_k}}\right)\)

  5. Generating the Output: Finally, we multiply the attention weights \(A\) with the Value vectors \(V\) to get the output \(O\):

    • \(O = AV\)

This process allows the model to weigh the importance of each word in the context of the entire sentence, effectively capturing the relationships and dependencies between words.

6.0 | The Transformer Architecture

The architecture of a transformer model consists of two main parts, the encoder and the decoder. Each part is composed of multiple layers that work together to process the input and generate the output.

Transformer Architecture - Vaswani et al. in 2017

Encoder

  • Input Embedding: Converts the input words into numerical vectors.
  • Positional Encoding: Adds information about the position of each word in the sentence.
  • Multi-Head Attention: Applies self-attention mechanism multiple times in parallel, allowing the model to focus on different parts of the sentence simultaneously.
  • Add & Norm: Adds the original input to the output of the attention layer (residual connection) and normalizes it.
  • Residual Connections
    • They help to smooth out the loss landscape and make it easier for optimization algorithms to find good solutions. Additionally, residual connections tend to favor flatter stationary points, which can lead to better generalization.
  • Batch Normalization
    • Similar to Residual Connections, Batch Normalization helps to smooth out the loss landscape by normalizing the activations to have a mean (\(\mu\)) of 0 and a variance (\(\sigma^2\)) of 1. This stabilization accelerates training and reduces the model’s sensitivity to initialization values.
  • Feed Forward: Applies a fully connected feed-forward network to each position separately (applies non-linearity).
  • Add & Norm: Another residual connection followed by normalization.

Decoder

  • Output Embedding: Converts the output words (shifted right) into numerical vectors.
  • Positional Encoding: Adds positional information to the output embeddings.
  • Masked Multi-Head Attention: Applies self-attention to the output sequence, but with a mask to prevent attending to future positions.
  • Add & Norm: Residual connection and normalization.
  • Multi-Head Attention: Applies attention mechanism to the encoder’s output, allowing the decoder to focus on relevant parts of the input sentence.
  • Add & Norm: Residual connection and normalization.
  • Feed Forward: Fully connected feed-forward network (applies non-linearity).
  • Add & Norm: Residual connection and normalization.

Final Layer

  • Linear and Softmax: Converts the decoder’s output into probabilities over the vocabulary, generating the final output sequence.

7.0 | Why Are Transformers So Powerful?

Transformers have several advantages over previous models:

  • Parallel Processing: They can process multiple words simultaneously, making them faster and more efficient.
  • Contextual Understanding: By looking at the whole sentence, transformers can understand the context better, leading to more accurate and coherent text generation.
  • Scalability: Transformers can be scaled up easily, allowing them to handle vast amounts of data and perform complex tasks.

8.0 | Applications of Transformers

The versatility of transformers has led to their adoption in various applications:

  • Natural Language Processing (NLP): Transformers are used in language translation, sentiment analysis, and text summarization.
  • Content Generation: They can write articles, stories, and even code.
  • Speech Recognition: Transformers help in converting spoken language into written text.
  • Image and Video Processing: Models like Vision Transformers (ViTs) have extended the transformer architecture to excel in image recognition and video analysis. As of the date of this article, OmniVec, a visually based transformer, holds the top score in the ImageNet competition.

9.0 | Conclusion

Transformers have revolutionized the field of AI by enabling machines to understand and generate human language with remarkable accuracy. Their ability to process information in parallel and understand context makes them incredibly powerful tools. As research continues, we can expect transformers to drive even more innovations in AI, bringing us closer to a future where machines can seamlessly understand and interact with the world around us.