Building a Vision Transformer Model from Scratch with PyTorch

Tech tutorial summary: Vision Transformer (ViT) from scratch in PyTorch

Goal / dataset

• Build a Vision Transformer from scratch and train it on CIFAR-10 for image classification.
• Dataset details:
  • 10 classes
  • 60k images
  • Image size: 32×32×3

Approach

• Use an end-to-end pipeline in Google Colab:
  1. Review theory
  2. Implement ViT components
  3. Train and evaluate the model
  4. Visualize predictions
  5. Repeat with data augmentation and compare results

---

ViT architecture concepts covered

Transformer foundation (encoder-only)

• Uses Transformer encoder blocks (no decoder, unlike typical LLM setups).
• Each encoder block includes:
  • Multi-head self-attention
  • Add + LayerNorm (residual connection)
  • Feed-forward (MLP) layer
  • Add + LayerNorm again

Patch embedding (image → sequence)

• Split the input image into fixed-size patches.
• Flatten patches since transformers operate on 1D sequences.
• Use a linear projection implemented via Conv2D:
  • kernel size and stride equal to patch size
  • to map patches into an embedding dimension

Positional encoding

• Added because transformers lack an inherent notion of spatial order.

CLS token and classification head

• Use a learnable CLS token prepended to the patch sequence.
• The CLS representation is passed to:
  • an MLP (multi-layer perceptron)
  • producing logits (softmax is implied for class probabilities)

---

Training and evaluation workflow (implementation guide)

Environment setup

• Run in Google Colab with T4 GPU.
• Version checks (as used in the project):
  • torch 2.6.0
  • torchvision ~0.21.0
  • CUDA ~12.4
• Device-agnostic code:
  • use cuda if available, otherwise cpu
• Set random seeds:
  • manual seed = 42

Core hyperparameters

• epochs = 10
• batch_size = 128
• learning_rate = 3e-4
• patch_size = 4
• embed_dim = 256
• num_heads = 8
• depth = 6
• mlp_dim = 512
• drop_rate = 0.1

Data preprocessing (baseline)

• Transformations:
  • ToTensor
  • Normalize(mean=0.5, std=0.5)

Data loaders

• Convert datasets into PyTorch DataLoaders:
  • Train loader: shuffle=True
  • Test loader: shuffle=False
• Training setup:
  • Cross-entropy loss
  • Adam optimizer

Training loop

• Use model.train()
• For each batch:
  • move X, y to device
  • optimizer.zero_grad()
  • forward pass
  • compute loss
  • loss.backward()
  • optimizer.step()
• Track accuracy via argmax over logits.

Evaluation loop

• Use model.eval()
• Wrap inference in torch.inference_mode() (no gradients)
• Compute accuracy on the test set.

---

Experiment 1: Training without data augmentation

• Train for 10 epochs.
• Approximate reported results:
  • Train accuracy: ~75%
  • Test accuracy: ~62%
• Includes:
  • plots of train vs. test accuracy over epochs
  • prediction visualization

---

Experiment 2: Fine-tuning with data augmentation

• Add augmentation to the training transform while keeping normalization:
  • RandomCrop (size 32, padding 4)
  • RandomHorizontalFlip
  • ColorJitter (brightness/contrast/saturation/hue ~0.2 each)
• Recreate the model with fresh weights and retrain with the same architecture/training setup.
• Approximate reported results:
  • Train accuracy: ~76%
  • Test accuracy: ~64%
• Updated accuracy curves and improved prediction quality.

---

Prediction / visualization tutorial (important feature)

• Implements a predict_and_plot_grid function:
  • sets model.eval()
  • randomly samples images from the test dataset
  • runs inference and uses argmax for prediction
  • unnormalizes images for correct plotting (reverses Normalize(0.5, 0.5))
  • displays a 3×3 grid of images
  • titles show true vs. predicted class:
    • green for correct predictions
    • red for incorrect predictions

---

Notable practical debugging detail

• Encountered a model instantiation error involving requires_grad being passed incorrectly (from subtitles: “make_subclass argument required grad … must be boolean not integer”).
• Fix described:
  • correct the placement/usage of position_embedding initialization
  • ensure it is created with the proper boolean/parameter setup

---

Main speakers / sources

• Speaker / course creator: Muhammad Al Abra
• Primary referenced data source: PyTorch TorchVision CIFAR-10 dataset (torchvision.datasets.CIFAR10)