260810 AI 강의Ⅱ- 자연어 처리 기본

Main ideas / lessons conveyed

1) Course framing and goals

• The video is an AI lecture (Samsung Software AI Academy) that builds from basic AI concepts toward deeper natural language processing (NLP).
• It promises a “foundation → deeper → real understanding” progression:
  • Session 1: Text comprehension and generation basics
  • Session 2: Word embeddings and sequence models (RNN/LSTM) leading toward language models
  • Session 3–4: Transformers (self-attention, positional encoding, multi-head attention, encoder–decoder structure)

2) NLP tasks and the structure of “text”

• Text is treated as hierarchical structure:
  • Words → sentences → longer connected documents
• Key conceptual challenge:
  • Words aren’t numeric, but machine learning models require numbers.
• Therefore, NLP relies on representing linguistic units (words/tokens/sentences) as vectors.

3) Text comprehension & text generation view

• The lecture splits into core directions:
  1. Understanding text
  2. Generating text
  3. Later: sequence modeling and language models / Transformers to support both

---

4) Word representation: from one-hot vectors to embeddings

One-hot encoding (baseline)

• A vocabulary/dictionary maps each word to a unique index.
• A word becomes a vector with:
  • 1 at its index
  • 0 elsewhere
• Example logic:
  • “hotel” = 1, “conference” = 0, “work” = 0 (in that scheme)

Limitations of one-hot representations

• Vectors are sparse and orthogonal:
  • Similar words (e.g., size vs capacity) do not become similar in vector space.
• When using dot products for similarity/search:
  • unrelated pairs often look equally unrelated (similarity fails).
• High dimensionality introduces issues:
  • Dimensional curse:
    • as vocabulary size grows, dimensions become huge
    • patterns become harder to learn/recognize

Word embeddings (the fix)

• Word embeddings aim to make related words have closer vector representations.
• Main idea:
  • Instead of treating each word as independent, use context:
    • learn a word meaning based on words appearing before/after it
• Result:
  • embedding dimensions are much smaller than one-hot
  • embeddings are continuous values (dense vectors), enabling similarity via dot product/cosine-like behavior

---

5) Word2Vec methodology (Skip-gram and CBOW)

The lecture explains Word2Vec as an encoder–decoder style learning setup.

Concepts

• Learn embeddings such that:
  • predict surrounding words given a center word (Skip-gram)
  • predict center word given surrounding words (CBOW)

Training procedure (detailed)

• Prepare a corpus of sentences (e.g., from web text).
• Choose a context window size.
• For each position in a sentence:
  • Identify the center word (w_t)
  • Identify surrounding words within the window
• Create training pairs:
  • Skip-gram
    • Input: center word
    • Output: each surrounding word
    • Learn embeddings by maximizing probability of observed context words
  • CBOW
    • Input: surrounding words
    • Output: center word
    • Learn embeddings by maximizing probability of the center word

Learning mechanism (how gradients update)

• Use an encoder–decoder structure:
  • Encoder maps input word(s) → embedding vector(s)
  • Decoder predicts a probability distribution over the vocabulary
• Compute loss as the difference between:
  • predicted distribution vs ground-truth target distribution
• Backpropagation updates encoder/decoder parameters.
• Tradeoff described:
  • Skip-gram uses more training updates per center word → slower
  • Can yield better representations for rarer words/phrases

---

6) From embeddings to sequence models: why order matters

• Word embeddings alone don’t fully capture:
  • word order
  • long-range dependencies
• Sequence data matters because:
  • syntax/meaning changes with order
  • sentences grow variable length
• Therefore:
  • use RNN/LSTM to process sequences (variable-length input/output)

7) RNN basics (handling variable sequences)

• RNN processes text one step at a time:
  • hidden state (h_t) summarizes previous context
• Recurrence:
  • new hidden state is computed from previous hidden state and current input
• Training:
  • loss computed at output(s)
  • backpropagation through time updates shared parameters

Major RNN problem: vanishing gradients

• Because the same transformations are repeatedly applied across time steps:
  • gradients may shrink
• Result:
  • early information becomes difficult to preserve
  • long dependencies are harder to learn

8) LSTM: gated memory for long-term dependencies

• LSTM introduces:
  • hidden state (h_t)
  • cell state (c_t) (long-term memory)
• LSTM uses three main gates:
  • Forget gate: decide what to erase from cell state
  • Input gate: decide what new information to add
  • Output gate: decide what part becomes (h_t)

Gate operation (conceptual)

• Forget gate uses a sigmoid producing values in [0,1]
  • near 0: forget
  • near 1: keep
• Input gate similarly controls writing to the cell state.
• Output gate controls what part becomes (h_t).

Benefit

• Selective retention:
  • keep useful long-term information
  • discard irrelevant details
• More robust learning of sentence-level tasks (sentiment, translation, etc.)

---

9) Language models and text generation framing

• A language model estimates probability of sequences:
  • given context → probability of next word
• Generation:
  • repeatedly sample/predict next token until an end condition
• Linked applications:
  • autocomplete, correction, translation, summarization, chat, code generation

---

10) Encoder–decoder learning for sequence-to-sequence tasks (translation/summarization)

Encoder–decoder concept

• Encoder: reads input sequence and encodes it into representations.
• Decoder: generates output sequence token by token.

Training view

• Use ground-truth targets and compute loss between predicted output distribution and correct distribution.
• At inference, generate tokens sequentially using model predictions.

11) Beam search for decoding

• Greedy decoding keeps only one best token path.
• Beam search keeps K best candidate sequences at each step.
• Procedure:
  • maintain K hypotheses
  • expand each hypothesis with top next tokens
  • prune back to K
  • stop when candidates produce an end-of-sentence token
• Length effects handled (noted in lecture):
  • via log probabilities and normalization/heuristics (e.g., average probability)

---

12) Attention: fixing the bottleneck of “compress everything into one vector”

• Problem:
  • encoder outputs must be compressed into a fixed-size representation; long inputs lose information.
• Attention allows the decoder to retrieve relevant encoder states dynamically for each output step.
• Mechanism:
  • compute attention scores (alignment) between:
    • decoder state (query) and encoder states (keys)
  • convert scores to weights via softmax
  • compute a weighted sum of encoder value vectors
  • use this combined context to predict the next token
• Benefit:
  • stronger performance on long sequences
  • improved gradient flow (less severe information bottleneck)

---

13) Transformers overview (sessions 3–4 focus)

Motivation

• Transformers overcome RNN limitations:
  • sequential slow computation
  • difficulty capturing very long dependencies

Core idea: self-attention in parallel

• Self-attention computes context in parallel across all tokens.

Key components described

• Self-attention
  • for each token, compare it to all tokens in the same sequence
  • produce new contextualized token representations
• Positional encoding
  • attention alone lacks explicit order awareness
  • add position information to token embeddings
• Masked self-attention (decoder)
  • prevents attending to future tokens (autoregressive generation)

Multi-head attention

• Instead of one attention calculation:
  • perform multiple attention “heads”
• Each head can attend to different relational patterns.
• Multi-head helps capture diverse dependencies.

Transformer module structure (as described)

• Encoder layer:
  • multi-head self-attention + feed-forward network
• Decoder layer:
  • masked multi-head self-attention + encoder–decoder attention + feed-forward network
• Residual connections and layer normalization:
  • stabilize training and help gradients

Scaled dot-product attention

• Attention score uses dot products.
• To keep softmax stable:
  • scale by a factor related to dimensionality (e.g., division by (\sqrt{d_k}))
• This prevents extremely large logits that would saturate attention.

---

14) Wrap-up of lecture themes

• NLP pipeline progression emphasized:
  • one-hot → word embeddings → Word2Vec → RNN/LSTM → language models → seq2seq encoder–decoder → beam search → attention → Transformers
• Repeated teaching goal:
  • understand the why (limitations and motivations) before formulas/architectures.

---

Speakers / sources featured (as named in subtitles)

1. Professor Yeo Jin-young (Yonsei University, Department of Artificial Intelligence)
2. Pro Haerin Kang (from SAPI Secretariat)
3. (Group/collective acknowledgments in subtitles; no additional named speakers)