Тупик ИИ: Почему нейросети перестали умнеть?

Main ideas / concepts

• AI progress has stalled (“hit a wall”)

  • Early language-model progress was rapid (e.g., models learning to write normal text).
  • Many expected a second revolution that never fully arrived despite many improved releases.

• Modern LLMs are largely “next-token guessers”

  • Inference loop (how output is produced):
    • Input text (e.g., “Lyokha”) is converted into a vector representation.
    • The vector passes through billions of neural network parameters.
    • The result is compared against a dictionary matrix (vocabulary mapping).
    • The model outputs the token with the highest score.
    • The generated token is appended to the input, and the process repeats—so the model effectively uses trial-and-error generation to produce text.

• How tokens become vectors (embeddings)

  • Text is split into tokens.
  • Each token is mapped to an ID in the vocabulary/dictionary.
  • There is an embedding table: for every vocabulary ID, there is a base vector.
  • These embedding vectors are learned during training:
    • Initially random.
    • As the model trains on many texts, frequently co-occurring words end up closer in embedding space; rare words end up farther—based on statistics.

• Why models “understand” prompts (training explains it)

  • LLM training involves:
    • Pre-training: given lots of texts, the model learns to continue them.
    • Further training: given a prompt, it’s trained to produce a correct/desired answer.
  • The apparent “reasoning” and prompt-following abilities come from how the model was trained, not from explicit symbolic understanding.

• Why scaling and more data aren’t producing the same leap anymore

  • The video claims progress requires new data and scaling, but both hit limits.
  • Data deficit problem:
    • Companies use artificial/generated data, which is said to be worse on average.
  • Scaling cost problem:
    • Scaling is extremely expensive and is said to yield diminishing returns.
  • A quoted industry figure (Ilya Sutskeber) argues:
    • “The era of simply scaling up pre-training and getting a leap in intelligence is ending.”

• A newer direction: “resonance models”

  • Presented as a “new approach,” but framed as essentially the same LLMs with changes:
    1. The model generates internal reasoning tokens (decomposes tasks into subtasks).
    2. The model is additionally trained on data that is verifiable (e.g., math where results are always correct).
  • The claim: these models may not “truly think,” but they guess better and handle algorithms more reliably.
  • Trade-off: slower responses, better performance on programming/algorithmic tasks.
  • The speaker argues this is not a true breakthrough because similar capabilities existed before, just worse.

• Adding “body kits”: agents and multimodal extensions

  • Agents
    • Positioned as automating routine tasks across support, data/document management, etc.
    • The video portrays agents as an “exoskeleton” that boosts performance rather than a fundamental new intelligence breakthrough.
  • Image generators
    • Explained as using diffusion models, which operate on noise removal in image space.
    • The video states modern image generation can be similar in structure to text generation:
      • GPT Image: generates visual tokens like text tokens.
      • Pros: better context understanding, editing capabilities.
      • Cons: worse image quality.
    • Hybrids are mentioned (combining such models with diffusion-like methods), but the speaker sees no clear generational breakthrough.

• There will likely not be a near-term “new revolution” in LLMs

  • The video’s conclusion: progress is incremental; the major “revolution” won’t come just from better LLM scaling/variants.
  • The speaker suggests fundamental revolutions would require architecture changes and likely new hardware.

• Why new hardware might be needed

  • Claim: for a neuron to function effectively, compute and memory should be co-located (like the brain).
  • This motivates “hybrid” architectures combining different compute paradigms.

• Hybrid intelligence example: “Dishbrain”

  • Mention of researchers combining a processor and a mouse brain.
  • The claim: this hybrid system can learn and form connections rapidly.
  • The video frames it as potentially the first meaningful step toward a “brain-in-a-jar”-like revolution, but highlights unresolved issues:
    • Few physical contacts between chip and brain.
    • The system was taught a limited task (ping-pong).
    • Uncertainty about how to copy/transfer learned models.
    • Control and replication remain major problems.

• Closing / speculative note

  • The speaker implies that until these hardware-biological problems are solved, wide replacement or major leap remains uncertain.
  • Ends with a humorous placeholder about an “Aunt Galya” who inputs documents into a database (suggesting current reality still involves human-like labor).

---

Methodology / instruction-like content (explicit steps described)

A) How an LLM generates text (high-level inference loop)

1. Convert input text into a vector representation.
2. Feed the vector through the neural network parameters (many learned weights).
3. Multiply the network’s output by the dictionary/vocabulary matrix.
4. Select the token with the greatest value (highest score/probability).
5. Append the chosen token to the input sequence.
6. Repeat the cycle until the output is complete.

B) How words/tokens are represented (embedding lookup)

1. Split input text into tokens.
2. For each token, look up its dictionary ID.
3. Use that ID to retrieve a corresponding base vector from the embedding table.
4. The embedding table’s vectors are learned during training (initially random), positioning frequent/co-occurring words closer in embedding space.

C) Training process described

• Pre-training
  • Provide large text corpora.
  • Train the model to continue the text (next-token prediction).
• Further training
  • Provide prompts and target “correct” outputs.
  • Train the model to produce answers that match expected outputs.

D) The speaker’s “from scratch” LLM implementation (conceptual)

• Model size (as stated by the speaker):
  • 16-character context window
  • about 8,000 parameters (very small for demonstration)
• Architecture conceptually includes three layers:
  • Layer 1: processes token + position features
    • 96 inputs per 16-character context window (explained via embeddings dimensions).
  • Layer 2: compresses/abstracts
    • receives 48 inputs (from the prior representation).
  • Layer 3: produces outputs
    • 32 inputs → 43 output probabilities (for selecting next character/letter).
• Training/debugging method mentioned:
  • Logging shows how weights change gradually.
  • Then weights are written to files.
  • Learning and testing are done after “teaching using instructions.”
• Demonstrated limitation:
  • Works for the currently trained question, but breaks down when asked a different question (illustrating the small model’s limitations).

---

Sources / speakers featured (as mentioned)

• The video speaker (unnamed; “I’ll explain…”, “I can help develop something like this”, and demonstrates their own LLM code)
• Ilya Sutskeber (named as “one of the fathers of the industry”)
• Researchers from Dishbrain (credited with the processor + mouse brain hybrid example)
• Mentions of:
  • Brain (mouse brain) (as part of the Dishbrain example)
  • “GPT Image” (as a referenced system/model name)
  • Futurama’s brain in a jar (cultural reference, not a real source)
  • “Uncle Neuron” (presented as a named component/personification of a neural network element; not a real external person)