Clang internals (in Russian)

Summary

This lecture (in Russian) explains Clang/LLVM internals, focusing on how Clang is organized as a pipeline—a driver + frontend + backend—and how to inspect and extend Clang using actions, plugins, AST visitors, and tooling interfaces.

---

What Clang is (architecture overview)

• Clang has both “driver” and “frontend” roles:

  • Running clang as a driver orchestrates compilation steps by invoking tools such as:
    • preprocessor + parser + code generation stages,
    • assembler,
    • linker.
  • Running the frontend directly (the lecture references a “mode/option” distinction) causes the frontend to run independently, and it may call itself as a dependent process.

• The typical flow is:

  • Clang driver → frontend (cc1 / preprocessing+parsing+codegen) → assembler → linker

---

Driver vs frontend options and “extras”

The speaker discusses how options are passed:

• Driver options are broadcast/forwarded to the frontend.
• Some options can be forwarded directly to the frontend without translation.

A key practical point:

• Clang options can reveal internal behavior.
• The lecture encourages inspecting what the driver actually runs (the speaker mentions a “toilet brush” option—i.e., an auto-generated stand-in for “show commands / explain”-style flags).

---

Clang’s internal “pipeline”: compilation actions

A core conceptual model:

• Clang is organized around a Compilation object created by the driver logic.
• Inside Clang, compilation is expressed as many “actions” connected into pipelines, depending on:
  • file types (extension/type),
  • selected modes,
  • heterogeneous compilation scenarios.

Overall idea:

• Build actions based on the input file kinds, then execute them to produce outputs.

---

Backend and linkage stages

The lecture distinguishes multiple tools acting like pipeline stages:

• The linker stage takes multiple inputs and produces a combined result.
• For non-trivial compilations, many internal actions occur.
• Clang can also mix file formats in one invocation, for example when converting between object/assembly/bitcode-like stages.

---

Frontend actions: FrontendAction hierarchy and constraints

Clang’s frontend is implemented via a FrontendAction class hierarchy:

• There are derived actions for behaviors such as:
  • preprocess-only,
  • parse-only,
  • code generation,
  • AST dumps,
  • HTML output printing, etc.

Important limitation:

• You usually can’t combine some frontend behaviors in the same run (e.g., compile + dump) because each run selects a single action type.
• Combining behaviors often requires a different design, frequently via plugins/extensions.

---

Producer/consumer model: parsing components

The AST construction framework is described as a producer/consumer pipeline:

• An AST producer parses and generates AST nodes.
• Consumers (registered during parsing) receive the AST (or parts of it).

This is where plugins hook in:

• register consumers/visitors to operate on the AST.

---

Lexer/parser/preprocessor interaction and lookahead/rollback

Clang uses:

• a preprocessor (macro expansion),
• a lexer producing tokens,
• a recursive-descent parser for C/C++-like grammars.

Challenges highlighted:

• The C grammar is context-dependent (e.g., distinguishing identifiers vs types).
• The parser may need rollback/backtracking to disambiguate constructs.

Optimization concept:

• Injecting inferred token information into the token stream (described conceptually as “Tokina/matting”), so the parser avoids repeating expensive inference.

---

Syntactic analysis and AST + semantics

The lecture defines:

• Syntactic analysis: building structure according to grammar productions, producing an AST.
• AST nodes correspond to constructs such as:
  • statements, declarations, types, and expressions.

Then:

• C++-style semantic analysis determines meaning that depends on context, including:
  • overloaded operators,
  • type-based interpretation.

Error recovery:

• Clang builds diagnostics and may insert recovery nodes instead of aborting.
• It supports “fix-it hints” (e.g., suggestions to add missing punctuation like . or ,).

---

AST structure: class hierarchies and how Clang organizes nodes

The AST is not just a simple tree:

• It is described as a directed graph / hierarchy using inheritance:
  • statement hierarchy,
  • expression hierarchy,
  • type hierarchy,
  • shared base classes enabling common behavior reuse.

The speaker’s claim:

• This organization enables flexible traversal and operations.

---

Visitors and traversals (RecursionVisitor)

The lecture explains the Visitor pattern for AST traversal:

• Parameterized visitor classes,
• Dispatch via visit methods,
• Macro-based generation of traversal logic.

It introduces traversal styles such as:

• visitors that can stop/recurse selectively,
• an “unlimited visitor / recursion visitor” approach where you rely less on manual recursion.

---

Instrumentation via tooling & scripting interfaces

Motivation: use Clang tooling as an ecosystem:

• Instrumentation with stable internal APIs.
• Build analysis tools without writing a full parser from scratch.

Recommendations include:

• Using Clang’s tooling libraries, and optionally wrapper/scripting logic (e.g., Python).

Outcome idea:

• Once you can consume the AST, you can build:
  • linters/static checks,
  • refactoring tools,
  • dictionary-based or rule-based analyzers.

---

Example tool: dictionary-based naming check (clang-checker style)

A conceptual tutorial example:

• Enforce naming conventions using Clang tooling:
  • build a dictionary of allowed variable names,
  • run an AST match to find declarations,
  • warn when identifiers don’t match dictionary words,
  • demonstrate compile-time warnings for forbidden names.
• Extend the approach to:
  • type names,
  • function names.

Implementation “shape”:

• AST matchers to mark/select relevant nodes,
• consumers that run after AST construction.

---

AST Matchers and how to match nodes

The lecture emphasizes AST Matchers as a practical way to specify patterns:

• “all declarations of a certain kind”
• “match expressions with operator X”
• refine matches using constraints (narrow/widen patterns)

Recommended workflow:

• prototype patterns using a matcher-testing tool (e.g., clang-query),
• then embed the matcher logic into your own checker/tool plugin.

---

Plugins: how to extend Clang

Plugin framework overview:

• Register factories for actions/consumers.
• Plugins can be inserted relative to the main action:
  • before/after/instead.

A described approach:

• provide a standard frontend action factory (parameterized by dictionary/config),
• create a custom action/consumer that performs AST matching or traversal.

---

“Clang tools” and stability of APIs

Closing recommendation:

• Prefer Clang’s existing traversal/AST/diagnostics/tooling rather than reimplementing parsing.
• Use stable interfaces where possible.

---

Reviews / recommended resources (explicitly listed)

The speaker recommends the following videos/resources:

• Official Clang documentation, especially:
  • plugin-related docs,
  • general LLVM/Clang docs.
• “Let’s get more of this video …” — an older report about clang (around 2013, per the speaker).
• A report by a “charismatic young teacher” on building Clang plugins (speaker mentions a GitHub with many plugins, possibly ~15 plugins).
• “Building … reflection system … (C++/clang plugin)” — uses Clang to inspect/operate on reflection-like metadata/structures (speaker references an author like “Gershman”).
• Another overview recorded in 2019:
  • “Great overview of Clang” (detailed but somewhat dry), suggested for architecture comprehension.

Final emphasis:

• Clang tooling + AST matchers + fix-it hints + plugins form a practical learning path.

---

Main speakers / sources

• Primary speaker: the lecturer (name not provided in the subtitles). The summary references other people (e.g., “Peter” and “Yuri”) in recommended materials, but the main teaching voice is the unnamed lecturer.
• Sources mentioned (recommended talks/projects):
  • “Boru / Boru is a fantastically charismatic young man” (the lecturer references him as a presenter in a recommended video),
  • “Gershman” (mentioned as an author of the reflection-system report),
  • “Peter” (mentioned regarding reports/videos and GitHub links).