03.09 Algorütm: Pipedrive AI koodiülevaatuste lahendus

Product reviewed

Pipedrive “AI code review” solution — an agent-based system integrated into CI/CD and pull requests, designed to scale and improve PR review quality beyond human-only review.

Key features / how it works (from the video)

Trigger-based workflow

• When a developer pushes code or opens a PR, they add a /review comment.
• Background agents generate review feedback that looks like a person wrote line-level comments.

Multi-agent pipeline (not one big prompt)

• Early versions used a single long prompt, which produced poor results.
• The system was redesigned into multiple specialized agents plus orchestration steps, such as:
  • generation
  • previews/artifacts
  • verification/validation

Cloud execution (and optional local runs)

• Runs via Cloud execution.
• Can also run earlier locally for developers.
• Sharing works through a plugin/marketplace approach, so the same review experience can occur from local development or CI.

Context enrichment beyond code

The workflow can incorporate:

• Static checks + existing tooling outputs, such as:
  • unit tests
  • Sonar/static analysis
  • a static configuration check tool
• Architecture context:
  • Agents read architectural guidelines from the repo.
  • A separate architectural decision service/knowledge map helps understand dependencies across services (including monorepo/dependency awareness).
• PR description + Jira ticket text:
  • Flags missing or unclear intent.
  • Provides guidance like: “Please improve the descriptions.”

Service criticality gates

• For less critical services:
  • AI may grant approval and a human can apply it directly (lighter gate).
• For more critical services:
  • final human approval is still required before deployment.

Extensibility via plugins / custom instructions

• Teams can add their own review plugins (e.g., accessibility testing, React performance).
• Agent selection is controlled by conditions so the correct specialized agent runs (frontend/backend, etc.).

Actionable feedback loop (“/implement” style behavior)

• Beyond reviewing, the system can also implement recommendations (AI applies fixes for certain issues).
• This depends on the input/task description being sufficiently clear.

Security / sandboxing approach

• Agents run with restricted rights in containers / permission profiles.
• This helps prevent destructive actions (e.g., rm -rf).
• The video suggests strong auditability (“everything leaves a trace”), but they note limited direct pentesting.

User experience / developer workflow impacts

Earlier feedback

• The review is intended to run as close to the developer as possible (even locally) so issues can be fixed before a full PR.

Improved quality of review text over iterations

• They moved away from long unstructured output that nobody read.
• They reduced “sprawl” by:
  • limiting the number of comment blocks
  • structuring suggestions into fewer, more digestible groups
  • avoiding notification spam
• They tuned output style with better models and simple formatting rules to look “more like a normal review.”

Knowledge-sharing benefit

• Reviews are positioned as learning and documentation:
  • juniors learn from senior-level decisions
  • not just bug discovery

Pros (explicitly supported by the discussion)

• Scales with higher PR volume
  • Human-only review becomes unsustainable once AI coding tools increase PR counts (requests reportedly doubled).
• Improved throughput without collapsing quality
  • They report very good and improved defect injection rate and SL/SLA-related measures.
• Higher review responsiveness
  • Time to first review roughly doubled while recommendation intake increased, as the system manages the PR flood.
• More informative reviews
  • Detects issues at:
    • line level (suggested changes)
    • conceptual level (unclear ticket description)
    • cross-service dependency risk (via architecture decision context)
• Configurable automation by criticality
  • Faster for low-risk services while still requiring human approval for high-risk services.
• Team extensibility
  • Plugins enable module teams to tailor reviews (performance, accessibility, etc.).

Cons / limitations (explicitly mentioned or implied)

Cost pressure from model changes / added agents

• Costs increased as they added deeper/extra agents:
  • early: ~$1/review
  • optimized: ~$0.50
  • later: ~$1.50 after adding “boot agents”
• They estimate a monthly upper bound around €2k–€3k per developer per month (depending on vendor/model dynamics).
• Advanced models can be ~2x more expensive without 2x quality, so they stayed with an older model.

Model/vendor instability

• Workflow can break if the AI provider (e.g., Anthropic/OpenAI) has issues.
• They lack robust fallback logic to switch models when failures occur.
• Provider “prompt/system prompt changes” can also shift behavior (e.g., reviewer not submitting correctly).

Input quality dependency

• AI implementation quality depends heavily on Jira/PR description quality.
• Vague tickets lead to misunderstandings.

Security posture still needs improvement

• They haven’t done direct pentesting of the system itself.
• While rights restriction is emphasized, further hardening is an improvement area.

Not fully replaceable for complex matters

• Human review is still needed for more complex and critical matters to avoid knowledge loss.

Comparisons / alternatives mentioned

Build vs buy

• Recommendation: small teams (≤ ~20 people) generally shouldn’t build their own; use market solutions.
• Larger orgs may build internally due to context/cost constraints.

Local vs CI execution

• They argue value in running reviews close to developers (local).
• The system also runs centrally via CI/CD.

Model comparison

• Opus 4.6 is mentioned as a good cost/quality balance.
• A newer model increased cost (~2x) without proportional quality gains.
• They tested different harness/configs and found one about 20–30% cheaper for the same model with comparable quality.

Ratings / numerical scores mentioned

• Recommendation usefulness
  • About 70% of AI recommendations are reported as “working or good,” determined by whether developers incorporated them.
• Cost per review over time
  • ~$1.00 early iterations
  • ~$0.50 after optimizations
  • ~$1.50 after adding more thorough “boot agents”
• Cost upper bound estimate
  • ~€2k–€3k per developer per month (scenario estimate)
• Model/harness benchmark
  • ~20–30% cheaper with a different harness for the same model
  • newer model ~double cost

Unique points list (distinct claims/themes)

1. AI agents generate PR reviews triggered by /review.
2. Multi-agent pipeline replaced early single-prompt approach due to poor results.
3. Reviews include line-level comments that resemble human feedback.
4. AI review still works even when code is generated via automation/labels (reducing reliance on constant developer involvement).
5. Review coverage helps as handwritten code proportion decreases.
6. AI reduces comment spam by structuring feedback.
7. Style differences across models led to tuning for readable reviews.
8. Opus 4.6 chosen for cost-efficient performance.
9. Inputs include unit tests, Sonar/static analysis, and static configuration checks.
10. Workflow can wait for check results and integrate them.
11. Reviews check PR description + Jira text and flag missing intent.
12. Architecture context comes from repo guidelines + an architectural decision knowledge map/service.
13. Cross-service dependency risk can be surfaced using dependency graph knowledge.
14. Reviews can run locally; sharing via a plugin marketplace + CI integration.
15. They track whether humans actually apply AI recommendations (plus metrics like defect injection).
16. Defect injection rate and related SL/SLA measures are reported as “very good” and improved.
17. Quality measured historically via a backfill-style metric (~70% recommendations followed/valid).
18. Workflow stability issues exist due to provider instability; limited fallback switching.
19. Security via constrained container rights and permission profiles.
20. Limited direct pentesting; encourages internal hackathon testing.
21. Plugins let module teams run specialized agents (accessibility, React performance, etc.).
22. Agent selection is dynamic using conditions (e.g., frontend/react vs other).
23. Agents can implement recommendations, but outcomes depend on ticket/task input quality.
24. Future direction: review “analysis/tasks/input” before coding (a fuller agent chain).
25. Context management is presented as core technique: avoid overly long/complex contexts.
26. Build-vs-buy guidance: buy market tools unless you can invest in iteration + monitoring.
27. Leadership guidance: keep human review for complex/critical matters to preserve learning/knowledge sharing.

Speaker-specific views/aspects

• Priit Liivak (host):
  • Frames scaling code review, asks about workflow mechanics, cost, metrics, and roadmap.
• Aleksander Smirnov (Tehisaru / AI platform engineer):
  • Explains agent architecture, orchestration, model/harness choices (Opus 4.6), style tuning, plugin/marketplace mechanics, and context management.
  • Covers security constraints and sandboxing.
• Martin Aun (Pipedrive head of Developer Productivity):
  • Explains business drivers: PR volume growth, sustainability, criticality gating (AI approval vs human final approval).
  • Presents metrics/outcomes (70% usefulness, defect injection improvements) and cost history.
  • Emphasizes human review for learning/knowledge sharing and “build vs buy.”

Concise verdict / recommendation

Recommendation: Worth adopting if your org has high PR volume and you can operationalize the system (agent orchestration, context enrichment with tests + architecture + strong ticket quality, and ongoing monitoring). The approach appears to improve scalability and defect outcomes per their metrics, but it’s sensitive to cost and provider instability, so it’s best suited to teams that can run it operationally—or use a proven third-party solution if you’re smaller.