How I Interview AI Engineers (And the Projects That'd Impress Me)

Summary

The video explains what applied-AI and agent-focused startups may look for in interviews, then recommends projects that help candidates demonstrate those skills. The central message is that knowing how to call an LLM is not enough: engineers should understand how to build, scale, evaluate, and monitor complete agent systems.

Interview Topics for Agent Engineering

Using a coding agent that works on GitHub issues as an example, the presenter outlines five areas interviewers may explore:

1. Architecture and sandbox scaling

   • Identify the services and infrastructure needed, including front ends, back ends, and isolated sandboxes where agents can safely run code.
   • Explain how to launch and shut down sandboxes as demand changes.
   • Consider how the system would handle many users running agents concurrently.

2. Resilience and task recovery

   • Long-running agent tasks may last much longer than ordinary web requests, so consider what happens if a worker, server, or sandbox fails.
   • Persist the agent–LLM message history and enough task state to resume work from a checkpoint.
   • Design the system so another worker or infrastructure component can take over.
   • The presenter recommends learning about Temporal as an example of technology for durable, resilient workflows.

3. Context management

   • Short interactions may work well without much intervention, but complex tasks can produce very long conversations.
   • Use techniques such as summarizing or compressing context to keep the agent effective and control costs.
   • Preserve important information while reducing the amount of context sent to the model.

4. Evaluations and quality measurement

   • Define objective tests to determine whether an agent or a new model version performs better—not just whether it seems better.
   • Build evaluations around real tasks and data, with deterministic ways to check results where possible.
   • Benchmarks such as SWE-bench and Terminal-Bench can provide examples.
   • The presenter expects custom evaluations and reinforcement-learning environments to become increasingly important.

5. Observability and operational metrics

   • Track whether agents complete tasks, where they fail, whether users are satisfied, and what caused errors.
   • Monitor the agent’s execution and interactions, not just conventional application health.
   • Build alerts and recovery mechanisms for failures such as unavailable infrastructure or compute providers.

Suggested Projects

The presenter recommends building these projects from scratch or studying their open-source implementations:

1. A terminal coding agent

   • Building a basic terminal agent is relatively approachable; making it perform well is the harder challenge.
   • Compare its results against established coding agents using a benchmark such as Terminal-Bench.
   • Analyze failures and iterate on the agent’s tools and behavior. The presenter mentions subagent orchestration as one capability that may improve performance.
   • Avoid treating benchmark-specific hardcoding as a complete solution. Use tests to understand how to improve the agent more generally.

2. An autonomous, multi-platform assistant

   • Build an agent with memory and integrations for services such as WhatsApp, Telegram, or Slack.
   • Give it the ability to take actions over time, rather than limiting it to one-off responses.
   • Study existing open-source projects to understand their architecture.

3. An AI-native Slack-style workspace

   • Create a shared workspace where agents participate alongside people.
   • Users could tag agents with different roles, such as coding or writing, and have them act within a conversation.
   • This explores a more collaborative interface for working with agents than using a terminal alone.

4. A multi-agent coding workspace

   • Build an interface for running different coding agents on different tasks from one place.
   • The presenter contrasts two approaches: launching command-line agents in separate tabs, or connecting an application interface directly to an agent.
   • Depending on the architecture, the project can teach either full-stack application development or more about agent interactions.

5. A generative learning interface

   • Create a tool that turns a topic into a learning plan, course material, slides, and interactive discussion with an agent.
   • The presenter sees generative interfaces as technically interesting, while expressing uncertainty about whether this particular learning experience is compelling as a product.
   • A related project idea is a classroom tool where an administrator creates lessons and quizzes for a group.

6. A smart model router

   • Build a system that directs requests to different models based on task complexity, cost, user permissions, or available credits.
   • Such a system might block inappropriate uses of company-provided access as well as select a suitable model.
   • The presenter describes this as an open-ended problem without an obvious established solution.

7. A benchmark and evaluation framework

   • Choose an open-source repository and create tasks, test environments, and evaluations that measure how well different models or agents solve its problems.
   • Expect this to be difficult: setting up the codebase, designing representative tasks, and verifying results can all take substantial effort.
   • The goal is to create assessments that genuinely reflect agent capability and help teams improve agents over time.

Broader Lessons

• Agent engineering combines software infrastructure, AI, and product design.
• Long-running tasks require durable state, recovery strategies, and suitable execution environments.
• Context quality, evaluations, and observability are core engineering concerns—not optional extras.
• A strong portfolio project should go beyond a working demo: test it, measure its performance, and explain how it could scale and recover from failure.
• Open-source projects can be useful references for understanding real-world architectures.

Speakers and Sources Featured

Speaker

• Harkirat Singh — the presenter and narrator.

People, organizations, products, and technical references

• Super30
• An unnamed programming-agent company described as having recently raised $15 million
• Devin
• Firecracker, AWS Lambda, and E2B
• OpenAI
• Temporal
• Manus (rendered as “Manis” in the subtitles)
• SWE-bench and Terminal-Bench
• New Relic, Datadog, Prometheus, Grafana, and Neatlogs
• Pi, Claude Code, Codex, Claude, and Opus
• Hermes, Cloudbot, Buzz, PromptQL, Slack, WhatsApp, and Telegram
• Jack Dorsey and Block
• Superset and T3 Code; Theo
• Lovable
• OpenRouter
• Uber
• dub.sh