How Intelligence Measures the Frontier

Summary

Intelligence, led by CEO Grace Li, builds tools to evaluate AI models in subjective creative domains. Its first product, Design Arena, lets people create websites, images, and videos with models. The company turns users’ prompts, conversations, edits, and choices into feedback that helps AI labs identify where their models fall short.

The core business insight is that creative quality needs a different kind of evaluation from math or other domains with objectively verifiable answers. Rather than relying on a single expert’s judgment, Intelligence gathers preference signals from many users in real product use.

Business strategy and product

• Turn evaluation into part of the product experience. Users can compare multiple generated options as part of creating something, much like a designer presenting several layouts to a client. This both helps users clarify what they want and generates preference data.
• Collect behavioral signals, not just survey answers. The platform uses what users ask for, what they edit, what they choose, and whether they continue using an output. The goal is to avoid interrupting the experience with repeated satisfaction questions.
• Use population-level data to diagnose subjective failures. A single person’s preference may not represent a target audience. Large-scale signals can reveal whether a model is failing for a particular group or use case.
• Serve model developers as a feedback loop. Intelligence interprets user actions much like product managers interpret product behavior: it turns raw interactions into guidance for development teams about what to improve.
• Build toward a marketplace of specialized models. The company sees a future where users can be matched with different models based on budget, speed, preferences, and task—not simply defaulted to one general-purpose model. This depends on a diverse supply of models, including open-source models, smaller providers, and leading labs.

Frameworks and operating playbooks

• “North star” for unverified domains: Find a measurable signal for what “better” means when there is no single correct answer. The company’s initial gaming work suggested signals such as player numbers and retention.
• Product-embedded measurement: Integrate evaluation into the task users already want to complete. Offer meaningful alternatives and observe choices rather than asking users to evaluate outputs in isolation.
• Stated versus revealed preferences: Distinguish what users say they want from what their behavior indicates they want. Both can change over time, so model improvement should account for preference drift.
• Audience-specific evaluation: Define “best” relative to the intended audience and task. The company noted that preferences—and therefore useful evaluation data—can differ across demographics and regions.
• Small-team decision-making: Maintain a shared approach to problem-solving so employees can challenge one another’s ideas rather than disengaging when they disagree. Trust and common standards are intended to reduce coordination overhead.
• Learn by doing: Enter emerging markets before all the infrastructure is ready, run experiments, and update beliefs based on what happens. Li’s shorthand: an “idiot in motion” can outperform a “genius sitting still.”
• Hiring for learning velocity: Prioritize people who can rapidly absorb expertise and learn unfamiliar areas, particularly when technology and job requirements change quickly.

Metrics and figures mentioned

• More than 7 million users across 192 countries.
• The interviewer cited the company reaching about $60 million in annual revenue/run rate in roughly a year; the figure was discussed as a rapid scale milestone.
• Li said that, in the company’s early period, three model launches per month felt productive; by the time of the interview, three launches in a day had become normal.
• The company uses signals such as user choices, edits, engagement, and retention to assess model performance. No specific benchmark scores or customer revenue figures were provided.
• Li described small teams as enabling high trust and said that, when people share a common decision-making basis, they may not need to revisit the reasoning behind a colleague’s choices about 80% of the time.
• She said the company’s early team was largely drawn from familiar circles, with roughly 80% of the team being Harvard graduates, and suggested that team-building dynamics change around the 25-person threshold.
• The company sees preference trends across a three-to-six-month horizon and observed that stated preferences in the first quarter can differ substantially from preferences in the third or fourth quarter.

Examples and lessons

• Origin in AI-generated games: About a year and a half before the interview, the team found that models could generate functional games but often produced games that were poorly designed, lacked compelling stories, or were not fun. That gap led them to ask how subjective quality could be measured.
• Gaming as an early signal: The team considered player activity and retention as possible indicators of whether a game was good, then extended the idea into a platform where users create and compare creative outputs.
• TikTok analogy: Li described time spent watching as a behavioral signal that can help predict whether a video holds an individual user’s attention. The broader point was that large-scale audience data can evaluate subjective outcomes without relying on one person’s taste.
• Different users need different models: A lightweight, inexpensive model may be sufficient for a simple task, while another use case may require a more capable model. The opportunity, in the company’s view, is to route users to the appropriate specialized intelligence.
• Act before a market is fully formed: Li argued that the timing of a marketplace depends on the availability of many models. Starting too early could mean too few providers; starting too late could mean the market already has efficient ways to choose among them.
• Treat rejection as less objective than it feels: Li advised founders to expect rejection to feel more severe to the person receiving it than it may appear to others. Her example was a prospective recruit who initially declined but later asked why the founders had not tried harder to persuade him.
• Scale requires new systems: Moving from an early product to a much larger business requires more than repeating what worked at small scale. Li highlighted hiring, internal systems, and finding a growth mode that can turn early successes into sustained expansion.

Actionable takeaways

1. For subjective products, identify behavioral signals that indicate value instead of waiting for a perfect objective score.
2. Make evaluation useful to the user—for example, by presenting choices that improve the creative workflow.
3. Segment feedback by audience and task; aggregate data does not mean every user wants the same thing.
4. Track how preferences change over time and use those shifts to guide product and model development.
5. Keep early teams small enough to build trust and resolve disagreements directly, while deliberately establishing shared norms for communication and truth-seeking.
6. Hire for the ability to learn quickly, not only for expertise in today’s tools.
7. In a fast-changing market, run experiments early; experience from execution can be more valuable than extended speculation.
8. When growth accelerates, invest in the people and internal systems needed to scale—not just in the product.

Presenters and sources: Grace Li, CEO of Intelligence; Ankit Gupta, YC. Source: YC Root Access interview, How Intelligence Measures the Frontier.