Agents on Rails: Building AI Pipelines in plain Ruby - Daniela Velasquez

Summary

Daniela Velasquez explains how to build proactive AI agents in regular Ruby on Rails, using an athlete-coaching app as an example. Rather than waiting for users to consult a dashboard or chatbot, an agent can analyze workout, sleep, and heart-rate data and send personalized suggestions.

Building the research agent

The pipeline starts with a Rails task that gathers the athlete’s data and sends it to an LLM with a prompt defining its role. For example, the agent might act as a running coach asked to recommend changes to next week’s training and justify them using the analyzed data.

The agent can also receive tools for querying workout history, sleep logs, weather forecasts, and web search. Tool use is optional, allowing the model to decide what is relevant.

Because research tasks can take 10–30 minutes, deployments or task restarts can interrupt them. One proposed solution is to persist the conversation’s messages and send the full history with each LLM request. The task can then resume with its prior context instead of starting over.

Evaluating suggestions with “judges”

Generated advice can be inaccurate, impractical, or inappropriate. Velasquez describes using additional LLMs as judges to assess suggestions against application-specific criteria, such as:

• Whether the advice is supportive and respectful
• Whether it is factually grounded
• Whether the athlete can actually follow it

These judges handle simpler classification tasks, so smaller, faster, less expensive models may be sufficient. Their prompts can be tested against labeled examples—for instance, 200 examples each of kind and unkind coaching—to compare prompt and model accuracy. Suggestions that fail relevant checks are withheld.

A Shopify research example illustrates why review matters: an agent repeatedly suggested bundling nonexistent shampoo and conditioner products because it had mistaken an example for a real recommendation.

Learning from user feedback

Athletes can provide feedback that adds information the app may not know, such as being unavailable on Sunday mornings. Velasquez suggests storing feedback as an application record linked to the athlete and the recommendation.

A separate, lightweight LLM task can turn raw feedback into a concise factual note, filtering out frustration or insults before adding the information to future research prompts.

A remaining challenge is keeping stored facts current, since schedules and circumstances can change. The presentation calls this ongoing review of retained information a “dreaming” phase.

Overall pipeline

1. A researcher agent analyzes user data and proposes personalized actions.
2. Judge agents evaluate each proposal against relevant criteria.
3. Only approved suggestions are shown to the user.
4. User feedback is processed into concise facts and fed into later runs.

Main speaker/source: Daniela Velasquez, presenting “Agents on Rails” at Rails World. She also recounts a Shopify example from her experience with AI-generated recommendations.