GPT-6 Sol & Luna, Opus 5.5, MiMo-V2.6 Pro & Flash, Grok 4.7 | Live Demo + Q&A (Sep 24th)

Overall context / platform

• The speaker, Peter (Roboflow), runs a VLM evaluation website called Vision Evals.
• The site measures and benchmarks multimodal (VLM) models across tasks including:
  • Object detection
  • Counting
  • OCR / data extraction
  • Image reasoning
  • (In the demo) Segmentation
• The stream is a live model comparison of four/five major releases from major labs using:
  • Leaderboard metrics (e.g., mAP for object detection)
  • Interactive “sandbox” re-runs on specific test images

Releases discussed (this week)

1. Grok 4.7 (xAI)
2. Xiaomi: MiMo V2.6 Pro and MiMo V2.6 Flash
3. Anthropic: Claude Opus 5.5
4. OpenAI: GPT-6 Sol and GPT-6 Luna

Grok 4.7 (xAI) — improvements but still behind

Reported gains vs prior Grok

• Better object detection / vision quality
• About +20 mAP versus the previous Grok version (on their evaluation)

Still lags the market

• Grok 4.7 scored about 41.2 mAP
• It is described as far behind market leaders

Example behavior differences

• Fewer failures than before:
  • Reduced cases where Grok became distracted and produced bad annotations
• Ongoing weaknesses:
  • Misses certain object types (example mentioned: “spikes”/tires, including winter tire cases)
  • Other detection mistakes still occur

Performance / latency

• Very slow: shown taking about 84 seconds to complete prediction on a test image
• In leaderboard sorting, it’s described as the slowest model on average, notably slower than mentioned quant baselines

GPT-6 Luna vs previous Luna — cheaper but worse overall

Key finding

• GPT-6 Luna is worse than Luna 5.6 in almost everything measured
• The only stated advantage:
  • Better high-level object detection (coarser detection)
• At low-level detection, it performs the worst among comparable tasks

Quantified deltas mentioned

• Around 6% in one calculation set
• Roughly ~5% (high level) and ~8% (low level) for reasoning/detection comparisons (Subtitles were imprecise, but the trend is consistent: Luna 6 regresses on detailed tasks.)

Cost advantage

• Luna 6 is much cheaper: roughly 2.5–3× cheaper (high/low ends cited)

Latency

• Mixed—sometimes faster or slower depending on task
• But cost improvement is consistent

Recommendation for auto-annotation

• The speaker advises not to use the new Luna for computer vision auto-annotation, because quality drops despite lower cost.

GPT-6 Sol and Claude Opus 5.5 — best quality/price improvements this week

Claude Opus 5.5 (Anthropic)

Quality jump

• Framed as a major quality improvement
• About +10 percentage points in almost every comparison versus older Opus
• In object detection comparisons:
  • Better than other models from Anthropic’s prior lineup
  • Nearly best among the new releases
  • Still behind GPT-6 Astra in overall quality

Cost

• Still described as expensive overall
• But the improvement is enough to reduce cost versus older baselines

GPT-6 Sol (OpenAI)

Improved vs previous Sol

• “New Sol is better than old Sol.”

Stability / fewer false detections

• Older Sol reportedly hallucinated detections (e.g., detecting “windows” where none exist)
• New Sol is described as correcting those behaviors

Pricing

• Sol is stated to be about ~4–5× cheaper than Astra, while still competitive

Relative ranking summary (speaker’s view)

• Top overall performer for the week’s releases: Opus 5.5
• New Sol: strong, but slightly behind Opus for object detection
• GPT-6 Astra: still described as best overall in their leaderboard comparisons

Pricing/latency comparisons (arena mode and playground)

• The speaker emphasizes cost per image/prediction, not only quality.
• Examples:
  • Opus predicted at about “a cent and a half per image” on average
  • Another arena example: Opus priced around ~3 cents
  • Gemini 3.5 Flash was almost as expensive due to longer runtime
  • Some cheaper models had worse quality, e.g., confusing an object concept like “tour” vs “elephant”
• Key takeaway:
  • Models can look cheap, but poor quality may remove the benefit (e.g., needing retries or post-processing).

Guidance: what to use for auto-annotation (detection + segmentation)

Object detection auto-annotation recommendations

• New Luna: worse than old Luna → no reason to use
• New Grok: better than old Grok, but still significantly worse than others → no reason to use
• Main recommended choices for detection auto-labeling:
  • New Sol
  • New Opus
• Close competition mentioned:
  • Astra
  • “Quen/Kwench 3.8 max” is referenced in the leaderboard context

Data extraction / OCR task findings

• Leaderboard trend: Gemini leads for data extraction
• Opus strengths: strong extraction of specific fields, including difficult/distorted numbers/text
• Sol performance: described as significantly worse than Opus and Gemini on extraction examples

Segmentation: key implementation tradeoffs (SAM vs VLMs)

Core evaluation insight

• SAM-style segmentation models
  • Produce high-quality masks
  • But do poorly with language understanding
• VLMs
  • Understand language/text
  • But may output segmentation differently and can be slower

Demonstration notes

• SAM creates masks with correct shapes, but can mislabel based on language cues:
  • Example: labeling everything “black” because it fixates on color words
• GPT-6 Luna mask quality described as unsatisfactory
• Sol improves
• Astra performs better but still misses some labels

Runtime and output format

• VLM segmentation outputs polygons (JSON with x/y points) that must be converted to masks, making it slower
• Timings mentioned on a particular image:
  • Astra: ~140s
  • Sol: ~112s segmentation and ~53s detection

Recommended hybrid workflow (best auto-annotation)

1. Use Astra for object detection (language-aware) to get frames/bounding boxes
2. Then query SAM-Free for accurate mask generation
3. This hybrid approach is presented as the optimal segmentation auto-annotation strategy

Other notes / mentions

• Keypoint/pose estimation: Astra can do “pretty good” keypoints, but still far from human-level markup (pose, hand gestures, face mesh mentioned).
• Comparison of SAM vs VLM segmentation and adjustable “effort level” in the playground (requires login).
• Mentions:
  • DeepSeek (speaker suggests current CV capabilities may be low; offers to evaluate later)
  • “Jeff Omni” on Hugging Face (open-source image-capable decision/orchestration model, positioned more for real-time decisions than detection/segmentation)

Main speakers / sources

• Peter (Roboflow / open source) — primary speaker and evaluator
• Model families/sources referenced:
  • xAI: Grok
  • Xiaomi: MiMo V2.6 Pro / Flash
  • Anthropic: Claude Opus 5.5
  • OpenAI: GPT-6 Sol, GPT-6 Luna, GPT-6 Astra
  • Also referenced: Gemini, SAM-Free, and “Quen/Kwench 3.8 max” in leaderboard comparisons