Video summary

I Tested Apple’s Secret macOS AI… The Results Made No Sense

Main summary

Key takeaways

Technology

Summary of technological concepts, features, and results

  • Hidden macOS AI (FM) in macOS “Golden Gate”

    • A macOS 27 beta reportedly includes a built-in CLI tool called “FM” that can run LLM inference locally from the terminal.
    • Claims: no need for external apps like Ollama, LM Studio, or MLX (at least for using FM).
    • FM is described as a local/private AI endpoint inside the OS that can be scripted and used to build other tooling.

  • Two execution modes

    • FM supports both:
      • Local on-device inference (runs on the Mac’s hardware)
      • A cloud mode via Apple’s Private Cloud Compute (faster, ~3× in the tests)
    • The creator focuses primarily on local performance and tests both modes.

  • Benchmarking difficulties

    • Standard “OpenAI-compatible” benchmarking reportedly fails because Apple’s endpoint doesn’t fully match expected timing/format conventions.
    • Attempted benchmark: LlamaBench (built on OpenAI-compatible endpoint assumptions).
    • Apple-supplied token-rate numbers appeared implausible, so the creator built a custom benchmark.

  • Creator-built benchmark: “Apple FM Bench”

    • The creator built a custom tool to measure token throughput more accurately.
    • On-device results (Mac Mini, M-series, macOS 27 beta):
      • ~50–52 tokens/s for prompt processing (prefill)
      • ~50–56 tokens/s for decode/generation (values vary slightly by chart/readout)
    • Key finding: adding more expensive hardware (Mac Studio with a higher-end chip) produced nearly identical local token speeds.

  • Hardware scaling mystery (Mac Mini vs Mac Studio)

    • Expected outcome: faster chips + higher memory bandwidth (especially affecting decode) should increase speed.
    • Observed outcome: prompt and decode speeds are roughly the same between:
      • Mac Mini (M4 Pro-class)
      • Mac Studio (M3 Ultra)
    • Hypothesis: both systems are bottlenecked by similar silicon behavior for this workflow.

  • Determining which engine runs which LLM stage (Neural Engine vs GPU)

    • The creator initially suspects the Neural Engine / NPU (A&E), but common monitoring tools seem unreliable:
      • Tools like macTop/AzyTop show Neural Engine at 0% while FM runs.
      • powermetrics suggests 0W from CPU/A&E during workloads (also inconsistent).
    • New “fight it out” method:
      • While FM runs, the creator stresses the Neural Engine and observes the FM slowdown pattern.
      • Then they stress the GPU and observe the slowdown again.
    • Conclusions from controlled stress tests:
      • Decode (token generation) slows when the Neural Engine is stressed → decode runs on the Neural Engine.
      • Prompt processing (prefill) slows when the GPU is stressed → prompt processing runs on the GPU.
    • Interpretation:
      • Apple’s FM likely uses a pipeline where Neural Engine decode is the limiting factor.
      • That limiting block appears similar across many M-series chips, so higher-end Mac Studio hardware doesn’t translate into higher throughput for this workload.

  • Practical takeaway

    • For users running Apple’s built-in FM locally, a Mac Mini can match Mac Studio-class performance for FM token throughput.
    • Suggested alternative: spend saved money on hardware/models that actually scale better, such as workflows using Llama.cpp and MLX.
      • MLX is discussed as using the GPU heavily, which may scale differently.

  • Availability / constraints / caveats

    • FM is beta and may change after release (including possible future GPU involvement).
    • Works only on Apple Silicon; not for Intel Macs (Golden Gate not available for Intel).
    • The creator provides:
      • A free GitHub benchmark/tool (“Apple FM Bench”) so viewers can test their own hardware.

Key points about reviews/guides/tutorials

  • Guide-like content: how to benchmark FM locally using a custom tool, and why standard OpenAI-compatible benchmarks may fail.
  • Benchmark release: “Apple FM Bench” is presented as available for free on GitHub.
  • DIY verification method: a “stress test” approach to identify whether FM’s decode uses the Neural Engine and whether prompt processing uses the GPU—without trusting system meters.

Main speakers/sources (from the subtitles)

  • Primary speaker: the video author/reviewer (first-person tester who builds “Apple FM Bench” and runs experiments).
  • Referenced external systems/tools:
    • Nori team / Super Nori (mentioned as a detour)
    • LlamaBench (initial benchmark tool used)
    • macTop / AzyTop (monitoring tools)
    • powermetrics (Apple CLI utility used under the hood)
    • CoreML / Claude Code (used to help build a CoreML-based neural-engine workload for testing)

Original video