Video summary

Abe Davis: New video technology that reveals an object's hidden properties

Main summary

Key takeaways

Technology

Core idea: “Motion microscopy” makes invisible motion visible

The video explains that many important motions—such as body pulse and breathing—are too subtle for the human eye.

It references MIT-developed motion microscope software that:

  • Detects tiny motions in video
  • Amplifies them so they become visible

Demonstrations

  • Pulse detection from wrist video The amplified pulse can be used to estimate heart rate.

  • Breathing monitoring from video of a sleeping infant The system estimates breathing without contact.

Extending “touch sensing” into “hearing” using video (visual microphone)

The speaker proposes treating sound as another form of motion:

  1. Record an object with a high-speed camera while sound makes it vibrate.
  2. Use algorithms to extract tiny vibration patterns from video that otherwise looks still.
  3. Convert the recovered vibration information into audio—turning objects into visual microphones.

Experimental setup (plant + loudspeaker)

  • Leaves move only about a micrometer (extremely small; roughly a fraction of a pixel), which is perceptually invisible.
  • Despite that, with correct processing, the sound source can still be reconstructed from silent video.

Why it works (key analysis)

  • Even tiny pixel shifts become meaningful when:
    • they are aggregated across many pixels, and
    • the vibration process is modeled correctly by the algorithm.

Major practical factors affecting performance

  • Object distance
  • Camera/lens
  • Lighting
  • Sound volume
  • Misconfiguration leads to noise rather than intelligible reconstruction.

Progression from “ridiculous but proof-of-concept” to usable systems

Early proof-of-concept

  • A potted chips bag filmed up close with very bright lamps while “Mary Had a Little Lamb” played.
  • The setup was intentionally crude—even melted bags—but it produced the first major milestone:
    • Recovering intelligible human speech/music from silent video.

Pushing realism with better conditions

  • The camera moved about 15 feet away, outdoors, behind a soundproof window, using natural sunlight.
  • Even then, “Mary Had a Little Lamb” was recovered from silent footage.

Quiet-source capability

  • Filmed earphones attached to a laptop.
  • Recovered laptop music from silent video—enough that the speaker claims to have Shazamed the results.

Using regular consumer cameras (rolling shutter method)

Beyond high-speed cameras (about 100× faster than typical phones), the work also uses rolling shutter artifacts:

  • Rolling shutter captures image rows sequentially.
  • When the object moves during a frame, you get time-delayed row artifacts.
  • A modified algorithm analyzes these artifacts to recover sound.

Demonstration

  • A bag of candy was filmed using a regular store-bought camera while music played.
  • The recovered audio was distorted, but still recognizable.

Surveillance concern, reframed as a new “lens” on objects

The speaker acknowledges people may associate this with surveillance.

  • Key point: surveillance already exists (e.g., laser eavesdropping).
  • What’s new: this method offers a way to visualize vibrations, providing a new basis for understanding:
    • forces (like sound causing vibrations), and
    • potentially the objects themselves (material/structural behavior).

“Interactive imaging”: using vibrations to build physics-like object simulations

New work (first public showing) shifts from sound reconstruction to interaction and physical inference.

Process

  1. Record a normal-looking video (even with a regular camera/cell phone).
  2. Induce vibrations by banging the surface where the object rests (example: a wire human-shaped figure).
  3. Use the vibration information to infer structural/material properties.

Output

  • The result is not a traditional video/image.
  • It becomes an interactive representation where a mouse can apply forces.
  • The system simulates how the real object would react to new, unseen forces.

Examples and scalability

Even longer videos can work without deliberate shaking:

  • Minute-long video of a bush in a breeze → simulation from subtle natural vibrations.
  • Two-minute video of a hanging curtain → natural air currents provided enough motion.

Key takeaways / “what this enables”

  • Non-invasive monitoring of physiological motion (pulse, breathing)
  • Visual microphones: reconstruct sound by extracting vibration information from video
  • Wider accessibility: sound recovery using consumer cameras via rolling shutter analysis
  • Beyond listening: reconstruct object properties and enable interaction/physics-based prediction from real-world footage

Main speakers / sources

  • Main speaker: Abe Davis
  • Referenced source technology: MIT colleagues (motion microscope software)
  • Video/music cited in experiments:
    • “Mary Had a Little Lamb”
    • “Under Pressure” by Queen
  • Event/organization credit: given at the end as a collective (“amazing people who worked with me”) without individual names in the subtitles.

Original video