import asyncio: Learn Python's AsyncIO #3 - Using Coroutines

Main ideas & lessons

• AsyncIO episode purpose (Episode #3):

  • Introduces and focuses on the async / await keywords and how they enable asynchronous, single-threaded concurrency.
  • Builds from earlier concepts (event loop + trampolines / scheduling) toward coroutines, waiting, tasks, and cancellation.
  • Notes that futures and pitfalls are deferred to the next episode, though they’re briefly previewed.

• What await does (core behavior):

  • Inside an async def function, await blocks the current coroutine until the awaited object completes.
  • While waiting, control returns to the event loop, allowing other coroutines/tasks to run.
  • The event loop is single-threaded, so concurrency is cooperative rather than true parallel execution.

• Event loop execution timing isn’t real-time:

  • “Sleep for 0.5 seconds” means at least that long; single-thread cooperative scheduling means you can’t guarantee nanosecond precision.

• Running async code:

  • asyncio.run(coro_or_main) is the typical way to start an async program (used to run an “entry point” async function to completion).
  • For an infinite coroutine, interruption (e.g., Ctrl+C) is needed unless you add a timeout.

• Timeouts and graceful handling:

  • Wrapping an awaitable with asyncio.wait_for(..., timeout=...) triggers a timeout exception when time expires.
  • A more production-friendly pattern is defining an async main() and using try/except to handle timeouts.

• Awaitable objects (key vocabulary and distinctions):

  • Awaitable: anything usable in an await expression.
  • Coroutine vs async function:
    • An async def defines a function that, when called, creates a coroutine object.
    • A coroutine object is what you actually await.
    • Key constraints demonstrated:
      • Calling an async function without awaiting it creates an unused coroutine (warning in debug mode).
      • A coroutine can only be awaited once; re-awaiting raises an error.

• Awaiting multiple things: sequential vs concurrent

  • Multiple awaits written one after another run sequentially:
    • second await doesn’t start until first completes.
  • Concurrent execution is achieved using asyncio.gather(...):
    • all coroutines are scheduled so they make progress “at the same time” (concurrently, still single-threaded).

• Cancellation is exception-based and propagates

  • When a wait_for times out, the awaited operation is cancelled.
  • Cancellation propagates:
    • wait_for cancels gather
    • gather cancels the coroutines inside it
    • each coroutine receives cancellation at its current await point (raising CancelledError).
  • To handle cancellation cleanly, you either:
    • catch asyncio.CancelledError inside the coroutine, or
    • let it bubble to the caller.

• Tasks enable background concurrency and management

  • asyncio.create_task(coro) schedules a coroutine to run in the background.
  • Tasks run when the event loop gets control (during awaits elsewhere).
  • Compared to just awaiting coroutines directly, tasks provide:
    • handles for later result retrieval or cancellation
    • better control over long-running background work

---

Methodologies / step-by-step patterns shown

1) Basic coroutine + await asyncio.sleep pattern

• Define:
  • a normal helper to print timestamps
  • an async def coroutine that loops forever:
    • repeatedly prints
    • await asyncio.sleep(interval)
• Run:
  • with asyncio.run(...)
• Stop it:
  • via Ctrl+C, or more cleanly via asyncio.wait_for(..., timeout=...) wrapped in try/except.

2) “Async main entry point” pattern with timeout

• Create an async main() coroutine and call it via asyncio.run(async_main()).
• Inside async main():
  • wrap the awaited operation with asyncio.wait_for(..., timeout=...)
  • catch the timeout exception (handled gracefully).
• Benefit:
  • avoids messy tracebacks from infinite loops.

3) Correct use of awaitables and coroutines (avoid common bugs)

• Ensure you await the correct object:
  • async def function call → returns coroutine object (awaitable)
  • await should be applied to that coroutine object (or gather result).
• Don’t forget await:
  • un-awaited coroutine objects trigger warnings in debug/developer modes.
• Don’t attempt to await the same coroutine object twice:
  • coroutine objects are single-use.

4) Run multiple coroutines concurrently with asyncio.gather

• Put multiple coroutines inside a single asyncio.gather(...) call.
• Wrap gather with asyncio.wait_for for a global timeout.
• Use this when you want “all progress together” rather than sequential execution.

5) Cancellation handling inside coroutines

• When cancellation occurs:
  • the coroutine raises asyncio.CancelledError at the currently-running await.
• Options:
  • Handle cancellation locally using:
    • try: await ... except asyncio.CancelledError: ...
  • Or let it bubble up to the code awaiting the coroutine.

6) Task-based web crawler: evolution from slow to concurrent and cancellable

The video demonstrates iterative improvements.

Initial version (slow / not ideal):

• A recursive async crawler:
  • uses await on recursive crawl(url) calls directly
• Problem types explicitly called out:
  • reporting progress inside tasks (mixing concerns)
  • deep recursion can be annoying at scale
  • only one URL at a time → limited concurrency
  • creating the HTTP client in an ad-hoc way instead of using a context manager

Improvement #1: separate progress reporting

• Add:
  • a progress() coroutine
  • a shared structure (a set) to track what remains
• Progress coroutine:
  • periodically prints status (using await asyncio.sleep(...))
  • measures elapsed time

Improvement #2: use create_task for concurrency

• Instead of await crawl(next_url):
  • schedule crawl(next_url) as a background task using asyncio.create_task(...)
• Maintain “pending work” via a set.

Improvement #3: track tasks so cancellations can stop pending work

• Keep a set of tasks (not just URLs).
• Modify progress to:
  • await asyncio.wait(tasks, timeout=..., return_when=...)
  • receive (done, pending) sets
  • update the tracking set accordingly
• Result:
  • faster overall (demonstrated: ~13 seconds vs ~94 seconds in the demo)
  • cleaner introspection into what’s running/pending

Improvement #4: graceful shutdown by cancelling pending tasks

• In a new async main:
  • on asyncio.CancelledError:
    • cancel all pending tasks in the tracking set
    • remove completed vs still-pending tasks during shutdown
    • note a possible edge case:
      • tasks might be added while cancellation is underway (handled conceptually as something to consider)

---

Key concepts explained (quick outline)

• Cooperative multitasking: single-thread, event-loop scheduling, switching only at await points.
• Sequential await vs concurrent await (gather).
• Coroutine lifecycle:
  • coroutine creation (calling async def)
  • awaiting (execution begins)
  • cancellation at await points.
• Tasks: background execution + explicit handles for management.
• Cancellation:
  • propagates through wait_for → gather → coroutines
  • implemented as CancelledError thrown at await sites
  • may require careful cleanup and potentially multiple shutdown passes.

---

Speakers / sources featured

• Lucas (HDB) — the presenter: “hi this is Lucas from HDB…”
• Referenced contributors / teachers (not present as speakers in the video text):
  • Shaw — mentioned as advising about subtitle contrast/legibility.
  • Dave Beasley — mentioned as sharing a similar viewpoint on subtitle/background contrast.