Asynchronous Programming allows a program to perform multiple tasks concurrently by using a single thread. In Python, the asyncio library provides a powerful set of tools for asynchronous programming, enabling us to write code that performs multiple tasks seemingly at once, without blocking the main program execution.
Asynchronous programming is ideal for I/O-bound tasks like network calls, file I/O, or any other operations that require waiting. Unlike multithreading or multiprocessing, asyncio achieves concurrency within a single thread by using an event loop, which manages asynchronous tasks.
Key Concepts in asyncio
Coroutines: Functions defined with async def that can be paused and resumed. They are the basic building blocks of asynchronous code in Python.
- Event Loop: The core of asyncio that runs asynchronous tasks. It manages the execution of coroutines and ensures they are scheduled and run in the right order.
- Tasks: Wrappers around coroutines that allow them to be scheduled in the event loop.
- Await: The await keyword pauses the execution of a coroutine until the awaited task completes.
- Futures: Represent the result of an asynchronous computation, similar to Task, but can be manually created and used.
1. Creating and Running Coroutines
To define a coroutine, use the async def syntax. Coroutines must be awaited to run; otherwise, they will not execute.
Example: Basic Coroutine
import asyncio
async def greet():
print("Hello")
await asyncio.sleep(1)
print("World")
# Run the coroutine
asyncio.run(greet())
Explanation:
- greet() is an asynchronous coroutine that prints “Hello”, waits for 1 second without blocking, and then prints “World”.
- asyncio.run() runs the coroutine and starts the event loop.
2. Using await to Pause Execution
The await keyword is used within coroutines to pause execution until an awaited coroutine or task completes. This non-blocking pause allows other tasks to run during the wait period.
Example: Multiple Awaits in a Coroutine
async def greet():
print("Hello")
await asyncio.sleep(2) # Pause for 2 seconds without blocking
print("World")
async def main():
print("Starting...")
await greet()
print("Finished!")
asyncio.run(main())
3. Running Multiple Coroutines Concurrently
To run multiple coroutines concurrently, you can use asyncio.gather() or create multiple asyncio.create_task() calls. This is useful for tasks that can run independently of each other.
Example: Running Coroutines with asyncio.gather
import asyncio
async def task1():
await asyncio.sleep(2)
print("Task 1 complete")
async def task2():
await asyncio.sleep(1)
print("Task 2 complete")
async def main():
await asyncio.gather(task1(), task2())
asyncio.run(main())
Explanation:
- asyncio.gather() allows task1 and task2 to run concurrently. Both tasks are awaited together, and each completes independently.
- Output order will be based on which task finishes first (in this case, task2).
Example: Using asyncio.create_task to Schedule Tasks
import asyncio
async def task1():
await asyncio.sleep(2)
print("Task 1 complete")
async def task2():
await asyncio.sleep(1)
print("Task 2 complete")
async def main():
task1_task = asyncio.create_task(task1())
task2_task = asyncio.create_task(task2())
await task1_task
await task2_task
asyncio.run(main())
asyncio.create_task() schedules each task to run concurrently. This is particularly useful when you want to control task execution individually.
4. Synchronizing Tasks with await and asyncio.gather
To wait for a collection of tasks to complete, use asyncio.gather(). This function collects all the results or waits for all tasks to complete before moving on.
Example: Running and Collecting Task Results
async def square(x):
await asyncio.sleep(1)
return x * x
async def main():
results = await asyncio.gather(square(1), square(2), square(3))
print(results)
asyncio.run(main())
# Output: [1, 4, 9]
Explanation: asyncio.gather() runs all square coroutines concurrently, waits for them to complete, and collects their results.
5. Managing Timeouts with asyncio.wait_for
You can set a timeout for a coroutine using asyncio.wait_for(). This will raise a TimeoutError if the coroutine takes too long.
Example: Using asyncio.wait_for to Set a Timeout
async def long_task():
await asyncio.sleep(5)
return "Task complete"
async def main():
try:
result = await asyncio.wait_for(long_task(), timeout=2)
print(result)
except asyncio.TimeoutError:
print("Task timed out")
asyncio.run(main())
# Output: Task timed out
Explanation: The long_task coroutine is designed to take 5 seconds, but asyncio.wait_for interrupts it after 2 seconds, raising a TimeoutError.
6. Using asyncio.Queue for Producer-Consumer Pattern
In asyncio, a Queue is a thread-safe data structure that allows multiple producers and consumers to interact. It is ideal for passing data between asynchronous tasks.
Example: Producer-Consumer with asyncio.Queue
import asyncio
async def producer(queue):
for i in range(5):
print(f"Producing {i}")
await queue.put(i)
await asyncio.sleep(1)
async def consumer(queue):
while True:
item = await queue.get()
if item is None:
break
print(f"Consuming {item}")
queue.task_done()
async def main():
queue = asyncio.Queue()
producer_task = asyncio.create_task(producer(queue))
consumer_task = asyncio.create_task(consumer(queue))
await producer_task
await queue.put(None) # Signal to consumer to stop
await consumer_task
asyncio.run(main())
Explanation: Producer: Puts items into the queue. Consumer: Consumes items from the queue until it finds None, which signals it to stop
7. Async Context Managers with async with
Async context managers use async with to handle asynchronous resources. For example, aiohttp (an async HTTP library) uses async context managers to open and close sessions properly.
Example: Using an Async Context Manager
import asyncio
import aiohttp
async def fetch_url(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
return await response.text()
async def main():
url = "https://example.com"
content = await fetch_url(url)
print(content[:100]) # Print the first 100 characters
asyncio.run(main())
Explanation: async with is used to handle asynchronous resources (like HTTP connections) that need to be closed after use.
8. Example: Asynchronous HTTP Requests with aiohttp
The aiohttp library provides asynchronous support for HTTP requests, which is perfect for making multiple requests concurrently.
Example: Fetch Multiple URLs
import asyncio
import aiohttp
async def fetch(session, url):
async with session.get(url) as response:
return await response.text()
async def fetch_all(urls):
async with aiohttp.ClientSession() as session:
tasks = [fetch(session, url) for url in urls]
return await asyncio.gather(*tasks)
async def main():
urls = ["https://example.com", "https://example.org", "https://example.net"]
contents = await fetch_all(urls)
for content in contents:
print(content[:100]) # Print the first 100 characters of each response
asyncio.run(main())