---
title: "One File. Everyone Waits."
description: "One slow disk operation at Dropbox could make unrelated requests wait too. What should a server do while the disk is busy?"
canonical_url: "https://otabek.io/blogs/one-file-everyone-waits"
md_url: "https://otabek.io/blogs/one-file-everyone-waits.md"
language: "en"
last_updated: "2026-03-26"
tags: ["Linux", "Systems", "Performance"]
---

# One File. Everyone Waits.

I came across a problem in Dropbox's engineering blog that stuck with me: **one slow disk operation could make unrelated requests wait too.** One file was taking its time, and everyone else got dragged into the queue.

![Otabek seated at a café counter, turned toward his white cat Mimi and smiling as they play with her tennis ball while coffee brews in the background](/images/blogs/one-file-everyone-waits/cover-otabek-mimi-playing.png)

In their **2017 post**, Dropbox described Nginx handling large transfers alongside requests that needed quick responses. Disk I/O could stall the event loop. Their solution was to hand that I/O work to a **thread pool**, a separate group of worker threads. This helped reduce the stalls. [Dropbox's server optimization post](https://dropbox.tech/infrastructure/optimizing-web-servers-for-high-throughput-and-low-latency)

That's the part I find interesting: **while the disk is busy, what is the program doing?** Handling other requests, or just watching? As if staring at a download would make it finish faster 😄

Let's start with an ordinary file read and see where we get stuck.

### Just Read the File?

**I/O** means input/output: reading or writing data, such as moving bytes between a program and a file or socket.

We might start with this pseudocode:

```text
read a chunk from the file
wait until the read finishes
send that chunk
repeat
```

With a **blocking read**, the calling thread can't continue until the call returns. Other threads can still work; the entire machine hasn't frozen. But if this is the thread handling many clients, one slow read can delay unrelated requests.

I think of a café queue. One cashier, several customers. Someone orders a sandwich, and the cashier goes to the kitchen to watch it cook. The entire queue is now emotionally invested in that sandwich.

**Customer:** “I only wanted water.”

**Cashier:** “The sandwich is at 73%.”

### What About More Threads?

A **thread pool** is a team of workers. The main thread hands a file operation to a worker and handles another customer. The worker may wait for the disk, but the main thread can keep moving.

That's a perfectly useful solution. Dropbox's **2020 Nucleus sync-engine article** describes a control thread handing filesystem I/O, network I/O, and hashing work to dedicated threads or pools. [Testing sync at Dropbox](https://dropbox.tech/infrastructure/-testing-our-new-sync-engine)

These Dropbox posts talk about threads, not io_uring. But they got me thinking: **could io_uring help with the same kind of work?**

So why consider another approach? Managing workers and handing work between threads also costs something. With lots of small operations, we'd like a cheaper way to tell the operating system what needs doing and collect the answers.

### io_uring

**io_uring is a Linux interface for asynchronous I/O.** We can submit work and collect its result later, allowing other work to progress in between. It supports operations on files and sockets. [io_uring manual](https://man7.org/linux/man-pages/man7/io_uring.7.html)

Our café cashier now takes several orders, hands them to the kitchen, and calls out the order numbers when food is ready. Nobody needs to stare at the toaster.

The operating system's **kernel** manages access to storage and networking. Our application runs in **user space**. A **system call**, or syscall, is one way the application asks the kernel to do something.

With io_uring, they also share two queues in memory:

| Queue | Plain meaning | Who adds entries? |
| --- | --- | --- |
| Submission Queue, **SQ** | Work we want done | Our application |
| Completion Queue, **CQ** | Results we can collect | The kernel |

The queues reuse slots in a circular arrangement, which is where **ring** comes from. They carry descriptions and results of operations; we aren't stuffing an entire video into the ring. [Jens Axboe's introduction](https://www.kernel.dk/io_uring.pdf)

```text
Application → “Read chunk A, B, C” → Submission Queue
                                         ↓
                                      Kernel
                                         ↓
Application ← “B finished; A finished” ← Completion Queue
```

An individual request is an **SQE**: submission queue entry. An individual completion is a **CQE**. Fancy names for an order ticket and a result ticket.

### Where Does the Speed Come From?

First, we can have several independent operations **in flight**: submitted but not yet finished. While one read waits for storage, other useful work can progress.

Second, we can **batch** requests. Prepare several entries, then submit them together. That can reduce the number of trips between the application and the kernel. `io_uring_submit()` submits prepared work; its return value doesn't mean those file operations have finished. [Submission documentation](https://man7.org/linux/man-pages/man3/io_uring_submit.3.html)

This doesn't make the SSD physically faster. It can help us use the storage we already have more efficiently.

You may also hear “io_uring needs no syscalls.” That's too broad. Normal operation still uses them. An optional mode called **SQPOLL** has a kernel thread watch for submissions, trading CPU time for fewer submission syscalls; waking that thread can still require a syscall. We don't need this mode to understand or start using io_uring. [Setup flags](https://man7.org/linux/man-pages/man2/io_uring_setup.2.html)

### Wait, Which File Finished?

Look at the diagram again: we requested A before B, but B finished first. Independent operations can complete in a different order. We need to recognize each result.

Give each request a ticket number through **`user_data`**. That value comes back with its completion. Ticket 42 still belongs to request 42, even if ticket 43 finishes first. [Request identifiers](https://man7.org/linux/man-pages/man3/io_uring_cqe_get_data.3.html)

Using **liburing**, a C helper library, preparing a read can look like this:

```c
io_uring_prep_read(sqe, fd, buffer, 4096, 0);
io_uring_sqe_set_data64(sqe, 42);
```

These two lines need some setup first: initialize the ring, open the file as `fd`, get an available `sqe`, and allocate a buffer of at least 4,096 bytes. Then we can prepare this read of **up to 4,096 bytes from offset 0**, tagged with ticket 42. We still need to submit it. [Read preparation](https://man7.org/linux/man-pages/man3/io_uring_prep_read.3.html)

When its completion arrives, `cqe->res` tells us the outcome: a positive value is the number of bytes read, zero means end of file for this read, and a negative value reports an error. A short read is possible. Asking for 4,096 bytes doesn't guarantee we got all 4,096. [Read results](https://man7.org/linux/man-pages/man3/io_uring_prep_read.3.html), [short reads](https://man7.org/linux/man-pages/man2/read.2.html)

**Submitted isn't finished. Finished isn't necessarily successful.**

If there's nothing useful left to do, `io_uring_wait_cqe()` can wait for a completion. Successfully waiting for a result doesn't mean the read itself succeeded; check `cqe->res` too. After handling the result, mark the CQE consumed with `io_uring_cqe_seen()`. [Waiting](https://man7.org/linux/man-pages/man3/io_uring_wait_cqe.3.html), [consuming completions](https://man7.org/linux/man-pages/man3/io_uring_cqe_seen.3.html)

### Back to Our Files

How would this look in the **Linux backend of a Dropbox-like service**? We could read independent chunks for several downloads in small batches. As reads finish, we handle their results and make room for the next batch.

One file might be slow while another is ready. We can work on the ready one instead of making every customer wait for the slow file. We must still send each file's bytes in the correct order and respect the network protocol. A faster queue isn't permission to shuffle someone's holiday video.

For uploads, a similar design could write received chunks while other transfers continue. Hashing or compressing those chunks is separate CPU work; io_uring doesn't automatically do that for us.

There's a trap here: **asynchronous doesn't mean unlimited**. Suppose our design allows 32 active reads, each with a 256 KiB buffer. That's **8 MiB of read buffers**, before other overhead. Allow 10,000 such reads and that becomes roughly **2.44 GiB**.

If a client downloads slowly, holding more and more completed chunks only fills memory. Set a limit, pause new reads when downstream work is backed up, and resume when buffers are available. This is **backpressure**: “we're full, wait before sending more.”

**Me:** “I removed the waiting.”

**RAM:** “You moved it into me.”

The number of unfinished requests is often called **queue depth**. More isn't always better. Enough concurrency can keep a device busy; too much can increase memory use and make individual requests wait longer.

Databases deal with this too. **PostgreSQL 18** introduced asynchronous reads for work including sequential scans, bitmap heap scans, and vacuum. Its `io_method = io_uring` option requires a build with liburing support; the default method is `worker`. Its documentation also warns that unnecessarily high I/O concurrency can increase latency. [PostgreSQL 18 release notes](https://www.postgresql.org/docs/18/release-18.html), [I/O settings](https://www.postgresql.org/docs/18/runtime-config-resource.html#RUNTIME-CONFIG-RESOURCE-IO)

So this isn't only about moving giant videos. A database reading many pages can face the same question: **how much useful work can we keep moving while storage catches up?**

### What Can Still Go Wrong?

We can keep several reads going now. But what happens if we reuse a buffer before its read finishes? Or start writing bytes we haven't received yet?

**Buffers must stay alive.** For our read above, the kernel needs the buffer until the operation completes. Don't free it or reuse it for another request while that read is pending. Rust's **tokio-uring** handles this through ownership: a read takes ownership of the buffer and returns it with the result. [tokio-uring's ownership model](https://docs.rs/tokio-uring/latest/tokio_uring/#submit-based-operations)

**Some work has to wait.** If a write needs bytes from a read, submitting both independently doesn't make the write wait for those bytes. For our example, let's wait for the read to succeed, then submit the write. io_uring also supports linking requests, but we can leave that for another time. [Operation ordering](https://man7.org/linux/man-pages/man7/io_uring.7.html)

**A completed write isn't automatically a durable file.** Buffered data can still be in memory. Crash-safe storage needs an appropriate durability protocol, including synchronization such as `fsync` and, where necessary, directory synchronization. And a local disk completion alone says nothing about remote replication. [File synchronization](https://man7.org/linux/man-pages/man2/fsync.2.html)

| Situation | What I'd consider first |
| --- | --- |
| A small service already meeting its latency target | Keep the simpler implementation. |
| CPU time goes into compression or hashing | Optimize that work; changing I/O submission won't remove it. |
| The network link is already full | More disk requests won't increase its bandwidth. |
| A desktop app must run on Linux, macOS, and Windows | Use a portable abstraction with platform-specific backends. |
| The host doesn't allow io_uring | Use a supported fallback; Linux can restrict ring creation. |

That last limitation is real: kernel policy can disable or restrict creating io_uring instances. Supported operations also depend on the kernel. Check the environment your program will actually run in. [Kernel configuration](https://www.kernel.org/doc/html/latest/admin-guide/sysctl/kernel.html#io-uring-disabled)

### Conclusion

io_uring gives us a way to submit I/O work in batches, let multiple operations progress, and collect their results. A Dropbox-like service might use that to keep file transfers moving without blocking the thread coordinating them.

Does that mean we should replace every thread pool? Of course not. Try both approaches with the same workload and compare throughput, CPU use, memory, and **p99 latency**. That last number tells us how long 99% of requests took to finish. Test with and without cached data, and with slow clients too. One good benchmark on my laptop doesn't tell us how your server will behave.

Before changing the code, I'd start with the question that caught my attention in Dropbox's post: **while the disk is busy, what is our program doing?**

Share this with the friend who puts `async` in front of everything. Let them finish their sandwich first 😄


## Sitemap

See the full [Markdown sitemap](/sitemap.md) for all pages.