04 Advanced File IO

Advanced File I/O

Scatter/Gather I/O (Vectored I/O)

Scatter/gather I/O permits a single system call to read from or write to a vector of disjoint buffers, interacting with a single data stream.

• Structure: Vectors are defined using the iovec structure, which contains a memory pointer (iov_base) and the buffer size (iov_len).
• System Calls: Implemented via readv() and writev(),. These calls fill or drain each buffer completely before sequentially advancing to the next segment (iov, iov, etc.),.
• Advantages over Linear I/O:
  • Allows intuitive manipulation of naturally segmented data (e.g., distinct fields in a predefined data structure).
  • Reduces system call overhead by replacing multiple linear operations with a single call.
  • Guarantees atomicity, preventing the risk of interleaved I/O from concurrent processes.
• Constraints & Optimizations:
  • The total segment count must be $>0$ and $\le$ IOV_MAX (capped at $1024$ on Linux),.
  • If the segment count is $\le 8$, the Linux kernel optimizes execution by constructing the segment array on the process’s kernel stack, bypassing dynamic memory allocation overhead,.

Grouping I/O operations improves linear throughput, but managing large numbers of distinct file descriptors concurrently requires efficient event monitoring subsystems.

Event Poll (epoll)

The epoll facility resolves the fundamental $O(N)$ scalability bottleneck of older multiplexing interfaces (poll() and select()), which forced the kernel to walk the entire file descriptor list on every invocation. Epoll decouples monitor registration from the actual event waiting.

• Initialization: epoll_create1(int flags) instantiates a new epoll context and returns a dedicated file descriptor handle.
• Context Control: epoll_ctl() adds (EPOLL_CTL_ADD), modifies (EPOLL_CTL_MOD), or removes (EPOLL_CTL_DEL) file descriptor watches within the epoll context,,.
• Event Structure: Operations utilize the epoll_event structure, containing a bitmask of monitored events (e.g., EPOLLIN for readable, EPOLLOUT for writable) and a user-data union (typically holding the target file descriptor),,,.
• Event Wait: epoll_wait() blocks until events occur on the monitored descriptors or a timeout elapses, returning populated epoll_event structures.
• Triggering Behaviors:
  • Level-Triggered (Default): Generates a notification as long as the state holds (e.g., while the descriptor remains readable),.
  • Edge-Triggered (EPOLLET): Generates a notification only when the state changes (e.g., upon data arrival),. This requires a non-blocking I/O design pattern and careful checking for EAGAIN errors to prevent missed data.

While epoll efficiently scales event notification for I/O bounds, direct mapping of files into memory can bypass traditional read/write paradigms entirely.

Memory-Mapped I/O

Memory-mapped I/O establishes a one-to-one correspondence between a process’s memory address space and words in a file.

• Creation: mmap() maps $len$ bytes of a file descriptor $fd$, starting at $offset$, into memory,.
• Alignment: The $offset$ and requested target address ($addr$) must be aligned to integer multiples of the system page size. If $len$ is not page-aligned, the mapping is rounded up to the next full page, with slack space filled with zeros.
• Permissions: Governed by $prot$ parameter (e.g., PROT_READ, PROT_WRITE, PROT_EXEC), which must not conflict with the file’s original open mode,.
• Mapping Types ($flags$):
  • MAP_SHARED: Modifications in memory are written to the underlying file and are visible to other processes mapping the same file,.
  • MAP_PRIVATE: Modifications trigger copy-on-write behavior, leaving the original file and other processes’ mappings unchanged.
• Manipulation Interfaces:
  • munmap(): Removes mappings and invalidates the memory region.
  • mremap(): Shrinks or expands existing mappings. Setting MREMAP_MAYMOVE permits the kernel to relocate the mapping to satisfy expansion requests,.
  • mprotect(): Alters access protection bits for existing memory pages,.
  • msync(): Synchronizes a dirty mapping back to disk, either synchronously (MS_SYNC) or asynchronously (MS_ASYNC),.
• Trade-offs: Mappings avoid the double-copy overhead of moving data between kernel space and user-space buffers. However, small files mapped to fixed page sizes introduce slack space fragmentation, and maintaining mapping data structures incurs distinct kernel overhead,.

Optimizing memory-mapped access and standard I/O relies heavily on how the kernel manages the page cache, a behavior applications can directly influence.

File and Mapping Advice

Applications can issue hints to the kernel regarding expected access patterns, enabling optimization of page caching and disk readahead behavior.

• Interfaces: madvise() acts on memory-mapped regions, while posix_fadvise() acts on file descriptors over specific offsets,.
• Advice Flags:
  • NORMAL: Standard kernel behavior with moderate readahead,.
  • RANDOM: Instructs the kernel to disable readahead, fetching only minimal data per physical read,.
  • SEQUENTIAL: Instructs the kernel to perform aggressive readahead, reading larger sequential chunks into the page cache,.
  • WILLNEED: Triggers immediate asynchronous readahead to populate the page cache before the application executes a blocking read,,.
  • DONTNEED: Evicts the specified range from the page cache. Useful for streaming applications to clear data that will not be accessed again,,.
• Direct Readahead: The Linux-specific readahead() system call directly populates the page cache for a specified file descriptor and range,.

Providing advice optimizes data caching, but dictating when operations return and when data reaches physical storage requires precise control over I/O synchronicity.

Synchronized, Synchronous, and Asynchronous Operations

I/O paradigms dictate how execution blocking and data persistence are structured,.

• Synchronous I/O: The operation blocks the thread until completed. A read blocks until data enters the user-space buffer; a write blocks until data is handed off to kernel buffers,.
• Asynchronous I/O (AIO): The operation returns immediately while the request executes in the background. The POSIX aio library (e.g., aio_read(), aio_write(), aio_suspend()) manages these non-blocking queues via the aiocb control block,,.
• Synchronized I/O: The operation guarantees that data (and often requisite metadata) is physically committed to the disk before returning. This restricts execution compared to nonsynchronized writes (which only guarantee data is stored in the kernel cache),.

Whether dispatched synchronously or asynchronously, the kernel ultimately aggregates and orders physical storage accesses to minimize mechanical latency.

I/O Schedulers and Performance Optimization

The kernel’s I/O scheduler reorganizes disk requests to minimize physical read/write head seek times across cylinders, heads, and sectors (CHS) via Logical Block Addressing (LBA),,,.

• Core Mechanisms: Schedulers rely on merging (coalescing adjacent block requests) and sorting (ordering requests in ascending block order),.
• The “Writes-Starving-Reads” Problem: Write operations typically stream asynchronously to the disk, whereas read operations block the calling process sequentially. Without intervention, continuous streaming writes can monopolize the disk head, heavily impacting read latency.
• Scheduler Implementations:
  • Deadline I/O Scheduler: Implements distinct read and write FIFO queues with hard expiration thresholds ($500$ ms for reads, $5$ seconds for writes). If a request hits its deadline, the scheduler abandons optimal block sorting to prioritize the expiring FIFO queue,.
  • Anticipatory I/O Scheduler: Extends the Deadline scheduler by pausing disk activity for up to $6$ ms after a read operation. It anticipates that the process will issue a subsequent dependent read in the same sector, saving the cost of two full disk seeks,.
  • CFQ (Completely Fair Queuing) Scheduler: Assigns timeslices to separate queues for each process. Queues are serviced round-robin. It favors synchronized requests (reads) over writes to combat starvation,.
  • Noop I/O Scheduler: Performs merging but does not sort requests. Primarily utilized for Solid-State Drives (SSDs) where random access incurs no rotational seek penalties,.
• User-Space I/O Scheduling: Mission-critical applications dispatching many concurrent requests can sort them in user space before submission to prevent scheduler backlog,,.
  • Sort by Path: Simple but inaccurate, heavily impacted by fragmentation,.
  • Sort by Inode: Extracted via stat(). A reliable heuristic for standard filesystems (like ext3/ext4) assuming adjacent inodes correlate with adjacent physical blocks,.
  • Sort by Physical Block: Extracted via ioctl(fd, FIBMAP). Maps logical file blocks to exact physical hardware blocks for perfect sorting, but requires CAP_SYS_RAWIO (root) privileges,,.