Asynchronous I/O in Windows for UNIX Programmers

Ryan Dahl ryan@joyent.com

This document assumes you are familiar with how non-blocking socket I/O is done in UNIX.

Windows has different notions for how asynchronous and non-blocking I/O are done. select() is supported in Window but it supports only 64 file descriptors—which is unacceptable. Microsoft understands how to make high-concurrency servers but they've choosen to do it with an system somewhat different than what one is used to UNIX. It is called overlapped I/O. The device by which overlapped socket I/O is polled for completion is an I/O completion port. It is more or less equivalent to kqueue (Macintosh and BSDs), epoll (Linux), event completion ports (Solaris), poll (modern UNIXes), or select (all operating systems). The main variation is that in UNIXes you generally ask the kernel to wait for file descriptors to change their readability or writablity, while in Windows you wait for asynchronous functions to complete.

For example, instead of waiting for a socket to become writable and then write(2) to it, as you do in UNIX operating systems, you would rather WSASend() a buffer and wait for it to have been sent.

The consequence of this different polling interface is that non-blocking write(2) and read(2) (among other calls) are not portable to Windows for high-performance servers.

In UNIX nearly everything has a file descriptor and read(2) and write(2) more or less work on all of them. This is a nice abstraction but for non-blocking I/O it does not dig as deep as one would like. The file system itself has no concept of non-blocking I/O—file descriptors for on disk files cannot be polled for readability, read(2) always has the possibility of blocking for an indefinite amount of time. UNIX users should not snub the Windows async API, in practice the explicit difference between sockets, pipes, on disk files, and TTYs seems make usage more clear where as in UNIX they deceptively seem seem like they should work similar but do not.

Almost every socket operation that you're familiar with has an overlapped counter-part. The following section tries to pair Windows overlapped I/O syscalls with non-blocking UNIX ones.

TCP Sockets

TCP Sockets are by far the most important stream to get right. Servers should expect to be handling tens of thousands of these per thread, concurrently. This is possible with overlapped I/O in Windows if one is careful to avoid UNIX-ism like file descriptors. (Windows has a hard limit of 2048 open file descriptors—see _setmaxstdio().)

send(2), write(2)

Windows: WSASend()

recv(2), read(2)

Windows: WSARecv()

connect(2)

Windows: ConnectEx()

Non-blocking connect() is has difficult semantics in UNIX. The proper way to connect to a remote host is this: call connect(2) while it returns EINPROGRESS poll on the file descriptor for writablity. Then use

int error;
socklen_t len = sizeof(int);
getsockopt(fd, SOL_SOCKET, SO_ERROR, &error, &len);

A zero error indicates that the connection succeeded. (Documented in connect(2) under EINPROGRESS on the Linux man page.)

accept(2)

Windows: AcceptEx()

sendfile(2)

Windows: TransmitFile()

The exact API of sendfile(2) on UNIX has not been agreed on yet. Each operating system does it slightly different. All sendfile(2) implementations (except possibly FreeBSD?) are blocking even on non-blocking sockets.

Marc Lehmann has written a portable version in libeio.

shutdown(2), graceful close, half-duplex connections

Graceful Shutdown, Linger Options, and Socket Closure
DisconnectEx()

close(2)

closesocket()

SOCKET

Named Pipes

AF_UNIX domain socketsAF_UNIX

/tmp/pipename

\\.\pipe\pipename

socket(AF_UNIX, SOCK_STREAM, 0), bind(2), listen(2): CreateNamedPipe()
Use FILE_FLAG_OVERLAPPED, PIPE_TYPE_BYTE, PIPE_NOWAIT.
send(2), write(2): WriteFileEx()
recv(2), read(2): ReadFileEx()
connect(2): CreateNamedPipe()
accept(2): ConnectNamedPipe()

On Disk Files

In UNIX file system files are not able to use non-blocking I/O. There are some operating systems that have asynchronous I/O but it is not standard and at least on Linux is done with pthreads in GNU libc. For this reason applications designed to be portable across different UNIXes must manage a thread pool for issuing file I/O syscalls.

The situation is better in Windows: true overlapped I/O is available when reading or writing a stream of data to a file.

write(2): Windows: WriteFileEx()
Solaris's event completion ports has true in-kernel async writes with aio_write(3RT)
read(2): Windows: ReadFileEx()
Solaris's event completion ports has true in-kernel async reads with aio_read(3RT)

Console/TTY

It is (usually?) possible to poll a UNIX TTY file descriptor for readability or writablity just like a TCP socket—this is very helpful and nice. In Windows the situation is worse, not only is it a completely different API but there are not overlapped versions to read and write to the TTY. Polling for readability can be accomplished by waiting in another thread with RegisterWaitForSingleObject().

read(2)

ReadConsole() and ReadConsoleInput() do not support overlapped I/O and there are no overlapped counter-parts. One strategy to get around this is

RegisterWaitForSingleObject(&tty_wait_handle, tty_handle,
  tty_want_poll, NULL, INFINITE, WT_EXECUTEINWAITTHREAD |
  WT_EXECUTEONLYONCE)

which will execute tty_want_poll() in a different thread. You can use this to notify the calling thread that ReadConsoleInput() will not block.

write(2)

WriteConsole() is also blocking but this is probably acceptable.

tcsetattr(3)

SetConsoleMode()

Asynchronous I/O in Windows for UNIX Programmers

TCP Sockets

Named Pipes

On Disk Files

Console/TTY

Links