luo980/libuv
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs

Browse Source
This commit is contained in:
Ryan Dahl 2011-03-23 16:31:29 -07:00
parent f3eb0d90f7
commit 508eeb42c4
3 changed files with 291 additions and 224 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

485
iocp-links.html
View File

@ -1,251 +1,304 @@
<style> <html>
body { <head>
font-size: 12pt; <style>
font-family: Arial; body {
max-width: 40em; max-width: 40em;
margin: 1em; margin: 2em;
} }
a { a {
color: inherit; color: inherit;
} }
a:hover { a:hover {
color: red; color: red;
} }
table td { dt { margin-top: 1em; }
min-width: 10em; dd { margin-bottom: 1em; }
vertical-align: top; </style>
padding: 0.5em; <title>Asynchronous I/O in Windows for UNIX Programmers</title>
} </head>
<body>
table td {
border-bottom: 1px solid #bbb;
}
table tr:first-child td {
border-top: 1px solid #bbb;
}
</style>
<h1>Asynchronous I/O in Windows for UNIX Programmers</h1> <h1>Asynchronous I/O in Windows for UNIX Programmers</h1>
<p>Ryan Dahl ry@tinyclouds.org <p>Ryan Dahl ryan@joyent.com
<p>This document assumes you are familiar with how non-blocking socket I/O <p>This document assumes you are familiar with how non-blocking socket I/O
is done in UNIX. is done in UNIX.
<p>Windows has very different notions for how asynchronous and non-blocking I/O <p>Windows has different notions for how asynchronous and non-blocking I/O
are done. While Windows has <code>select()</code> it supports only 64 are done. <code>select()</code> is supported in Window but it supports only 64
file descriptors. Obviously Microsoft does understand how to make file descriptors&mdash;which is unacceptable.
high-concurrency servers, they've simply choosen a different paradigm for Microsoft understands how to make high-concurrency servers but they've
this called <a choosen to do it with an system somewhat different than what one is used to
UNIX. It is called <a
href="http://msdn.microsoft.com/en-us/library/ms686358(v=vs.85).aspx">overlapped href="http://msdn.microsoft.com/en-us/library/ms686358(v=vs.85).aspx">overlapped
I/O</a>. The mechanism in Windows by which multiple sockets are polled I/O</a>. The device by which overlapped socket I/O is polled for
for completion is called completion is an <a
<a href="http://msdn.microsoft.com/en-us/library/aa365198(VS.85).aspx">I/O href="http://msdn.microsoft.com/en-us/library/aa365198(VS.85).aspx">I/O
completion ports</a>. More or less equivlant to <a completion port</a>. It is more or less equivalent to <a
href="http://en.wikipedia.org/wiki/Kqueue">kqueue</a> (Macintosh, href="http://en.wikipedia.org/wiki/Kqueue">kqueue</a> (Macintosh and
FreeBSD, other BSDs), <a href="http://en.wikipedia.org/wiki/Epoll">epoll</a> BSDs), <a href="http://en.wikipedia.org/wiki/Epoll">epoll</a>
(Linux), <a (Linux), <a
href="http://developers.sun.com/solaris/articles/event_completion.html">event href="http://developers.sun.com/solaris/articles/event_completion.html">event
completion ports</a> (Solaris), <a href="">poll</a> (modern UNIXes), or <a completion ports</a> (Solaris), <a href="">poll</a> (modern UNIXes), or <a
href="http://www.kernel.org/doc/man-pages/online/pages/man2/select.2.html">select</a> href="http://www.kernel.org/doc/man-pages/online/pages/man2/select.2.html">select</a>
(all operating systems). The main difference is that in UNIX you ask the (all operating systems). The main variation is that in UNIXes you generally
kernel to wait for file descriptors to change their readability or ask the kernel to wait for file descriptors to change their readability or
writablity while in windows you wait for asynchronous functions to complete. writablity, while in Windows you wait for asynchronous functions to complete.
<p>
For example, instead of waiting for a socket to become writable and then For example, instead of waiting for a socket to become writable and then
<a <a
href="http://www.kernel.org/doc/man-pages/online/pages/man2/write.2.html"><code>write(2)</code></a> href="http://www.kernel.org/doc/man-pages/online/pages/man2/write.2.html"><code>write(2)</code></a>
to it, as you do in UNIX operating systems, you rather <a to it, as you do in UNIX operating systems, you would rather <a
href="http://msdn.microsoft.com/en-us/library/ms742203(v=vs.85).aspx"><code>WSASend()</code></a> href="http://msdn.microsoft.com/en-us/library/ms742203(v=vs.85).aspx"><code>WSASend()</code></a>
a buffer and wait for it to have been sent. a buffer and wait for it to have been sent.
The result is that non-blocking <code>write(2)</code> and <code>read(2)</code>
are non-portable to Windows. This tends to throw the poor sap assigned with
the job of porting your app to Windows into compulsive nervous twitches.
<p> <p>
Almost every socket operation that you're familar with has an The consequence of this different polling interface is that non-blocking
overlapped counter-part (<a href="#table-foot">see table</a>). <code>write(2)</code> and <code>read(2)</code> (among other calls) are not
portable to Windows for high-performance servers.
<p id="table-foot"> <p>In UNIX nearly everything has a file descriptor and <code>read(2)</code>
<table cellspacing=0> and <code>write(2)</code> more or less work on all of them. This is a nice
<!-- TODO: links --> abstraction but for non-blocking I/O it does not dig as deep as one would
<tr> like. The file system itself has no concept of non-blocking I/O&mdash;file
<td></td> descriptors for on disk files cannot be polled for readability,
<td> <code>read(2)</code> always has the possibility of blocking for an
<pre>int fd;</pre> indefinite amount of time. UNIX users should not snub the Windows async API,
</td> in practice the explicit difference between sockets, pipes, on disk files,
<td> and TTYs seems make usage more clear where as in UNIX they deceptively seem
<pre>HANDLE handle;</pre> seem like they should work similar but do not.
<pre>SOCKET socket;</pre>
(the two are the same type)
</tr>
<tr>
<td>socket or pipe</td>
<td>
<code>send(2)</code>,
<code>write(2)</code>
</td>
<td>
<a href="http://msdn.microsoft.com/en-us/library/ms742203(v=vs.85).aspx"><code>WSASend()</code></a>
</td>
</tr>
<tr>
<td>socket or pipe</td>
<td>
<code>recv(2)</code>,
<code>read(2)</code>
</td>
<td>
<a href="http://msdn.microsoft.com/en-us/library/ms741688(v=VS.85).aspx"><code>WSARecv()</code></a>
</td>
</tr>
<tr>
<td>socket</td> <p>
<td> Almost every socket operation that you're familiar with has an
<pre>connect(2)</pre> overlapped counter-part. The following section tries to pair Windows
Non-blocking <code>connect()</code> is has difficult semantics in overlapped I/O syscalls with non-blocking UNIX ones.
UNIX. The proper way to connect to a remote host is this: call
<code>connect(2)</code> which will usually return <code>EAGAIN</code>.
Poll on the file descriptor for writablity. Then use <h3>TCP Sockets</h3>
<pre>int error;
TCP Sockets are by far the most important stream to get right.
Servers should expect to be handling tens of thousands of these
per thread, concurrently. This is possible with overlapped I/O in Windows if
one is careful to avoid UNIX-ism like file descriptors. (Windows has a
hard limit of 2048 open file descriptors&mdash;see
<a
href="http://msdn.microsoft.com/en-us/library/6e3b887c.aspx"><code>_setmaxstdio()</code></a>.)
<dl>
<dt><code>send(2)</code>, <code>write(2)</code></dt>
<dd>Windows: <a href="http://msdn.microsoft.com/en-us/library/ms742203(v=vs.85).aspx"><code>WSASend()</code></a>
</dd>
<dt><code>recv(2)</code>, <code>read(2)</code></dt>
<dd>
Windows: <a href="http://msdn.microsoft.com/en-us/library/ms741688(v=VS.85).aspx"><code>WSARecv()</code></a>
</dd>
<dt><code>connect(2)</code></dt>
<dd>
Windows: <a href="http://msdn.microsoft.com/en-us/library/ms737606(VS.85).aspx"><code>ConnectEx()</code></a>
<p>
Non-blocking <code>connect()</code> is has difficult semantics in
UNIX. The proper way to connect to a remote host is this: call
<code>connect(2)</code> while it returns
<code>EINPROGRESS</code> poll on the file descriptor for writablity.
Then use
<pre>int error;
socklen_t len = sizeof(int); socklen_t len = sizeof(int);
getsockopt(fd, SOL_SOCKET, SO_ERROR, &error, &len);</pre> getsockopt(fd, SOL_SOCKET, SO_ERROR, &error, &len);</pre>
The <code>error</code> should be zero if the connection succeeded. A zero <code>error</code> indicates that the connection succeeded.
(Documented in <code>connect(2)</code> under <code>EINPROGRESS</code> (Documented in <code>connect(2)</code> under <code>EINPROGRESS</code>
on the Linux man page.) on the Linux man page.)
</td> </dd>
<td>
<a href="http://msdn.microsoft.com/en-us/library/ms737606(VS.85).aspx"><code>ConnectEx()</code></a>
</td>
</tr>
<tr>
<td>pipe</td>
<td>
<pre>connect(2)</pre>
</td>
<td>
<a
href="http://msdn.microsoft.com/en-us/library/aa365146(v=VS.85).aspx"><code>ConnectNamedPipe()</code></a>
Be sure to set <code>PIPE_NOWAIT</code> in <code>CreateNamedPipe()</code>
</td>
</tr>
<tr> <dt><code>accept(2)</code></dt>
<td>socket</td> <dd>
<td> Windows: <a href="http://msdn.microsoft.com/en-us/library/ms737524(v=VS.85).aspx"><code>AcceptEx()</code></a>
<pre>accept(2)</pre> </dd>
</td>
<td>
<a
href="http://msdn.microsoft.com/en-us/library/ms737524(v=VS.85).aspx"><code>AcceptEx()</code></a>
</td>
</tr>
<tr>
<td>pipe</td>
<td>
<pre>accept(2)</pre>
</td>
<td>
<a
href="http://msdn.microsoft.com/en-us/library/aa365146(v=VS.85).aspx"><code>ConnectNamedPipe()</code></a>
</td>
</tr>
<tr> <dt><code>sendfile(2)</code></dt>
<td>file</td> <dd>
<td> Windows: <a href="http://msdn.microsoft.com/en-us/library/ms740565(v=VS.85).aspx"><code>TransmitFile()</code></a>
<code>write(2)</code>
</td>
<td>
<a
href="http://msdn.microsoft.com/en-us/library/aa365748(v=VS.85).aspx"><code>WriteFileEx()</code></a>
</td>
</tr>
<tr>
<td>file</td>
<td>
<code>read(2)</code>
</td>
<td>
<a
href="http://msdn.microsoft.com/en-us/library/aa365468(v=VS.85).aspx"><code>ReadFileEx()</code></a>
</td>
</tr>
<tr>
<td>socket and file</td>
<td>
<code>sendfile()</code> [<a href="#sendfile-foot">1</a>]
</td>
<td>
<a
href="http://msdn.microsoft.com/en-us/library/ms740565(v=VS.85).aspx"><code>TransmitFile()</code></a>
</td>
</tr>
<tr> <p> The exact API of <code>sendfile(2)</code> on UNIX has not been agreed
<td>tty</td> on yet. Each operating system does it slightly different. All
<td> <code>sendfile(2)</code> implementations (except possibly FreeBSD?) are blocking
<a even on non-blocking sockets.
href="http://www.kernel.org/doc/man-pages/online/pages/man3/tcsetattr.3.html"><code>tcsetattr(3)</code></a> <ul>
</td> <li><a href="http://www.kernel.org/doc/man-pages/online/pages/man2/sendfile.2.html">Linux <code>sendfile(2)</code></a>
<td> <li><a href="http://www.freebsd.org/cgi/man.cgi?query=sendfile&sektion=2">FreeBSD <code>sendfile(2)</code></a>
<a href="http://msdn.microsoft.com/en-us/library/ms686033(VS.85).aspx"><code>SetConsoleMode()</code></a> <li><a href="http://www.manpagez.com/man/2/sendfile/">Darwin <code>sendfile(2)</code></a>
</td> </ul>
</tr> Marc Lehmann has written <a
href="https://github.com/joyent/node/blob/2c185a9dfd3be8e718858b946333c433c375c295/deps/libeio/eio.c#L954-1080">a
<tr> portable version in libeio</a>.
<td>tty</td> </dd>
<td>
<code>read(2)</code>
</td>
<td>
<a
href="http://msdn.microsoft.com/en-us/library/ms684958(v=VS.85).aspx"><code>ReadConsole()</code></a>
and
<a
href="http://msdn.microsoft.com/en-us/library/ms684961(v=VS.85).aspx"><code>ReadConsoleInput()</code></a>
do not support overlapped I/O and there are no overlapped
counter-parts. One strategy to get around this is
<pre><a
href="http://msdn.microsoft.com/en-us/library/ms685061(VS.85).aspx">RegisterWaitForSingleObject</a>(&tty_wait_handle, tty_handle,
tty_want_poll, NULL, INFINITE, WT_EXECUTEINWAITTHREAD |
WT_EXECUTEONLYONCE)</pre>
which will execute <code>tty_want_poll()</code> in a different thread.
You can use this to notify the calling thread that
<code>ReadConsoleInput()</code> will not block.
</td>
</tr>
<tr>
<td>tty</td>
<td>
<code>write(2)</code>
</td>
<td>
<a
href="http://msdn.microsoft.com/en-us/library/ms687401(v=VS.85).aspx"><code>WriteConsole()</code></a>
is also blocking but this is probably acceptable.
</td>
</tr>
</table>
<p id="sendfile-foot">[1] <code>sendfile()</code> on UNIX has not been agreed
on yet. Each operating system has a slightly different API.
<p id="foot2"> The following are nearly same in Windows overlapped and UNIX
non-blocking sockets. The only difference is that the UNIX variants
take integer file descriptors while Windows uses <code>SOCKET</code>.
<ul>
<li><a href="http://msdn.microsoft.com/en-us/library/ms740496(v=VS.85).aspx"><code>sockaddr</code></a>
<li><a href="http://msdn.microsoft.com/en-us/library/ms737550(v=VS.85).aspx"><code>bind()</code></a>
<li><a href="http://msdn.microsoft.com/en-us/library/ms738543(v=VS.85).aspx"><code>getsockname()</code></a>
</ul>
<h3>Named Pipes</h3>
Windows has "named pipes" which are more or less the same as <a
href="http://www.kernel.org/doc/man-pages/online/pages/man7/unix.7.html"><code>AF_UNIX</code>
domain sockets</a>. <code>AF_UNIX</code> sockets exist in the file system
often looking like
<pre>/tmp/<i>pipename</i></pre>
Windows named pipes have a path, but they are not directly part of the file
system; instead they look like
<pre>\\.\pipe\<i>pipename</i></pre>
<dl>
<dt><code>socket(AF_UNIX, SOCK_STREAM, 0), bind(2), listen(2)</code></dt>
<dd>
<a href="http://msdn.microsoft.com/en-us/library/aa365150(VS.85).aspx"><code>CreateNamedPipe()</code></a>
<p>Use <code>FILE_FLAG_OVERLAPPED</code>, <code>PIPE_TYPE_BYTE</code>,
<code>PIPE_NOWAIT</code>.
</dd>
<dt><code>send(2)</code>, <code>write(2)</code></dt>
<dd>
<a href="http://msdn.microsoft.com/en-us/library/aa365748(v=VS.85).aspx"><code>WriteFileEx()</code></a>
</dd>
<dt><code>recv(2)</code>, <code>read(2)</code></dt>
<dd>
<a href="http://msdn.microsoft.com/en-us/library/aa365468(v=VS.85).aspx"><code>ReadFileEx()</code></a>
</dd>
<dt><code>connect(2)</code></dt>
<dd>
<a href="http://msdn.microsoft.com/en-us/library/aa365150(VS.85).aspx"><code>CreateNamedPipe()</code></a>
</dd>
<dt><code>accept(2)</code></dt>
<dd>
<a href="http://msdn.microsoft.com/en-us/library/aa365146(v=VS.85).aspx"><code>ConnectNamedPipe()</code></a>
</dd>
</dl>
Examples:
<ul>
<li><a
href="http://msdn.microsoft.com/en-us/library/aa365601(v=VS.85).aspx">Named
Pipe Server Using Completion Routines</a>
<li><a
href="http://msdn.microsoft.com/en-us/library/aa365603(v=VS.85).aspx">Named
Pipe Server Using Overlapped I/O</a>
</ul>
<h3>On Disk Files</h3>
<p>
In UNIX file system files are not able to use non-blocking I/O. There are
some operating systems that have asynchronous I/O but it is not standard and
at least on Linux is done with pthreads in GNU libc. For this reason
applications designed to be portable across different UNIXes must manage a
thread pool for issuing file I/O syscalls.
<p>
The situation is better in Windows: true overlapped I/O is available when
reading or writing a stream of data to a file.
<dl>
<dt><code>write(2)</code></dt>
<dd> Windows:
<a href="http://msdn.microsoft.com/en-us/library/aa365748(v=VS.85).aspx"><code>WriteFileEx()</code></a>
<p>Solaris's event completion ports has true in-kernel async writes with <a
href="http://download.oracle.com/docs/cd/E19253-01/816-5171/aio-write-3rt/index.html">aio_write(3RT)</a>
</dd>
<dt><code>read(2)</code></dt>
<dd> Windows:
<a href="http://msdn.microsoft.com/en-us/library/aa365468(v=VS.85).aspx"><code>ReadFileEx()</code></a>
<p>Solaris's event completion ports has true in-kernel async reads with <a
href="http://download.oracle.com/docs/cd/E19253-01/816-5171/aio-read-3rt/index.html">aio_read(3RT)</a>
</dd>
</dl>
<h3>Console/TTY</h3>
<p>It is (usually?) possible to poll a UNIX TTY file descriptor for
readability or writablity just like a TCP socket&mdash;this is very helpful
and nice. In Windows the situation is worse, not only is it a completely
different API but there are not overlapped versions to read and write to the
TTY. Polling for readability can be accomplished by waiting in another
thread with <a
href="http://msdn.microsoft.com/en-us/library/ms685061(VS.85).aspx"><code>RegisterWaitForSingleObject()</code></a>.
<dl>
<dt><code>read(2)</code></dt>
<dd>
<a
href="http://msdn.microsoft.com/en-us/library/ms684958(v=VS.85).aspx"><code>ReadConsole()</code></a>
and
<a
href="http://msdn.microsoft.com/en-us/library/ms684961(v=VS.85).aspx"><code>ReadConsoleInput()</code></a>
do not support overlapped I/O and there are no overlapped
counter-parts. One strategy to get around this is
<pre><a href="http://msdn.microsoft.com/en-us/library/ms685061(VS.85).aspx">RegisterWaitForSingleObject</a>(&tty_wait_handle, tty_handle,
tty_want_poll, NULL, INFINITE, WT_EXECUTEINWAITTHREAD |
WT_EXECUTEONLYONCE)</pre>
which will execute <code>tty_want_poll()</code> in a different thread.
You can use this to notify the calling thread that
<code>ReadConsoleInput()</code> will not block.
</dd>
<dt><code>write(2)</code></dt>
<dd>
<a href="http://msdn.microsoft.com/en-us/library/ms687401(v=VS.85).aspx"><code>WriteConsole()</code></a>
is also blocking but this is probably acceptable.
</dd>
<dt><a
href="http://www.kernel.org/doc/man-pages/online/pages/man3/tcsetattr.3.html"><code>tcsetattr(3)</code></a></dt>
<dd>
<a href="http://msdn.microsoft.com/en-us/library/ms686033(VS.85).aspx"><code>SetConsoleMode()</code></a>
</dd>
</dl>
<h2 id="foot2">Links</h2>
<p> <p>
tips tips
<ul> <ul>
@ -297,3 +350,5 @@ Pipes:
&mdash; like <code>accept</code> is for UNIX pipes. &mdash; like <code>accept</code> is for UNIX pipes.
<li><a href="http://msdn.microsoft.com/en-us/library/aa365146(v=VS.85).aspx"><code>ConnectNamedPipe</code></a> <li><a href="http://msdn.microsoft.com/en-us/library/aa365146(v=VS.85).aspx"><code>ConnectNamedPipe</code></a>
</ul> </ul>
</body></html>

1
ol.h
View File

@ -18,7 +18,6 @@ typedef ol_connect_cb void(*)();
struct ol_buf; struct ol_buf;
/** /**
* Creates a tcp h. If bind_addr is NULL a random * Creates a tcp h. If bind_addr is NULL a random
* port will be bound. * port will be bound.

29
ol_unix_ev.c
View File

@ -22,6 +22,7 @@ ol_loop* ol_associate(ol_handle* handle)
{ {
} }
void ol_run(ol_loop *loop) { void ol_run(ol_loop *loop) {
ev_run(loop, 0); ev_run(loop, 0);
} }
@ -48,12 +49,17 @@ ol_handle* ol_tcp_new(int v4, ol_read_cb read_cb, ol_close_cb close_cb) {
} }
void handle_tcp_io() {
}
int try_connect(ol_handle* h) { int try_connect(ol_handle* h) {
int r = connect(h->fd, h->connect_addr, h->connect_addrlen); int r = connect(h->fd, h->connect_addr, h->connect_addrlen);
if (r != 0) { if (r != 0) {
if (errno == EINPROGRESS) { if (errno == EINPROGRESS) {
/* Wait for fd to become writable */ /* Wait for fd to become writable. */
h->connecting = 1; h->connecting = 1;
ev_io_init(&h->write_watcher, handle_tcp_io, h->fd, EV_WRITE); ev_io_init(&h->write_watcher, handle_tcp_io, h->fd, EV_WRITE);
ev_io_start(h->loop, &h->write_watcher); ev_io_start(h->loop, &h->write_watcher);
@ -61,6 +67,13 @@ int try_connect(ol_handle* h) {
return got_error("connect", errno); return got_error("connect", errno);
} }
/* Connected */
if (h->connect_cb) {
h->connect_cb(h);
h->connecting = 0;
h->connect_cb = NULL;
}
return 0; return 0;
} }
@ -77,14 +90,14 @@ int ol_connect(ol_handle* h, sockaddr* addr, sockaddr_len addrlen,
if (buf) { if (buf) {
ol_write(h, buf, 1, bytes_sent, cb); ol_write(h, buf, 1, bytes_sent, cb);
} else {
h->connect_cb = cb; h->connect_cb = cb;
} }
if (0 == try_connect(h)) { return try_connect(h);
if ( }
}
int ol_get_fd(ol_handle* h) {
return h->fd;
return 0;
} }