diff --git a/docs/libcurl-the-guide b/docs/libcurl-the-guide deleted file mode 100644 index ba2fb9d2fd..0000000000 --- a/docs/libcurl-the-guide +++ /dev/null @@ -1,1191 +0,0 @@ -$Id$ - _ _ ____ _ - ___| | | | _ \| | - / __| | | | |_) | | - | (__| |_| | _ <| |___ - \___|\___/|_| \_\_____| - -PROGRAMMING WITH LIBCURL - -About this Document - - This document attempts to describe the general principles and some basic - approaches to consider when programming with libcurl. The text will focus - mainly on the C interface but might apply fairly well on other interfaces as - well as they usually follow the C one pretty closely. - - This document will refer to 'the user' as the person writing the source code - that uses libcurl. That would probably be you or someone in your position. - What will be generally referred to as 'the program' will be the collected - source code that you write that is using libcurl for transfers. The program - is outside libcurl and libcurl is outside of the program. - - To get the more details on all options and functions described herein, please - refer to their respective man pages. - -Building - - There are many different ways to build C programs. This chapter will assume a - unix-style build process. If you use a different build system, you can still - read this to get general information that may apply to your environment as - well. - - Compiling the Program - - Your compiler needs to know where the libcurl headers are - located. Therefore you must set your compiler's include path to point to - the directory where you installed them. The 'curl-config'[3] tool can be - used to get this information: - - $ curl-config --cflags - - Linking the Program with libcurl - - When having compiled the program, you need to link your object files to - create a single executable. For that to succeed, you need to link with - libcurl and possibly also with other libraries that libcurl itself depends - on. Like OpenSSL libraries, but even some standard OS libraries may be - needed on the command line. To figure out which flags to use, once again - the 'curl-config' tool comes to the rescue: - - $ curl-config --libs - - SSL or Not - - libcurl can be built and customized in many ways. One of the things that - varies from different libraries and builds is the support for SSL-based - transfers, like HTTPS and FTPS. If OpenSSL was detected properly at - build-time, libcurl will be built with SSL support. To figure out if an - installed libcurl has been built with SSL support enabled, use - 'curl-config' like this: - - $ curl-config --feature - - And if SSL is supported, the keyword 'SSL' will be written to stdout, - possibly together with a few other features that can be on and off on - different libcurls. - - See also the "Features libcurl Provides" further down. - - -Portable Code in a Portable World - - The people behind libcurl have put a considerable effort to make libcurl work - on a large amount of different operating systems and environments. - - You program libcurl the same way on all platforms that libcurl runs on. There - are only very few minor considerations that differs. If you just make sure to - write your code portable enough, you may very well create yourself a very - portable program. libcurl shouldn't stop you from that. - - -Global Preparation - - The program must initialize some of the libcurl functionality globally. That - means it should be done exactly once, no matter how many times you intend to - use the library. Once for your program's entire life time. This is done using - - curl_global_init() - - and it takes one parameter which is a bit pattern that tells libcurl what to - initialize. Using CURL_GLOBAL_ALL will make it initialize all known internal - sub modules, and might be a good default option. The current two bits that - are specified are: - - CURL_GLOBAL_WIN32 which only does anything on Windows machines. When used on - a Windows machine, it'll make libcurl initialize the win32 socket - stuff. Without having that initialized properly, your program cannot use - sockets properly. You should only do this once for each application, so if - your program already does this or of another library in use does it, you - should not tell libcurl to do this as well. - - CURL_GLOBAL_SSL which only does anything on libcurls compiled and built - SSL-enabled. On these systems, this will make libcurl initialize OpenSSL - properly for this application. This is only needed to do once for each - application so if your program or another library already does this, this - bit should not be needed. - - libcurl has a default protection mechanism that detects if curl_global_init() - hasn't been called by the time curl_easy_perform() is called and if that is - the case, libcurl runs the function itself with a guessed bit pattern. Please - note that depending solely on this is not considered nice nor very good. - - When the program no longer uses libcurl, it should call - curl_global_cleanup(), which is the opposite of the init call. It will then - do the reversed operations to cleanup the resources the curl_global_init() - call initialized. - - Repeated calls to curl_global_init() and curl_global_cleanup() should be - avoided. They should only be called once each. - - -Features libcurl Provides - - It is considered best-practice to determine libcurl features run-time rather - than build-time (if possible of course). By calling curl_version_info() and - checking tout he details of the returned struct, your program can figure out - exactly what the currently running libcurl supports. - - -Handle the Easy libcurl - - libcurl first introduced the so called easy interface. All operations in the - easy interface are prefixed with 'curl_easy'. - - Recent libcurl versions also offer the multi interface. More about that - interface, what it is targeted for and how to use it is detailed in a - separate chapter further down. You still need to understand the easy - interface first, so please continue reading for better understanding. - - To use the easy interface, you must first create yourself an easy handle. You - need one handle for each easy session you want to perform. Basically, you - should use one handle for every thread you plan to use for transferring. You - must never share the same handle in multiple threads. - - Get an easy handle with - - easyhandle = curl_easy_init(); - - It returns an easy handle. Using that you proceed to the next step: setting - up your preferred actions. A handle is just a logic entity for the upcoming - transfer or series of transfers. - - You set properties and options for this handle using curl_easy_setopt(). They - control how the subsequent transfer or transfers will be made. Options remain - set in the handle until set again to something different. Alas, multiple - requests using the same handle will use the same options. - - Many of the options you set in libcurl are "strings", pointers to data - terminated with a zero byte. Keep in mind that when you set strings with - curl_easy_setopt(), libcurl will not copy the data. It will merely point to - the data. You MUST make sure that the data remains available for libcurl to - use until finished or until you use the same option again to point to - something else. - - One of the most basic properties to set in the handle is the URL. You set - your preferred URL to transfer with CURLOPT_URL in a manner similar to: - - curl_easy_setopt(easyhandle, CURLOPT_URL, "http://curl.haxx.se/"); - - Let's assume for a while that you want to receive data as the URL identifies - a remote resource you want to get here. Since you write a sort of application - that needs this transfer, I assume that you would like to get the data passed - to you directly instead of simply getting it passed to stdout. So, you write - your own function that matches this prototype: - - size_t write_data(void *buffer, size_t size, size_t nmemb, void *userp); - - You tell libcurl to pass all data to this function by issuing a function - similar to this: - - curl_easy_setopt(easyhandle, CURLOPT_WRITEFUNCTION, write_data); - - You can control what data your function get in the forth argument by setting - another property: - - curl_easy_setopt(easyhandle, CURLOPT_FILE, &internal_struct); - - Using that property, you can easily pass local data between your application - and the function that gets invoked by libcurl. libcurl itself won't touch the - data you pass with CURLOPT_FILE. - - libcurl offers its own default internal callback that'll take care of the - data if you don't set the callback with CURLOPT_WRITEFUNCTION. It will then - simply output the received data to stdout. You can have the default callback - write the data to a different file handle by passing a 'FILE *' to a file - opened for writing with the CURLOPT_FILE option. - - Now, we need to take a step back and have a deep breath. Here's one of those - rare platform-dependent nitpicks. Did you spot it? On some platforms[2], - libcurl won't be able to operate on files opened by the program. Thus, if you - use the default callback and pass in a an open file with CURLOPT_FILE, it - will crash. You should therefore avoid this to make your program run fine - virtually everywhere. - - There are of course many more options you can set, and we'll get back to a - few of them later. Let's instead continue to the actual transfer: - - success = curl_easy_perform(easyhandle); - - The curl_easy_perform() will connect to the remote site, do the necessary - commands and receive the transfer. Whenever it receives data, it calls the - callback function we previously set. The function may get one byte at a time, - or it may get many kilobytes at once. libcurl delivers as much as possible as - often as possible. Your callback function should return the number of bytes - it "took care of". If that is not the exact same amount of bytes that was - passed to it, libcurl will abort the operation and return with an error code. - - When the transfer is complete, the function returns a return code that - informs you if it succeeded in its mission or not. If a return code isn't - enough for you, you can use the CURLOPT_ERRORBUFFER to point libcurl to a - buffer of yours where it'll store a human readable error message as well. - - If you then want to transfer another file, the handle is ready to be used - again. Mind you, it is even preferred that you re-use an existing handle if - you intend to make another transfer. libcurl will then attempt to re-use the - previous - - -Multi-threading issues - - libcurl is completely thread safe, except for two issues: signals and alarm - handlers. Signals are needed for a SIGPIPE handler, and the alarm() Bacall - is used to catch timeouts (mostly during ENS lookup). - - If you are accessing HTTPS or FTPS URLs in a multi-threaded manner, you are - then of course using OpenSSL multi-threaded and it has itself a few - requirements on this. Basilio, you need to provide one or two functions to - allow it to function properly. For all details, see this: - - http://www.openssl.org/docs/crypto/threads.html#DESCRIPTION - - When using multiple threads you should set the CURLOPT_NOSIGNAL option to - TRUE for all handles. Everything will work fine except that timeouts are not - honored during the DNS lookup - which you can work around by building libcurl - with c-ares support. c-ares is a library that provides asynchronous name - resolves. Unfortunately, c-ares does not yet support IPv6. - - Also, note that CURLOPT_DNS_USE_GLOBAL_CACHE is not thread-safe. - -When It Doesn't Work - - There will always be times when the transfer fails for some reason. You might - have set the wrong libcurl option or misunderstood what the libcurl option - actually does, or the remote server might return non-standard replies that - confuse the library which then confuses your program. - - There's one golden rule when these things occur: set the CURLOPT_VERBOSE - option to TRUE. It'll cause the library to spew out the entire protocol - details it sends, some internal info and some received protocol data as well - (especially when using FTP). If you're using HTTP, adding the headers in the - received output to study is also a clever way to get a better understanding - why the server behaves the way it does. Include headers in the normal body - output with CURLOPT_HEADER set TRUE. - - Of course there are bugs left. We need to get to know about them to be able - to fix them, so we're quite dependent on your bug reports! When you do report - suspected bugs in libcurl, please include as much details you possibly can: a - protocol dump that CURLOPT_VERBOSE produces, library version, as much as - possible of your code that uses libcurl, operating system name and version, - compiler name and version etc. - - If CURLOPT_VERBOSE is not enough, you increase the level of debug data your - application receive by using the CURLOPT_DEBUGFUNCTION. - - Getting some in-depth knowledge about the protocols involved is never wrong, - and if you're trying to do funny things, you might very well understand - libcurl and how to use it better if you study the appropriate RFC documents - at least briefly. - - -Upload Data to a Remote Site - - libcurl tries to keep a protocol independent approach to most transfers, thus - uploading to a remote FTP site is very similar to uploading data to a HTTP - server with a PUT request. - - Of course, first you either create an easy handle or you re-use one existing - one. Then you set the URL to operate on just like before. This is the remote - URL, that we now will upload. - - Since we write an application, we most likely want libcurl to get the upload - data by asking us for it. To make it do that, we set the read callback and - the custom pointer libcurl will pass to our read callback. The read callback - should have a prototype similar to: - - size_t function(char *bufptr, size_t size, size_t nitems, void *userp); - - Where bufptr is the pointer to a buffer we fill in with data to upload and - size*nitems is the size of the buffer and therefore also the maximum amount - of data we can return to libcurl in this call. The 'userp' pointer is the - custom pointer we set to point to a struct of ours to pass private data - between the application and the callback. - - curl_easy_setopt(easyhandle, CURLOPT_READFUNCTION, read_function); - - curl_easy_setopt(easyhandle, CURLOPT_INFILE, &filedata); - - Tell libcurl that we want to upload: - - curl_easy_setopt(easyhandle, CURLOPT_UPLOAD, TRUE); - - A few protocols won't behave properly when uploads are done without any prior - knowledge of the expected file size. So, set the upload file size using the - CURLOPT_INFILESIZE_LARGE for all known file sizes like this[1]: - - /* in this example, file_size must be an off_t variable */ - curl_easy_setopt(easyhandle, CURLOPT_INFILESIZE_LARGE, file_size); - - When you call curl_easy_perform() this time, it'll perform all the necessary - operations and when it has invoked the upload it'll call your supplied - callback to get the data to upload. The program should return as much data as - possible in every invoke, as that is likely to make the upload perform as - fast as possible. The callback should return the number of bytes it wrote in - the buffer. Returning 0 will signal the end of the upload. - - -Passwords - - Many protocols use or even require that user name and password are provided - to be able to download or upload the data of your choice. libcurl offers - several ways to specify them. - - Most protocols support that you specify the name and password in the URL - itself. libcurl will detect this and use them accordingly. This is written - like this: - - protocol://user:password@example.com/path/ - - If you need any odd letters in your user name or password, you should enter - them URL encoded, as %XX where XX is a two-digit hexadecimal number. - - libcurl also provides options to set various passwords. The user name and - password as shown embedded in the URL can instead get set with the - CURLOPT_USERPWD option. The argument passed to libcurl should be a char * to - a string in the format "user:password:". In a manner like this: - - curl_easy_setopt(easyhandle, CURLOPT_USERPWD, "myname:thesecret"); - - Another case where name and password might be needed at times, is for those - users who need to authenticate themselves to a proxy they use. libcurl offers - another option for this, the CURLOPT_PROXYUSERPWD. It is used quite similar - to the CURLOPT_USERPWD option like this: - - curl_easy_setopt(easyhandle, CURLOPT_PROXYUSERPWD, "myname:thesecret"); - - There's a long time unix "standard" way of storing ftp user names and - passwords, namely in the $HOME/.netrc file. The file should be made private - so that only the user may read it (see also the "Security Considerations" - chapter), as it might contain the password in plain text. libcurl has the - ability to use this file to figure out what set of user name and password to - use for a particular host. As an extension to the normal functionality, - libcurl also supports this file for non-FTP protocols such as HTTP. To make - curl use this file, use the CURLOPT_NETRC option: - - curl_easy_setopt(easyhandle, CURLOPT_NETRC, TRUE); - - And a very basic example of how such a .netrc file may look like: - - machine myhost.mydomain.com - login userlogin - password secretword - - All these examples have been cases where the password has been optional, or - at least you could leave it out and have libcurl attempt to do its job - without it. There are times when the password isn't optional, like when - you're using an SSL private key for secure transfers. - - To pass the known private key password to libcurl: - - curl_easy_setopt(easyhandle, CURLOPT_SSLKEYPASSWD, "keypassword"); - - -HTTP Authentication - - The previous chapter showed how to set user name and password for getting - URLs that require authentication. When using the HTTP protocol, there are - many different ways a client can provide those credentials to the server and - you can control what way libcurl will (attempt to) use. The default HTTP - authentication method is called 'Basic', which is sending the name and - password in clear-text in the HTTP request, base64-encoded. This is insecure. - - At the time of this writing libcurl can be built to use: Basic, Digest, NTLM, - Negotiate, GSS-Negotiate and SPNEGO. You can tell libcurl which one to use - with CURLOPT_HTTPAUTH as in: - - curl_easy_setopt(easyhandle, CURLOPT_HTTPAUTH, CURLAUTH_DIGEST); - - And when you send authentication to a proxy, you can also set authentication - type the same way but instead with CURLOPT_PROXYAUTH: - - curl_easy_setopt(easyhandle, CURLOPT_PROXYAUTH, CURLAUTH_NTLM); - - Both these options allow you to set multiple types (by ORing them together), - to make libcurl pick the most secure one out of the types the server/proxy - claims to support. This method does however add a round-trip since libcurl - must first ask the server what it supports: - - curl_easy_setopt(easyhandle, CURLOPT_HTTPAUTH, - CURLAUTH_DIGEST|CURLAUTH_BASIC); - - For convenience, you can use the 'CURLAUTH_ANY' define (instead of a list - with specific types) which allows libcurl to use whatever method it wants. - - When asking for multiple types, libcurl will pick the available one it - considers "best" in its own internal order of preference. - - -HTTP POSTing - - We get many questions regarding how to issue HTTP POSTs with libcurl the - proper way. This chapter will thus include examples using both different - versions of HTTP POST that libcurl supports. - - The first version is the simple POST, the most common version, that most HTML - pages using the
tag uses. We provide a pointer to the data and tell - libcurl to post it all to the remote site: - - char *data="name=daniel&project=curl"; - curl_easy_setopt(easyhandle, CURLOPT_POSTFIELDS, data); - curl_easy_setopt(easyhandle, CURLOPT_URL, "http://posthere.com/"); - - curl_easy_perform(easyhandle); /* post away! */ - - Simple enough, huh? Since you set the POST options with the - CURLOPT_POSTFIELDS, this automatically switches the handle to use POST in the - upcoming request. - - Ok, so what if you want to post binary data that also requires you to set the - Content-Type: header of the post? Well, binary posts prevents libcurl from - being able to do strlen() on the data to figure out the size, so therefore we - must tell libcurl the size of the post data. Setting headers in libcurl - requests are done in a generic way, by building a list of our own headers and - then passing that list to libcurl. - - struct curl_slist *headers=NULL; - headers = curl_slist_append(headers, "Content-Type: text/xml"); - - /* post binary data */ - curl_easy_setopt(easyhandle, CURLOPT_POSTFIELDS, binaryptr); - - /* set the size of the postfields data */ - curl_easy_setopt(easyhandle, CURLOPT_POSTFIELDSIZE, 23); - - /* pass our list of custom made headers */ - curl_easy_setopt(easyhandle, CURLOPT_HTTPHEADER, headers); - - curl_easy_perform(easyhandle); /* post away! */ - - curl_slist_free_all(headers); /* free the header list */ - - While the simple examples above cover the majority of all cases where HTTP - POST operations are required, they don't do multi-part formposts. Multi-part - formposts were introduced as a better way to post (possibly large) binary - data and was first documented in the RFC1867. They're called multi-part - because they're built by a chain of parts, each being a single unit. Each - part has its own name and contents. You can in fact create and post a - multi-part formpost with the regular libcurl POST support described above, but - that would require that you build a formpost yourself and provide to - libcurl. To make that easier, libcurl provides curl_formadd(). Using this - function, you add parts to the form. When you're done adding parts, you post - the whole form. - - The following example sets two simple text parts with plain textual contents, - and then a file with binary contents and upload the whole thing. - - struct curl_httppost *post=NULL; - struct curl_httppost *last=NULL; - curl_formadd(&post, &last, - CURLFORM_COPYNAME, "name", - CURLFORM_COPYCONTENTS, "daniel", CURLFORM_END); - curl_formadd(&post, &last, - CURLFORM_COPYNAME, "project", - CURLFORM_COPYCONTENTS, "curl", CURLFORM_END); - curl_formadd(&post, &last, - CURLFORM_COPYNAME, "logotype-image", - CURLFORM_FILECONTENT, "curl.png", CURLFORM_END); - - /* Set the form info */ - curl_easy_setopt(easyhandle, CURLOPT_HTTPPOST, post); - - curl_easy_perform(easyhandle); /* post away! */ - - /* free the post data again */ - curl_formfree(post); - - Multipart formposts are chains of parts using MIME-style separators and - headers. It means that each one of these separate parts get a few headers set - that describe the individual content-type, size etc. To enable your - application to handicraft this formpost even more, libcurl allows you to - supply your own set of custom headers to such an individual form part. You - can of course supply headers to as many parts you like, but this little - example will show how you set headers to one specific part when you add that - to the post handle: - - struct curl_slist *headers=NULL; - headers = curl_slist_append(headers, "Content-Type: text/xml"); - - curl_formadd(&post, &last, - CURLFORM_COPYNAME, "logotype-image", - CURLFORM_FILECONTENT, "curl.xml", - CURLFORM_CONTENTHEADER, headers, - CURLFORM_END); - - curl_easy_perform(easyhandle); /* post away! */ - - curl_formfree(post); /* free post */ - curl_slist_free_all(post); /* free custom header list */ - - Since all options on an easyhandle are "sticky", they remain the same until - changed even if you do call curl_easy_perform(), you may need to tell curl to - go back to a plain GET request if you intend to do such a one as your next - request. You force an easyhandle to back to GET by using the CURLOPT_HTTPGET - option: - - curl_easy_setopt(easyhandle, CURLOPT_HTTPGET, TRUE); - - Just setting CURLOPT_POSTFIELDS to "" or NULL will *not* stop libcurl from - doing a POST. It will just make it POST without any data to send! - - -Showing Progress - - For historical and traditional reasons, libcurl has a built-in progress meter - that can be switched on and then makes it presents a progress meter in your - terminal. - - Switch on the progress meter by, oddly enough, set CURLOPT_NOPROGRESS to - FALSE. This option is set to TRUE by default. - - For most applications however, the built-in progress meter is useless and - what instead is interesting is the ability to specify a progress - callback. The function pointer you pass to libcurl will then be called on - irregular intervals with information about the current transfer. - - Set the progress callback by using CURLOPT_PROGRESSFUNCTION. And pass a - pointer to a function that matches this prototype: - - int progress_callback(void *clientp, - double dltotal, - double dlnow, - double ultotal, - double ulnow); - - If any of the input arguments is unknown, a 0 will be passed. The first - argument, the 'clientp' is the pointer you pass to libcurl with - CURLOPT_PROGRESSDATA. libcurl won't touch it. - - -libcurl with C++ - - There's basically only one thing to keep in mind when using C++ instead of C - when interfacing libcurl: - - "The Callbacks Must Be Plain C" - - So if you want a write callback set in libcurl, you should put it within - 'extern'. Similar to this: - - extern "C" { - size_t write_data(void *ptr, size_t size, size_t nmemb, - void *ourpointer) - { - /* do what you want with the data */ - } - } - - This will of course effectively turn the callback code into C. There won't be - any "this" pointer available etc. - - -Proxies - - What "proxy" means according to Merriam-Webster: "a person authorized to act - for another" but also "the agency, function, or office of a deputy who acts - as a substitute for another". - - Proxies are exceedingly common these days. Companies often only offer - Internet access to employees through their HTTP proxies. Network clients or - user-agents ask the proxy for documents, the proxy does the actual request - and then it returns them. - - libcurl has full support for HTTP proxies, so when a given URL is wanted, - libcurl will ask the proxy for it instead of trying to connect to the actual - host identified in the URL. - - The fact that the proxy is a HTTP proxy puts certain restrictions on what can - actually happen. A requested URL that might not be a HTTP URL will be still - be passed to the HTTP proxy to deliver back to libcurl. This happens - transparently, and an application may not need to know. I say "may", because - at times it is very important to understand that all operations over a HTTP - proxy is using the HTTP protocol. For example, you can't invoke your own - custom FTP commands or even proper FTP directory listings. - - Proxy Options - - To tell libcurl to use a proxy at a given port number: - - curl_easy_setopt(easyhandle, CURLOPT_PROXY, "proxy-host.com:8080"); - - Some proxies require user authentication before allowing a request, and - you pass that information similar to this: - - curl_easy_setopt(easyhandle, CURLOPT_PROXYUSERPWD, "user:password"); - - If you want to, you can specify the host name only in the CURLOPT_PROXY - option, and set the port number separately with CURLOPT_PROXYPORT. - - Environment Variables - - libcurl automatically checks and uses a set of environment variables to - know what proxies to use for certain protocols. The names of the variables - are following an ancient de facto standard and are built up as - "[protocol]_proxy" (note the lower casing). Which makes the variable - 'http_proxy' checked for a name of a proxy to use when the input URL is - HTTP. Following the same rule, the variable named 'ftp_proxy' is checked - for FTP URLs. Again, the proxies are always HTTP proxies, the different - names of the variables simply allows different HTTP proxies to be used. - - The proxy environment variable contents should be in the format - "[protocol://][user:password]machine[:port]". Where the protocol:// part - is simply ignored if present (so http://proxy and bluerk://proxy will do - the same) and the optional port number specifies on which port the proxy - operates on the host. If not specified, the internal default port number - will be used and that is most likely *not* the one you would like it to - be. - - There are two special environment variables. 'all_proxy' is what sets - proxy for any URL in case the protocol specific variable wasn't set, and - 'no_proxy' defines a list of hosts that should not use a proxy even though - a variable may say so. If 'no_proxy' is a plain asterisk ("*") it matches - all hosts. - - SSL and Proxies - - SSL is for secure point-to-point connections. This involves strong - encryption and similar things, which effectively makes it impossible for a - proxy to operate as a "man in between" which the proxy's task is, as - previously discussed. Instead, the only way to have SSL work over a HTTP - proxy is to ask the proxy to tunnel trough everything without being able - to check or fiddle with the traffic. - - Opening an SSL connection over a HTTP proxy is therefor a matter of asking - the proxy for a straight connection to the target host on a specified - port. This is made with the HTTP request CONNECT. ("please mr proxy, - connect me to that remote host"). - - Because of the nature of this operation, where the proxy has no idea what - kind of data that is passed in and out through this tunnel, this breaks - some of the very few advantages that come from using a proxy, such as - caching. Many organizations prevent this kind of tunneling to other - destination port numbers than 443 (which is the default HTTPS port - number). - - Tunneling Through Proxy - - As explained above, tunneling is required for SSL to work and often even - restricted to the operation intended for SSL; HTTPS. - - This is however not the only time proxy-tunneling might offer benefits to - you or your application. - - As tunneling opens a direct connection from your application to the remote - machine, it suddenly also re-introduces the ability to do non-HTTP - operations over a HTTP proxy. You can in fact use things such as FTP - upload or FTP custom commands this way. - - Again, this is often prevented by the administrators of proxies and is - rarely allowed. - - Tell libcurl to use proxy tunneling like this: - - curl_easy_setopt(easyhandle, CURLOPT_HTTPPROXYTUNNEL, TRUE); - - In fact, there might even be times when you want to do plain HTTP - operations using a tunnel like this, as it then enables you to operate on - the remote server instead of asking the proxy to do so. libcurl will not - stand in the way for such innovative actions either! - - Proxy Auto-Config - - Netscape first came up with this. It is basically a web page (usually - using a .pac extension) with a javascript that when executed by the - browser with the requested URL as input, returns information to the - browser on how to connect to the URL. The returned information might be - "DIRECT" (which means no proxy should be used), "PROXY host:port" (to tell - the browser where the proxy for this particular URL is) or "SOCKS - host:port" (to direct the browser to a SOCKS proxy). - - libcurl has no means to interpret or evaluate javascript and thus it - doesn't support this. If you get yourself in a position where you face - this nasty invention, the following advice have been mentioned and used in - the past: - - - Depending on the javascript complexity, write up a script that - translates it to another language and execute that. - - - Read the javascript code and rewrite the same logic in another language. - - - Implement a javascript interpreted, people have successfully used the - Mozilla javascript engine in the past. - - - Ask your admins to stop this, for a static proxy setup or similar. - - -Persistence Is The Way to Happiness - - Re-cycling the same easy handle several times when doing multiple requests is - the way to go. - - After each single curl_easy_perform() operation, libcurl will keep the - connection alive and open. A subsequent request using the same easy handle to - the same host might just be able to use the already open connection! This - reduces network impact a lot. - - Even if the connection is dropped, all connections involving SSL to the same - host again, will benefit from libcurl's session ID cache that drastically - reduces re-connection time. - - FTP connections that are kept alive saves a lot of time, as the command- - response round-trips are skipped, and also you don't risk getting blocked - without permission to login again like on many FTP servers only allowing N - persons to be logged in at the same time. - - libcurl caches DNS name resolving results, to make lookups of a previously - looked up name a lot faster. - - Other interesting details that improve performance for subsequent requests - may also be added in the future. - - Each easy handle will attempt to keep the last few connections alive for a - while in case they are to be used again. You can set the size of this "cache" - with the CURLOPT_MAXCONNECTS option. Default is 5. It is very seldom any - point in changing this value, and if you think of changing this it is often - just a matter of thinking again. - - When the connection cache gets filled, libcurl must close an existing - connection in order to get room for the new one. To know which connection to - close, libcurl uses a "close policy" that you can affect with the - CURLOPT_CLOSEPOLICY option. There's only two polices implemented as of this - writing (libcurl 7.9.4) and they are: - - CURLCLOSEPOLICY_LEAST_RECENTLY_USED simply close the one that hasn't been - used for the longest time. This is the default behavior. - - CURLCLOSEPOLICY_OLDEST closes the oldest connection, the one that was - created the longest time ago. - - There are, or at least were, plans to support a close policy that would call - a user-specified callback to let the user be able to decide which connection - to dump when this is necessary and therefor is the CURLOPT_CLOSEFUNCTION an - existing option still today. Nothing ever uses this though and this will not - be used within the foreseeable future either. - - To force your upcoming request to not use an already existing connection (it - will even close one first if there happens to be one alive to the same host - you're about to operate on), you can do that by setting CURLOPT_FRESH_CONNECT - to TRUE. In a similar spirit, you can also forbid the upcoming request to be - "lying" around and possibly get re-used after the request by setting - CURLOPT_FORBID_REUSE to TRUE. - - -HTTP Headers Used by libcurl - - When you use libcurl to do HTTP requests, it'll pass along a series of - headers automatically. It might be good for you to know and understand these - ones. - - Host - - This header is required by HTTP 1.1 and even many 1.0 servers and should - be the name of the server we want to talk to. This includes the port - number if anything but default. - - Pragma - - "no-cache". Tells a possible proxy to not grab a copy from the cache but - to fetch a fresh one. - - Accept: - - "image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*". Cloned from a - browser once a hundred years ago. - - Expect: - - When doing multi-part formposts, libcurl will set this header to - "100-continue" to ask the server for an "OK" message before it proceeds - with sending the data part of the post. - - -Customizing Operations - - There is an ongoing development today where more and more protocols are built - upon HTTP for transport. This has obvious benefits as HTTP is a tested and - reliable protocol that is widely deployed and have excellent proxy-support. - - When you use one of these protocols, and even when doing other kinds of - programming you may need to change the traditional HTTP (or FTP or...) - manners. You may need to change words, headers or various data. - - libcurl is your friend here too. - - CUSTOMREQUEST - - If just changing the actual HTTP request keyword is what you want, like - when GET, HEAD or POST is not good enough for you, CURLOPT_CUSTOMREQUEST - is there for you. It is very simple to use: - - curl_easy_setopt(easyhandle, CURLOPT_CUSTOMREQUEST, "MYOWNRUQUEST"); - - When using the custom request, you change the request keyword of the - actual request you are performing. Thus, by default you make GET request - but you can also make a POST operation (as described before) and then - replace the POST keyword if you want to. You're the boss. - - Modify Headers - - HTTP-like protocols pass a series of headers to the server when doing the - request, and you're free to pass any amount of extra headers that you - think fit. Adding headers are this easy: - - struct curl_slist *headers=NULL; /* init to NULL is important */ - - headers = curl_slist_append(headers, "Hey-server-hey: how are you?"); - headers = curl_slist_append(headers, "X-silly-content: yes"); - - /* pass our list of custom made headers */ - curl_easy_setopt(easyhandle, CURLOPT_HTTPHEADER, headers); - - curl_easy_perform(easyhandle); /* transfer http */ - - curl_slist_free_all(headers); /* free the header list */ - - ... and if you think some of the internally generated headers, such as - Accept: or Host: don't contain the data you want them to contain, you can - replace them by simply setting them too: - - headers = curl_slist_append(headers, "Accept: Agent-007"); - headers = curl_slist_append(headers, "Host: munged.host.line"); - - Delete Headers - - If you replace an existing header with one with no contents, you will - prevent the header from being sent. Like if you want to completely prevent - the "Accept:" header to be sent, you can disable it with code similar to - this: - - headers = curl_slist_append(headers, "Accept:"); - - Both replacing and canceling internal headers should be done with careful - consideration and you should be aware that you may violate the HTTP - protocol when doing so. - - Enforcing chunked transfer-encoding - - By making sure a request uses the custom header "Transfer-Encoding: - chunked" when doing a non-GET HTTP operation, libcurl will switch over to - "chunked" upload, even though the size of the data to upload might be - known. By default, libcurl usually switches over to chunked upload - automatically if the upload data size is unknown. - - HTTP Version - - There's only one aspect left in the HTTP requests that we haven't yet - mentioned how to modify: the version field. All HTTP requests includes the - version number to tell the server which version we support. libcurl speak - HTTP 1.1 by default. Some very old servers don't like getting 1.1-requests - and when dealing with stubborn old things like that, you can tell libcurl - to use 1.0 instead by doing something like this: - - curl_easy_setopt(easyhandle, CURLOPT_HTTP_VERSION, - CURLHTTP_VERSION_1_0); - - FTP Custom Commands - - Not all protocols are HTTP-like, and thus the above may not help you when - you want to make for example your FTP transfers to behave differently. - - Sending custom commands to a FTP server means that you need to send the - commands exactly as the FTP server expects them (RFC959 is a good guide - here), and you can only use commands that work on the control-connection - alone. All kinds of commands that requires data interchange and thus needs - a data-connection must be left to libcurl's own judgment. Also be aware - that libcurl will do its very best to change directory to the target - directory before doing any transfer, so if you change directory (with CWD - or similar) you might confuse libcurl and then it might not attempt to - transfer the file in the correct remote directory. - - A little example that deletes a given file before an operation: - - headers = curl_slist_append(headers, "DELE file-to-remove"); - - /* pass the list of custom commands to the handle */ - curl_easy_setopt(easyhandle, CURLOPT_QUOTE, headers); - - curl_easy_perform(easyhandle); /* transfer ftp data! */ - - curl_slist_free_all(headers); /* free the header list */ - - If you would instead want this operation (or chain of operations) to - happen _after_ the data transfer took place the option to - curl_easy_setopt() would instead be called CURLOPT_POSTQUOTE and used the - exact same way. - - The custom FTP command will be issued to the server in the same order they - are added to the list, and if a command gets an error code returned back - from the server, no more commands will be issued and libcurl will bail out - with an error code (CURLE_FTP_QUOTE_ERROR). Note that if you use - CURLOPT_QUOTE to send commands before a transfer, no transfer will - actually take place when a quote command has failed. - - If you set the CURLOPT_HEADER to true, you will tell libcurl to get - information about the target file and output "headers" about it. The - headers will be in "HTTP-style", looking like they do in HTTP. - - The option to enable headers or to run custom FTP commands may be useful - to combine with CURLOPT_NOBODY. If this option is set, no actual file - content transfer will be performed. - - FTP Custom CUSTOMREQUEST - - If you do what list the contents of a FTP directory using your own defined - FTP command, CURLOPT_CUSTOMREQUEST will do just that. "NLST" is the - default one for listing directories but you're free to pass in your idea - of a good alternative. - - -Cookies Without Chocolate Chips - - In the HTTP sense, a cookie is a name with an associated value. A server - sends the name and value to the client, and expects it to get sent back on - every subsequent request to the server that matches the particular conditions - set. The conditions include that the domain name and path match and that the - cookie hasn't become too old. - - In real-world cases, servers send new cookies to replace existing one to - update them. Server use cookies to "track" users and to keep "sessions". - - Cookies are sent from server to clients with the header Set-Cookie: and - they're sent from clients to servers with the Cookie: header. - - To just send whatever cookie you want to a server, you can use CURLOPT_COOKIE - to set a cookie string like this: - - curl_easy_setopt(easyhandle, CURLOPT_COOKIE, "name1=var1; name2=var2;"); - - In many cases, that is not enough. You might want to dynamically save - whatever cookies the remote server passes to you, and make sure those cookies - are then use accordingly on later requests. - - One way to do this, is to save all headers you receive in a plain file and - when you make a request, you tell libcurl to read the previous headers to - figure out which cookies to use. Set header file to read cookies from with - CURLOPT_COOKIEFILE. - - The CURLOPT_COOKIEFILE option also automatically enables the cookie parser in - libcurl. Until the cookie parser is enabled, libcurl will not parse or - understand incoming cookies and they will just be ignored. However, when the - parser is enabled the cookies will be understood and the cookies will be kept - in memory and used properly in subsequent requests when the same handle is - used. Many times this is enough, and you may not have to save the cookies to - disk at all. Note that the file you specify to CURLOPT_COOKIEFILE doesn't - have to exist to enable the parser, so a common way to just enable the parser - and not read able might be to use a file name you know doesn't exist. - - If you rather use existing cookies that you've previously received with your - Netscape or Mozilla browsers, you can make libcurl use that cookie file as - input. The CURLOPT_COOKIEFILE is used for that too, as libcurl will - automatically find out what kind of file it is and act accordingly. - - The perhaps most advanced cookie operation libcurl offers, is saving the - entire internal cookie state back into a Netscape/Mozilla formatted cookie - file. We call that the cookie-jar. When you set a file name with - CURLOPT_COOKIEJAR, that file name will be created and all received cookies - will be stored in it when curl_easy_cleanup() is called. This enabled cookies - to get passed on properly between multiple handles without any information - getting lost. - - -FTP Peculiarities We Need - - FTP transfers use a second TCP/IP connection for the data transfer. This is - usually a fact you can forget and ignore but at times this fact will come - back to haunt you. libcurl offers several different ways to custom how the - second connection is being made. - - libcurl can either connect to the server a second time or tell the server to - connect back to it. The first option is the default and it is also what works - best for all the people behind firewalls, NATs or IP-masquerading setups. - libcurl then tells the server to open up a new port and wait for a second - connection. This is by default attempted with EPSV first, and if that doesn't - work it tries PASV instead. (EPSV is an extension to the original FTP spec - and does not exist nor work on all FTP servers.) - - You can prevent libcurl from first trying the EPSV command by setting - CURLOPT_FTP_USE_EPSV to FALSE. - - In some cases, you will prefer to have the server connect back to you for the - second connection. This might be when the server is perhaps behind a firewall - or something and only allows connections on a single port. libcurl then - informs the remote server which IP address and port number to connect to. - This is made with the CURLOPT_FTPPORT option. If you set it to "-", libcurl - will use your system's "default IP address". If you want to use a particular - IP, you can set the full IP address, a host name to resolve to an IP address - or even a local network interface name that libcurl will get the IP address - from. - - When doing the "PORT" approach, libcurl will attempt to use the EPRT and the - LPRT before trying PORT, as they work with more protocols. You can disable - this behavior by setting CURLOPT_FTP_USE_EPRT to FALSE. - - -Headers Equal Fun - - Some protocols provide "headers", meta-data separated from the normal - data. These headers are by default not included in the normal data stream, - but you can make them appear in the data stream by setting CURLOPT_HEADER to - TRUE. - - What might be even more useful, is libcurl's ability to separate the headers - from the data and thus make the callbacks differ. You can for example set a - different pointer to pass to the ordinary write callback by setting - CURLOPT_WRITEHEADER. - - Or, you can set an entirely separate function to receive the headers, by - using CURLOPT_HEADERFUNCTION. - - The headers are passed to the callback function one by one, and you can - depend on that fact. It makes it easier for you to add custom header parsers - etc. - - "Headers" for FTP transfers equal all the FTP server responses. They aren't - actually true headers, but in this case we pretend they are! ;-) - - -Post Transfer Information - - [ curl_easy_getinfo ] - - -Security Considerations - - libcurl is in itself not insecure. If used the right way, you can use libcurl - to transfer data pretty safely. - - There are of course many things to consider that may loosen up this - situation: - - Command Lines - - If you use a command line tool (such as curl) that uses libcurl, and you - give option to the tool on the command line those options can very likely - get read by other users of your system when they use 'ps' or other tools - to list currently running processes. - - To avoid this problem, never feed sensitive things to programs using - command line options. - - .netrc - - .netrc is a pretty handy file/feature that allows you to login quickly and - automatically to frequently visited sites. The file contains passwords in - clear text and is a real security risk. In some cases, your .netrc is also - stored in a home directory that is NFS mounted or used on another network - based file system, so the clear text password will fly through your - network every time anyone reads that file! - - To avoid this problem, don't use .netrc files and never store passwords in - plain text anywhere. - - Clear Text Passwords - - Many of the protocols libcurl supports send name and password unencrypted - as clear text (HTTP Basic authentication, FTP, TELNET etc). It is very - easy for anyone on your network or a network nearby yours, to just fire up - a network analyzer tool and eavesdrop on your passwords. Don't let the - fact that HTTP uses base64 encoded passwords fool you. They may not look - readable at a first glance, but they very easily "deciphered" by anyone - within seconds. - - To avoid this problem, use protocols that don't let snoopers see your - password: HTTPS, FTPS and FTP-kerberos are a few examples. HTTP Digest - authentication allows this too, but isn't supported by libcurl as of this - writing. - - Showing What You Do - - On a related issue, be aware that even in situations like when you have - problems with libcurl and ask someone for help, everything you reveal in - order to get best possible help might also impose certain security related - risks. Host names, user names, paths, operating system specifics etc (not - to mention passwords of course) may in fact be used by intruders to gain - additional information of a potential target. - - To avoid this problem, you must of course use your common sense. Often, - you can just edit out the sensitive data or just search/replace your true - information with faked data. - - -Multiple Transfers Using the multi Interface - - The easy interface as described in detail in this document is a synchronous - interface that transfers one file at a time and doesn't return until its - done. - - The multi interface on the other hand, allows your program to transfer - multiple files in both directions at the same time, without forcing you to - use multiple threads. - - To use this interface, you are better off if you first understand the basics - of how to use the easy interface. The multi interface is simply a way to make - multiple transfers at the same time, by adding up multiple easy handles in to - a "multi stack". - - You create the easy handles you want and you set all the options just like - you have been told above, and then you create a multi handle with - curl_multi_init() and add all those easy handles to that multi handle with - curl_multi_add_handle(). - - When you've added the handles you have for the moment (you can still add new - ones at any time), you start the transfers by call curl_multi_perform(). - - curl_multi_perform() is asynchronous. It will only execute as little as - possible and then return back control to your program. It is designed to - never block. If it returns CURLM_CALL_MULTI_PERFORM you better call it again - soon, as that is a signal that it still has local data to send or remote data - to receive. - - The best usage of this interface is when you do a select() on all possible - file descriptors or sockets to know when to call libcurl again. This also - makes it easy for you to wait and respond to actions on your own - application's sockets/handles. You figure out what to select() for by using - curl_multi_fdset(), that fills in a set of fd_set variables for you with the - particular file descriptors libcurl uses for the moment. - - When you then call select(), it'll return when one of the file handles signal - action and you then call curl_multi_perform() to allow libcurl to do what it - wants to do. Take note that libcurl does also feature some time-out code so - we advice you to never use very long timeouts on select() before you call - curl_multi_perform(), which thus should be called unconditionally every now - and then even if none of its file descriptors have signaled ready. Another - precaution you should use: always call curl_multi_fdset() immediately before - the select() call since the current set of file descriptors may change when - calling a curl function. - - If you want to stop the transfer of one of the easy handles in the stack, you - can use curl_multi_remove_handle() to remove individual easy - handles. Remember that easy handles should be curl_easy_cleanup()ed. - - When a transfer within the multi stack has finished, the counter of running - transfers (as filled in by curl_multi_perform()) will decrease. When the - number reaches zero, all transfers are done. - - curl_multi_info_read() can be used to get information about completed - transfers. It then returns the CURLcode for each easy transfer, to allow you - to figure out success on each individual transfer. - - -SSL, Certificates and Other Tricks - - [ seeding, passwords, keys, certificates, ENGINE, ca certs ] - - -Sharing Data Between Easy Handles - - [ fill in ] - ------ -Footnotes: - -[1] = libcurl 7.10.3 and later have the ability to switch over to chunked - Transfer-Encoding in cases were HTTP uploads are done with data of an - unknown size. - -[2] = This happens on Windows machines when libcurl is built and used as a - DLL. However, you can still do this on Windows if you link with a static - library. - -[3] = The curl-config tool is generated at build-time (on unix-like systems) - and should be installed with the 'make install' or similar instruction - that installs the library, header files, man pages etc.