HTTPClient

class HTTPClient

Bases: ReferenceCount

Handles contacting an HTTP server and retrieving a document. Each HTTPClient object represents a separate context, and stores its own list of cookies, passwords, and certificates; however, a given HTTPClient is capable of making multiple simultaneous requests to the same or different servers.

It is up to the programmer whether one HTTPClient should be used to retrieve all documents, or a separate one should be created each time. There is a default, global HTTPClient available in HTTPClient::get_global_ptr().

Inheritance diagram

enum VerifySSL

enumerator VS_no_verify = 0

Don’t care who we talk to

enumerator VS_no_date_check = 1

Must identify certs, but old, expired certs are OK

enumerator VS_normal = 2

Identify certs and also check expiration dates.

HTTPClient(void)

HTTPClient(HTTPClient const &copy)

void add_direct_host(std::string const &hostname)

Adds the indicated name to the set of hostnames that are connected to directly, without using a proxy. This name may be either a DNS name or an IP address, and it may include the * as a wildcard character.

bool add_preapproved_server_certificate_filename(URLSpec const &url, Filename const &filename)

Adds the certificate defined in the indicated PEM filename as a “pre- approved” certificate for the indicated server, defined by the hostname and port (only) from the given URL.

If the server offers this particular certificate on a secure connection, it will be accepted without question. This is particularly useful for communicating with a server using a known self-signed certificate.

See also the similar add_preapproved_server_certificate_pem(), and the weaker add_preapproved_server_certificate_name().

bool add_preapproved_server_certificate_name(URLSpec const &url, std::string const &name)

Adds the certificate name only, as a “pre-approved” certificate name for the indicated server, defined by the hostname and port (only) from the given URL.

This is a weaker function than add_preapproved_server_certificate_filename(). This checks only the subject name of the certificate, without checking for a particular certificate by key. This means that a variety of server certificates may match the indicated name.

Because this is a weaker verification, it only applies to server certificates that are signed by a recognized certificate authority. Thus, it cannot be used to pre-approve self-signed certificates, but it can be used to accept a server certificate offered by a different hostname than the one in the cert itself.

The certificate name should be formatted in the form type0=value0/type1=value1/type2=…

bool add_preapproved_server_certificate_pem(URLSpec const &url, std::string const &pem)

Adds the certificate defined in the indicated data string, formatted as a PEM block, as a “pre-approved” certificate for the indicated server, defined by the hostname and port (only) from the given URL.

If the server offers this particular certificate on a secure connection, it will be accepted without question. This is particularly useful for communicating with a server using a known self-signed certificate.

See also the similar add_preapproved_server_certificate_filename(), and the weaker add_preapproved_server_certificate_name().

void add_proxy(std::string const &scheme, URLSpec const &proxy)

Adds the indicated proxy host as a proxy for communications on the given scheme. Usually the scheme is “http” or “https”. It may be the empty string to indicate a general proxy. The proxy string may be the empty URL to indicate a direct connection.

std::string base64_decode(std::string const &s)

Implements HTTPAuthorization::base64_decode(). This is provided here just as a convenient place to publish it for access by the scripting language; C++ code should probably use HTTPAuthorization directly.

std::string base64_encode(std::string const &s)

Implements HTTPAuthorization::base64_encode(). This is provided here just as a convenient place to publish it for access by the scripting language; C++ code should probably use HTTPAuthorization directly.

void clear_all_cookies(void)

Removes the all stored cookies from the client.

void clear_all_preapproved_server_certificates(void)

Removes all preapproved server certificates for all servers.

bool clear_cookie(HTTPCookie const &cookie)

Removes the cookie with the matching domain/path/name from the client’s list of cookies. Returns true if it was removed, false if the cookie was not matched.

void clear_direct_host(void)

Resets the set of direct hosts to empty. Subsequent calls to add_direct_host() may be made to build up the list of hosts that do not require a proxy connection.

void clear_preapproved_server_certificates(URLSpec const &url)

Removes all preapproved server certificates for the indicated server and port.

void clear_proxy(void)

Resets the proxy spec to empty. Subsequent calls to add_proxy() may be made to build up the set of proxy servers.

void copy_cookies_from(HTTPClient const &other)

Copies all the cookies from the indicated HTTPClient into this one. Existing cookies in this client are not affected, unless they are shadowed by the new cookies.

std::string const &get_cipher_list(void) const

Returns the set of ciphers as set by set_cipher_list(). See set_cipher_list().

HTTPCookie get_cookie(HTTPCookie const &cookie) const

Looks up and returns the cookie in the client matching the given cookie’s domain/path/name. If there is no matching cookie, returns an empty cookie.

std::string get_direct_host_spec(void) const

Returns the set of hosts that should be connected to directly, without using a proxy, as a semicolon-separated list of hostnames that may contain wildcard characters (“*”).

PointerTo<HTTPChannel> get_document(URLSpec const &url)

Opens the named document for reading. Returns a new HTTPChannel object whether the document is successfully read or not; you can test is_valid() and get_return_code() to determine whether the document was retrieved.

static HTTPClient *get_global_ptr(void)

Returns the default global HTTPClient.

PointerTo<HTTPChannel> get_header(URLSpec const &url)

Like get_document(), except only the header associated with the document is retrieved. This may be used to test for existence of the document; it might also return the size of the document (if the server gives us this information).

HTTPEnum::HTTPVersion get_http_version(void) const

Returns the client’s current setting for HTTP version. See set_http_version().

std::string get_http_version_string(void) const

Returns the current HTTP version setting as a string, e.g. “HTTP/1.0” or “HTTP/1.1”.

std::string get_proxies_for_url(URLSpec const &url) const

Fills up the indicated vector with the list of URLSpec objects, in the order in which they should be tried, that are appropriate proxies to try for the indicated URL. The empty URL is returned for a direct connection.

It is the user’s responsibility to empty this vector before calling this method; otherwise, the proxy URL’s will simply be appended to the existing list.

Returns a semicolon-delimited list of proxies, in the order in which they should be tried, that are appropriate for the indicated URL. The keyword DIRECT indicates a direct connection should be tried.

std::string get_proxy_spec(void) const

Returns the complete set of proxies to use for all schemes. This is a string of the form specified by set_proxy_spec(), above. Note that the string returned by this function may not be exactly the same as the string passed into set_proxy_spec(), since the string is regenerated from the internal storage structures and may therefore be reordered.

bool get_try_all_direct(void) const

Returns whether a failed connection through a proxy will be followed up by a direct connection attempt, false otherwise.

std::string get_username(std::string const &server, std::string const &realm) const

Returns the username:password string set for this server/realm pair, or empty string if nothing has been set. See set_username().

VerifySSL get_verify_ssl(void) const

Returns whether the client will insist on verifying the identity of the servers it connects to via SSL (that is, https). See set_verify_ssl().

bool has_cookie(HTTPCookie const &cookie) const

Returns true if there is a cookie in the client matching the given cookie’s domain/path/name, false otherwise.

static void init_random_seed(void)

This may be called once, presumably at the beginning of an application, to initialize OpenSSL’s random seed. On Windows, it is particularly important to call this at startup if you are going to be performing any https operations or otherwise use encryption, since the Windows algorithm for getting a random seed takes 2-3 seconds at startup, but can take 30 seconds or more after you have opened a 3-D graphics window and started rendering.

There is no harm in calling this method multiple times, or in not calling it at all.

bool load_certificates(Filename const &filename)

Reads the certificate(s) (delimited by —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—–) from the indicated file and makes them known as trusted public keys for validating future connections. Returns true on success, false otherwise.

bool load_client_certificate(void)

Attempts to load the certificate named by set_client_certificate_filename() immediately, and returns true if successful, false otherwise.

Normally this need not be explicitly called, since it will be called automatically if the server requests a certificate, but it may be useful to determine ahead of time if the certificate can be loaded correctly.

PointerTo<HTTPChannel> make_channel(bool persistent_connection)

Returns a new HTTPChannel object that may be used for reading multiple documents using the same connection, for greater network efficiency than calling HTTPClient::get_document() repeatedly (which would force a new connection for each document).

Also, HTTPChannel has some additional, less common interface methods than the basic interface methods that exist on HTTPClient; if you wish to call any of these methods you must first obtain an HTTPChannel.

Pass true for persistent_connection to gain this network efficiency. If, on the other hand, your intention is to use the channel to retrieve only one document, then pass false to inform the server that we will be dropping the connection after the first document.

static HTTPEnum::HTTPVersion parse_http_version_string(std::string const &version)

Matches the string representing a particular HTTP version against any of the known versions and returns the appropriate enumerated value, or HV_other if the version is unknown.

PointerTo<HTTPChannel> post_form(URLSpec const &url, std::string const &body)

Posts form data to a particular URL and retrieves the response. Returns a new HTTPChannel object whether the document is successfully read or not; you can test is_valid() and get_return_code() to determine whether the document was retrieved.

void send_cookies(std::ostream &out, URLSpec const &url)

Writes to the indicated ostream a “Cookie” header line for sending the cookies appropriate to the indicated URL along with an HTTP request. This also removes expired cookies.

void set_cipher_list(std::string const &cipher_list)

Specifies the set of ciphers that are to be made available for SSL connections. This is a string as described in the ciphers(1) man page of the OpenSSL documentation (or see https://www.openssl.org/docs/man1.1.1/man1/ciphers.html ). If this isn’t specified, the default is provided by the Config file. You may also specify “DEFAULT” to use the built-in OpenSSL default value.

void set_client_certificate_filename(Filename const &filename)

Sets the filename of the pem-formatted file that will be read for the client public and private keys if an SSL server requests a certificate. Either this or set_client_certificate_pem() may be used to specify a client certificate.

void set_client_certificate_passphrase(std::string const &passphrase)

Sets the passphrase used to decrypt the private key in the certificate named by set_client_certificate_filename() or set_client_certificate_pem().

void set_client_certificate_pem(std::string const &pem)

Sets the pem-formatted contents of the certificate that will be parsed for the client public and private keys if an SSL server requests a certificate. Either this or set_client_certificate_filename() may be used to specify a client certificate.

void set_cookie(HTTPCookie const &cookie)

Stores the indicated cookie in the client’s list of cookies, as if it had been received from a server.

void set_direct_host_spec(std::string const &direct_host_spec)

Specifies the set of hosts that should be connected to directly, without using a proxy. This is a semicolon-separated list of hostnames that may contain wildcard characters (“*”).

void set_http_version(HTTPEnum::HTTPVersion version)

Specifies the version of HTTP that the client uses to identify itself to the server. The default is HV_11, or HTTP 1.0; you can set this to HV_10 (HTTP 1.0) to request the server use the older interface.

void set_proxy_spec(std::string const &proxy_spec)

Specifies the complete set of proxies to use for all schemes. This is either a semicolon-delimited set of hostname:ports, or a semicolon- delimited set of pairs of the form “scheme=hostname:port”, or a combination. Use the keyword DIRECT, or an empty string, to represent a direct connection. A particular scheme and/or proxy host may be listed more than once. This is a convenience function that can be used in place of explicit calls to add_proxy() for each scheme/proxy pair.

void set_try_all_direct(bool try_all_direct)

If this is set true, then after a connection attempt through a proxy fails, we always try a direct connection, regardless of whether the host is listed on the direct_host_spec list. If this is false, a direct attempt is not made when we have a proxy in effect, even if the proxy fails.

void set_username(std::string const &server, std::string const &realm, std::string const &username)

Specifies the username:password string corresponding to a particular server and/or realm, when demanded by the server. Either or both of the server or realm may be empty; if so, they match anything. Also, the server may be set to the special string "*proxy", which will match any proxy server.

If the username is set to the empty string, this clears the password for the particular server/realm pair.

void set_verify_ssl(HTTPClient::VerifySSL verify_ssl)

Specifies whether the client will insist on verifying the identity of the servers it connects to via SSL (that is, https).

The parameter value is an enumerated type which indicates the level of security to which the client will insist upon.

void write_cookies(std::ostream &out) const

Outputs the complete list of cookies stored on the client, for all domains, including the expired cookies (which will normally not be sent back to a host).