Perform a simple Web request (HTTP, HTTPS, FTP). More...

#include <tsWebRequest.h>

Inheritance diagram for ts::WebRequest:

Public Member Functions
	WebRequest (Report &report)
	Constructor.

virtual	~WebRequest ()
	Destructor.

void	abort ()
	Abort a transfer in progress.

size_t	announdedContentSize () const
	Get the announced content size in bytes.

void	clearPostData ()
	Clear previous POST data.

void	clearRequestHeaders ()
	Clear all headers which will be sent with the request.

bool	close ()
	Close the transfer.

size_t	contentSize () const
	Get the size in bytes of the downloaded content.

bool	deleteCookiesFile () const
	Delete the cookies file, if one was defined.

void	deleteRequestHeader (const UString &name)
	Delete all headers with a given name.

void	disableCookies ()
	Disable the use of cookies for all requests.

bool	downloadBinaryContent (const UString &url, ByteBlock &data, size_t chunkSize=DEFAULT_CHUNK_SIZE)
	Download the content of the URL as binary data in one operation.

bool	downloadFile (const UString &url, const fs::path &fileName, size_t chunkSize=DEFAULT_CHUNK_SIZE)
	Download the content of the URL in a file in one operation.

bool	downloadTextContent (const UString &url, UString &text, size_t chunkSize=DEFAULT_CHUNK_SIZE)
	Download the content of the URL as text in one operation.

void	enableCompression (bool on=true)
	Enable compression.

void	enableCookies (const fs::path &fileName=fs::path())
	Enable the use of cookies for all requests using this instance.

UString	finalURL () const
	Get the final URL of the actual download operation.

fs::path	getCookiesFileName () const
	Get the file name to use for cookies for all requests using this instance.

void	getResponseHeaders (UStringToUStringMultiMap &headers) const
	Get all response headers.

int	httpStatus () const
	Get the HTTP status code (200, 404, etc).

int	httpSuccess () const
	Check if the HTTP status code indicates success.

bool	isOpen () const
	Check if a transfer is open.

UString	mimeType (bool simple=true, bool lowercase=true) const
	Get the MIME type in the response headers.

bool	open (const UString &url)
	Open an URL and start the transfer.

UString	originalURL () const
	Get the original URL, as set by setURL().

const UString &	proxyHost () const
	Get the current actual proxy host.

const UString &	proxyPassword () const
	Get the current actual proxy user password.

uint16_t	proxyPort () const
	Get the current actual proxy port number.

const UString &	proxyUser () const
	Get the current actual proxy user name.

bool	receive (void *buffer, size_t maxSize, size_t &retSize)
	Receive data.

UString	reponseHeader (const UString &name) const
	Get the value of one header.

Report &	report ()
	Use the Report object of this instance.

const UStringToUStringMultiMap &	responseHeaders () const
	Get all response headers.

void	setArgs (const WebRequestArgs &args)
	Set various arguments from command line.

void	setAutoRedirect (bool on)
	Enable or disable the automatic redirection of HTTP requests.

void	setConnectionTimeout (cn::milliseconds timeout)
	Set the connection timeout for this request.

void	setInsecure (bool on=true)
	Enable or disable HTTPS security (certificate validation).

void	setPostData (const ByteBlock &data)
	Set data to POST.

void	setPostData (const UString &data, const UString content_type=u"text/plain; charset=utf-8")
	Set data to POST.

void	setProxyHost (const UString &host, uint16_t port)
	Set the optional proxy host and port for this request.

void	setProxyUser (const UString &user, const UString &password)
	Set the optional proxy authentication for this request.

void	setReceiveTimeout (cn::milliseconds timeout)
	Set the timeout for each receive operation.

void	setRequestHeader (const UString &name, const UString &value)
	Set a header which will be sent with the request.

void	setUserAgent (const UString &name=UString())
	Set the user agent name to use in HTTP headers.

const UString &	userAgent () const
	Get the current user agent name to use in HTTP headers.

Static Public Member Functions
static UString	GetLibraryVersion ()
	Get the version of the underlying HTTP library.

static void	SetDefaultProxyHost (const UString &host, uint16_t port)
	Set the default proxy host and port for all subsequent requests.

static void	SetDefaultProxyUser (const UString &user, const UString &password)
	Set the default proxy authentication for all subsequent requests.

Static Public Attributes
static constexpr size_t	DEFAULT_CHUNK_SIZE = 64 * 1024
	Default download chunk size for bulk transfers.

static constexpr uint16_t	DEFAULT_HTTP_PORT = 80
	Default TCP port for HTTP.

static constexpr uint16_t	DEFAULT_HTTPS_PORT = 443
	Default TCP port for HTTPS.

static constexpr const UChar *	DEFAULT_USER_AGENT = u"tsduck"
	Default user agent string ("tsduck").

Detailed Description

Perform a simple Web request (HTTP, HTTPS, FTP).

On UNIX systems, the implementation uses libcurl. On Windows systems, the implementation uses Microsoft Wininet. We could have used libcurl on Windows but building it was a pain...

The proxy and transfer settings must be set before starting any download operation. The HTTP status and the response headers are available after a successful download start.

By default, no proxy is used. If no proxy is set, the default proxy is used (system configuration on Windows, http_proxy environment on Unix systems).

Constructor & Destructor Documentation

◆ WebRequest()

ts::WebRequest::WebRequest ( Report & report )

Constructor.

Parameters

[in,out] report Where to report errors.

Member Function Documentation

◆ report()

Report & ts::WebRequest::report ( )

inline

Use the Report object of this instance.

Returns: A reference to the Report object of this instance.

◆ setConnectionTimeout()

void ts::WebRequest::setConnectionTimeout ( cn::milliseconds timeout )

inline

Set the connection timeout for this request.

Parameters

[in] timeout Connection timeout in milliseconds.

◆ setReceiveTimeout()

void ts::WebRequest::setReceiveTimeout ( cn::milliseconds timeout )

inline

Set the timeout for each receive operation.

Parameters

[in] timeout Reception timeout in milliseconds.

◆ setProxyHost()

void ts::WebRequest::setProxyHost	(	const UString &	host,
		uint16_t	port
	)

Set the optional proxy host and port for this request.

Parameters

[in]	host	Proxy host name or address.
[in]	port	Proxy port number.

◆ setProxyUser()

void ts::WebRequest::setProxyUser	(	const UString &	user,
		const UString &	password
	)

Set the optional proxy authentication for this request.

Parameters

[in]	user	Proxy user name.
[in]	password	Proxy user's password.

◆ SetDefaultProxyHost()

static void ts::WebRequest::SetDefaultProxyHost	(	const UString &	host,
		uint16_t	port
	)

static

Set the default proxy host and port for all subsequent requests.

Parameters

[in]	host	Proxy host name or address.
[in]	port	Proxy port number.

◆ SetDefaultProxyUser()

static void ts::WebRequest::SetDefaultProxyUser	(	const UString &	user,
		const UString &	password
	)

static

Set the default proxy authentication for all subsequent requests.

Parameters

[in]	user	Proxy user name.
[in]	password	Proxy user's password.

◆ proxyHost()

const UString & ts::WebRequest::proxyHost ( ) const

Get the current actual proxy host.

Returns: A constant reference to the proxy host name.

◆ proxyPort()

uint16_t ts::WebRequest::proxyPort ( ) const

Get the current actual proxy port number.

Returns: The proxy port number.

◆ proxyUser()

const UString & ts::WebRequest::proxyUser ( ) const

Get the current actual proxy user name.

Returns: A constant reference to the proxy user name.

◆ proxyPassword()

const UString & ts::WebRequest::proxyPassword ( ) const

Get the current actual proxy user password.

Returns: A constant reference to the proxy user password.

◆ enableCookies()

void ts::WebRequest::enableCookies ( const fs::path & fileName = fs::path() )

Enable the use of cookies for all requests using this instance.

Parameters

[in] fileName The name of the file to use to load and store cookies. On Windows, there is an implicit per-user cookie repository and fileName is ignored. On Unix systems, this file is used to store and retrieve cookies in the libcurl format. When fileName is empty, use a temporary file name.

◆ disableCookies()

void ts::WebRequest::disableCookies ( )

Disable the use of cookies for all requests.

Cookies are initially disabled by default.

◆ getCookiesFileName()

fs::path ts::WebRequest::getCookiesFileName ( ) const

Get the file name to use for cookies for all requests using this instance.

On Linux, return the current cookie file name, possibly the name of a temporary file if EnableCookies() was called with an empty string.
On Windows, the cookie repository is defined per user. There is no specific per-application file and this method always report an empty string.
Returns
The cookie file name.

◆ deleteCookiesFile()

bool ts::WebRequest::deleteCookiesFile ( ) const

Delete the cookies file, if one was defined.

Returns: True on success, false on error.

◆ setUserAgent()

void ts::WebRequest::setUserAgent ( const UString & name = UString() )

inline

Set the user agent name to use in HTTP headers.

Parameters

[in] name The user agent name. If empty, DEFAULT_USER_AGENT is used.

◆ userAgent()

const UString & ts::WebRequest::userAgent ( ) const

inline

Get the current user agent name to use in HTTP headers.

Returns: A constant reference to the user agent name to use in HTTP headers.

◆ enableCompression()

void ts::WebRequest::enableCompression ( bool on = true )

inline

Enable compression.

Compression is disabled by default.

Parameters

[in] on Boolean setting compression on or off.

◆ setInsecure()

void ts::WebRequest::setInsecure ( bool on = true )

inline

Enable or disable HTTPS security (certificate validation).

Certificate validation is enabled by default.

Parameters

[in] on If true, disable certificate validation.

◆ setAutoRedirect()

void ts::WebRequest::setAutoRedirect ( bool on )

inline

Enable or disable the automatic redirection of HTTP requests.

This option is active by default.

Parameters

[in] on If true, allow automatic redirection of HTTP requests.

◆ setArgs()

void ts::WebRequest::setArgs ( const WebRequestArgs & args )

Set various arguments from command line.

Parameters

[in] args Command line arguments.

◆ setRequestHeader()

void ts::WebRequest::setRequestHeader	(	const UString &	name,
		const UString &	value
	)

Set a header which will be sent with the request.

If the same header already exists with another value, a new header is added.

Parameters

[in]	name	The header name.
[in]	value	The header value.

◆ deleteRequestHeader()

void ts::WebRequest::deleteRequestHeader ( const UString & name )

Delete all headers with a given name.

Parameters

[in] name The header name.

◆ setPostData() [1/2]

void ts::WebRequest::setPostData	(	const UString &	data,
		const UString	content_type = `u"text/plain; charset=utf-8"`
	)

Set data to POST.

The request will be a POST one.

Parameters

[in]	data	Text POST data. The text will be sent in UTF-8 format.
[in]	content_type	The content type to set in the request headers. The default "Content-Type" header is "text/plain; charset=utf-8", which is usually appropriate. When set to the empty string, no header is set.

◆ setPostData() [2/2]

void ts::WebRequest::setPostData ( const ByteBlock & data )

Set data to POST.

The request will be a POST one.

Parameters

[in] data Binary POST data.

◆ clearPostData()

void ts::WebRequest::clearPostData ( )

Clear previous POST data.

The request will be a GET one.

◆ open()

bool ts::WebRequest::open ( const UString & url )

Open an URL and start the transfer.

For HTTP request, perform all redirections and get response headers.

Parameters

[in] url The complete URL to fetch.

Returns: True on success, false on error.

◆ isOpen()

bool ts::WebRequest::isOpen ( ) const

inline

Check if a transfer is open.

Returns: True if a transfer is open, false otherwise.

◆ httpStatus()

int ts::WebRequest::httpStatus ( ) const

inline

Get the HTTP status code (200, 404, etc).

Returns: The HTTP status code.

◆ httpSuccess()

int ts::WebRequest::httpSuccess ( ) const

inline

Check if the HTTP status code indicates success.

The HTTP status codes are classified as follow (Wikipedia):

1xx informational response – the request was received, continuing process
2xx successful – the request was successfully received, understood, and accepted
3xx redirection – further action needs to be taken in order to complete the request
4xx client error – the request contains bad syntax or cannot be fulfilled
5xx server error – the server failed to fulfil an apparently valid request
Returns
True if the HTTP status code indicates success.

◆ announdedContentSize()

size_t ts::WebRequest::announdedContentSize ( ) const

inline

Get the announced content size in bytes.

This is the value which was sent in the content headers. This mat be zero, this may not be the actual size of the content to download.

Returns: Announced content size in bytes.

◆ getResponseHeaders()

void ts::WebRequest::getResponseHeaders ( UStringToUStringMultiMap & headers ) const

inline

Get all response headers.

Parameters

[out] headers A multimap of all response headers.

◆ responseHeaders()

const UStringToUStringMultiMap & ts::WebRequest::responseHeaders ( ) const

inline

Get all response headers.

Returns: A constant reference to a map of response headers.

◆ reponseHeader()

UString ts::WebRequest::reponseHeader ( const UString & name ) const

Get the value of one header.

Parameters

[in] name Header name, case sensitive.

Returns: Header value or an empty string when the header is not found. If the header is present more than once, the first value is returned.

◆ mimeType()

UString ts::WebRequest::mimeType	(	bool	simple = `true`,
		bool	lowercase = `true`
	)		const

Get the MIME type in the response headers.

Parameters

[in]	simple	If true, simple type name. If false, return the full specification with options.
[in]	lowercase	Force lowercase in the result.

Returns: The MIME type.

◆ originalURL()

UString ts::WebRequest::originalURL ( ) const

inline

Get the original URL, as set by setURL().

Returns: The original URL.

◆ finalURL()

UString ts::WebRequest::finalURL ( ) const

inline

Get the final URL of the actual download operation.

It can be different from originalURL() if some HTTP redirections were performed. When called before a download operation, return originalURL().

If redirections are disabled using setAutoRedirect() and the site returned a redirection, finalURL() returns the redirected URL.

Returns: The final / redirected URL.

◆ receive()

bool ts::WebRequest::receive	(	void *	buffer,
		size_t	maxSize,
		size_t &	retSize
	)

Receive data.

Parameters

[out]	buffer	Address of the buffer for the received data.
[in]	maxSize	Size in bytes of the reception buffer.
[out]	retSize	Size in bytes of the received data. Will never be larger than max_size. When ret_size is zero, this is the end of the transfer.

Returns: True on success, false on error. A successful end of transfer is reported when ret_size is zero and the returned value is true.

◆ close()

bool ts::WebRequest::close ( )

Close the transfer.

Returns: True on success, false on error.

◆ abort()

void ts::WebRequest::abort ( )

Abort a transfer in progress.

Can be called from another thread.

◆ contentSize()

size_t ts::WebRequest::contentSize ( ) const

inline

Get the size in bytes of the downloaded content.

Returns: Size in bytes of the downloaded content.

◆ downloadBinaryContent()

bool ts::WebRequest::downloadBinaryContent	(	const UString &	url,
		ByteBlock &	data,
		size_t	chunkSize = `DEFAULT_CHUNK_SIZE`
	)

Download the content of the URL as binary data in one operation.

The open/read/close session is embedded in this method.

Parameters

[in]	url	The complete URL to fetch.
[out]	data	The content of the URL.
[in]	chunkSize	Individual download chunk size.

Returns: True on success, false on error.

◆ downloadTextContent()

bool ts::WebRequest::downloadTextContent	(	const UString &	url,
		UString &	text,
		size_t	chunkSize = `DEFAULT_CHUNK_SIZE`
	)

Download the content of the URL as text in one operation.

The open/read/close session is embedded in this method.. The downloaded text is converted from UTF-8. End of lines are normalized as LF.

Parameters

[in]	url	The complete URL to fetch.
[out]	text	The content of the URL.
[in]	chunkSize	Individual download chunk size.

Returns: True on success, false on error.

◆ downloadFile()

bool ts::WebRequest::downloadFile	(	const UString &	url,
		const fs::path &	fileName,
		size_t	chunkSize = `DEFAULT_CHUNK_SIZE`
	)

Download the content of the URL in a file in one operation.

The open/read/close session is embedded in this method.. No transformation is applied to the data.

Parameters

[in]	url	The complete URL to fetch.
[in]	fileName	Name of the file to create.
[in]	chunkSize	Individual download chunk size.

Returns: True on success, false on error.

◆ GetLibraryVersion()

static UString ts::WebRequest::GetLibraryVersion ( )

static

Get the version of the underlying HTTP library.

Returns: The library version.

The documentation for this class was generated from the following file:

tsWebRequest.h

Public Member Functions

Static Public Member Functions

Static Public Attributes

Detailed Description

Constructor & Destructor Documentation

◆ WebRequest()

Member Function Documentation

◆ report()

◆ setConnectionTimeout()

◆ setReceiveTimeout()

◆ setProxyHost()

◆ setProxyUser()

◆ SetDefaultProxyHost()

◆ SetDefaultProxyUser()

◆ proxyHost()

◆ proxyPort()

◆ proxyUser()

◆ proxyPassword()

◆ enableCookies()

◆ disableCookies()

◆ getCookiesFileName()

◆ deleteCookiesFile()

◆ setUserAgent()

◆ userAgent()

◆ enableCompression()

◆ setInsecure()

◆ setAutoRedirect()

◆ setArgs()

◆ setRequestHeader()

◆ deleteRequestHeader()

◆ setPostData() [1/2]

◆ setPostData() [2/2]

◆ clearPostData()

◆ open()

◆ isOpen()

◆ httpStatus()

◆ httpSuccess()

◆ announdedContentSize()

◆ getResponseHeaders()

◆ responseHeaders()

◆ reponseHeader()

◆ mimeType()

◆ originalURL()

◆ finalURL()

◆ receive()

◆ close()

◆ abort()

◆ contentSize()

◆ downloadBinaryContent()

◆ downloadTextContent()

◆ downloadFile()

◆ GetLibraryVersion()