TSDuck v3.38-3675
MPEG Transport Stream Toolkit
Loading...
Searching...
No Matches
ts::WebRequest Class Reference

Perform a simple Web request (HTTP, HTTPS, FTP). More...

#include <tsWebRequest.h>

Public Types

using HeadersMap = std::multimap< UString, UString >
 Representation of request or reponse headers.
 

Public Member Functions

 WebRequest (Report &report)
 Constructor.
 
virtual ~WebRequest ()
 Destructor.
 
void abort ()
 Abort a transfer in progress.
 
size_t announdedContentSize () const
 Get the announced content size in bytes.
 
void clearRequestHeaders ()
 Clear all headers which will be sent with the request.
 
bool close ()
 Close the transfer.
 
size_t contentSize () const
 Get the size in bytes of the downloaded content.
 
bool deleteCookiesFile () const
 Delete the cookies file, if one was defined.
 
void disableCookies ()
 Disable the use of cookies for all requests.
 
bool downloadBinaryContent (const UString &url, ByteBlock &data, size_t chunkSize=DEFAULT_CHUNK_SIZE)
 Download the content of the URL as binary data in one operation.
 
bool downloadFile (const UString &url, const fs::path &fileName, size_t chunkSize=DEFAULT_CHUNK_SIZE)
 Download the content of the URL in a file in one operation.
 
bool downloadTextContent (const UString &url, UString &text, size_t chunkSize=DEFAULT_CHUNK_SIZE)
 Download the content of the URL as text in one operation.
 
void enableCompression (bool on=true)
 Enable compression.
 
void enableCookies (const fs::path &fileName=fs::path())
 Enable the use of cookies for all requests using this instance.
 
UString finalURL () const
 Get the final URL of the actual download operation.
 
fs::path getCookiesFileName () const
 Get the file name to use for cookies for all requests using this instance.
 
void getResponseHeaders (HeadersMap &headers) const
 Get all response headers.
 
int httpStatus () const
 Get the HTTP status code (200, 404, etc).
 
bool isOpen () const
 Check if a transfer is open.
 
UString mimeType (bool simple=true, bool lowercase=true) const
 Get the MIME type in the response headers.
 
bool open (const UString &url)
 Open an URL and start the transfer.
 
UString originalURL () const
 Get the original URL, as set by setURL().
 
const UStringproxyHost () const
 Get the current actual proxy host.
 
const UStringproxyPassword () const
 Get the current actual proxy user password.
 
uint16_t proxyPort () const
 Get the current actual proxy port number.
 
const UStringproxyUser () const
 Get the current actual proxy user name.
 
bool receive (void *buffer, size_t maxSize, size_t &retSize)
 Receive data.
 
UString reponseHeader (const UString &name) const
 Get the value of one header.
 
void setArgs (const WebRequestArgs &args)
 Set various arguments from command line.
 
void setAutoRedirect (bool on)
 Enable or disable the automatic redirection of HTTP requests.
 
void setConnectionTimeout (cn::milliseconds timeout)
 Set the connection timeout for this request.
 
void setProxyHost (const UString &host, uint16_t port)
 Set the optional proxy host and port for this request.
 
void setProxyUser (const UString &user, const UString &password)
 Set the optional proxy authentication for this request.
 
void setReceiveTimeout (cn::milliseconds timeout)
 Set the timeout for each receive operation.
 
void setRequestHeader (const UString &name, const UString &value)
 Set a header which will be sent with the request.
 
void setUserAgent (const UString &name=UString())
 Set the user agent name to use in HTTP headers.
 
const UStringuserAgent () const
 Get the current user agent name to use in HTTP headers.
 

Static Public Member Functions

static UString GetLibraryVersion ()
 Get the version of the underlying HTTP library.
 
static void SetDefaultProxyHost (const UString &host, uint16_t port)
 Set the default proxy host and port for all subsequent requests.
 
static void SetDefaultProxyUser (const UString &user, const UString &password)
 Set the default proxy authentication for all subsequent requests.
 

Static Public Attributes

static constexpr size_t DEFAULT_CHUNK_SIZE = 64 * 1024
 Default download chunk size for bulk transfers.
 
static constexpr const UCharDEFAULT_USER_AGENT = u"tsduck"
 Default user agent string ("tsduck").
 

Detailed Description

Perform a simple Web request (HTTP, HTTPS, FTP).

On UNIX systems, the implementation uses libcurl. On Windows systems, the implementation uses Microsoft Wininet. We could have used libcurl on Windows but building it was a pain...

The proxy and transfer settings must be set before starting any download operation. The HTTP status and the response headers are available after a successful download start.

By default, no proxy is used. If no proxy is set, the default proxy is used (system configuration on Windows, http_proxy environment on Unix systems).

Member Typedef Documentation

◆ HeadersMap

using ts::WebRequest::HeadersMap = std::multimap<UString,UString>

Representation of request or reponse headers.

The keys of the map are the header names.

Constructor & Destructor Documentation

◆ WebRequest()

ts::WebRequest::WebRequest ( Report report)

Constructor.

Parameters
[in,out]reportWhere to report errors.

Member Function Documentation

◆ setConnectionTimeout()

void ts::WebRequest::setConnectionTimeout ( cn::milliseconds  timeout)
inline

Set the connection timeout for this request.

Parameters
[in]timeoutConnection timeout in milliseconds.

◆ setReceiveTimeout()

void ts::WebRequest::setReceiveTimeout ( cn::milliseconds  timeout)
inline

Set the timeout for each receive operation.

Parameters
[in]timeoutReception timeout in milliseconds.

◆ setProxyHost()

void ts::WebRequest::setProxyHost ( const UString host,
uint16_t  port 
)

Set the optional proxy host and port for this request.

Parameters
[in]hostProxy host name or address.
[in]portProxy port number.

◆ setProxyUser()

void ts::WebRequest::setProxyUser ( const UString user,
const UString password 
)

Set the optional proxy authentication for this request.

Parameters
[in]userProxy user name.
[in]passwordProxy user's password.

◆ SetDefaultProxyHost()

static void ts::WebRequest::SetDefaultProxyHost ( const UString host,
uint16_t  port 
)
static

Set the default proxy host and port for all subsequent requests.

Parameters
[in]hostProxy host name or address.
[in]portProxy port number.

◆ SetDefaultProxyUser()

static void ts::WebRequest::SetDefaultProxyUser ( const UString user,
const UString password 
)
static

Set the default proxy authentication for all subsequent requests.

Parameters
[in]userProxy user name.
[in]passwordProxy user's password.

◆ proxyHost()

const UString & ts::WebRequest::proxyHost ( ) const

Get the current actual proxy host.

Returns
A constant reference to the proxy host name.

◆ proxyPort()

uint16_t ts::WebRequest::proxyPort ( ) const

Get the current actual proxy port number.

Returns
The proxy port number.

◆ proxyUser()

const UString & ts::WebRequest::proxyUser ( ) const

Get the current actual proxy user name.

Returns
A constant reference to the proxy user name.

◆ proxyPassword()

const UString & ts::WebRequest::proxyPassword ( ) const

Get the current actual proxy user password.

Returns
A constant reference to the proxy user password.

◆ enableCookies()

void ts::WebRequest::enableCookies ( const fs::path &  fileName = fs::path())

Enable the use of cookies for all requests using this instance.

Parameters
[in]fileNameThe name of the file to use to load and store cookies. On Windows, there is an implicit per-user cookie repository and fileName is ignored. On Unix systems, this file is used to store and retrieve cookies in the libcurl format. When fileName is empty, use a temporary file name.

◆ disableCookies()

void ts::WebRequest::disableCookies ( )

Disable the use of cookies for all requests.

Cookies are initially disabled by default.

◆ getCookiesFileName()

fs::path ts::WebRequest::getCookiesFileName ( ) const

Get the file name to use for cookies for all requests using this instance.

  • On Linux, return the current cookie file name, possibly the name of a temporary file if EnableCookies() was called with an empty string.
  • On Windows, the cookie repository is defined per user. There is no specific per-application file and this method always report an empty string.
    Returns
    The cookie file name.

◆ deleteCookiesFile()

bool ts::WebRequest::deleteCookiesFile ( ) const

Delete the cookies file, if one was defined.

Returns
True on success, false on error.

◆ setUserAgent()

void ts::WebRequest::setUserAgent ( const UString name = UString())
inline

Set the user agent name to use in HTTP headers.

Parameters
[in]nameThe user agent name. If empty, DEFAULT_USER_AGENT is used.

◆ enableCompression()

void ts::WebRequest::enableCompression ( bool  on = true)
inline

Enable compression.

Compression is disabled by default.

Parameters
[in]onBoolean setting compression on or off.

◆ userAgent()

const UString & ts::WebRequest::userAgent ( ) const
inline

Get the current user agent name to use in HTTP headers.

Returns
A constant reference to the user agent name to use in HTTP headers.

◆ setAutoRedirect()

void ts::WebRequest::setAutoRedirect ( bool  on)
inline

Enable or disable the automatic redirection of HTTP requests.

This option is active by default.

Parameters
[in]onIf true, allow automatic redirection of HTTP requests.

◆ setArgs()

void ts::WebRequest::setArgs ( const WebRequestArgs args)

Set various arguments from command line.

Parameters
[in]argsCommand line arguments.

◆ setRequestHeader()

void ts::WebRequest::setRequestHeader ( const UString name,
const UString value 
)

Set a header which will be sent with the request.

Parameters
[in]nameThe header name.
[in]valueThe header value.

◆ open()

bool ts::WebRequest::open ( const UString url)

Open an URL and start the transfer.

For HTTP request, perform all redirections and get response headers.

Parameters
[in]urlThe complete URL to fetch.
Returns
True on success, false on error.

◆ isOpen()

bool ts::WebRequest::isOpen ( ) const
inline

Check if a transfer is open.

Returns
True if a transfer is open, false otherwise.

◆ httpStatus()

int ts::WebRequest::httpStatus ( ) const
inline

Get the HTTP status code (200, 404, etc).

Returns
The HTTP status code.

◆ announdedContentSize()

size_t ts::WebRequest::announdedContentSize ( ) const
inline

Get the announced content size in bytes.

This is the value which was sent in the content headers. This mat be zero, this may not be the actual size of the content to download.

Returns
Announced content size in bytes.

◆ getResponseHeaders()

void ts::WebRequest::getResponseHeaders ( HeadersMap headers) const

Get all response headers.

Parameters
[out]headersA multimap of all response headers.

◆ reponseHeader()

UString ts::WebRequest::reponseHeader ( const UString name) const

Get the value of one header.

Parameters
[in]nameHeader name, case sensitive.
Returns
Header value or an empty string when the header is not found. If the header is present more than once, the first value is returned.

◆ mimeType()

UString ts::WebRequest::mimeType ( bool  simple = true,
bool  lowercase = true 
) const

Get the MIME type in the response headers.

Parameters
[in]simpleIf true, simple type name. If false, return the full specification with options.
[in]lowercaseForce lowercase in the result.
Returns
The MIME type.

◆ originalURL()

UString ts::WebRequest::originalURL ( ) const
inline

Get the original URL, as set by setURL().

Returns
The original URL.

◆ finalURL()

UString ts::WebRequest::finalURL ( ) const
inline

Get the final URL of the actual download operation.

It can be different from originalURL() if some HTTP redirections were performed. When called before a download operation, return originalURL().

If redirections are disabled using setAutoRedirect() and the site returned a redirection, finalURL() returns the redirected URL.

Returns
The final / redirected URL.

◆ receive()

bool ts::WebRequest::receive ( void *  buffer,
size_t  maxSize,
size_t &  retSize 
)

Receive data.

Parameters
[out]bufferAddress of the buffer for the received data.
[in]maxSizeSize in bytes of the reception buffer.
[out]retSizeSize in bytes of the received data. Will never be larger than max_size. When ret_size is zero, this is the end of the transfer.
Returns
True on success, false on error. A successful end of transfer is reported when ret_size is zero and the returned value is true.

◆ close()

bool ts::WebRequest::close ( )

Close the transfer.

Returns
True on success, false on error.

◆ abort()

void ts::WebRequest::abort ( )

Abort a transfer in progress.

Can be called from another thread.

◆ contentSize()

size_t ts::WebRequest::contentSize ( ) const
inline

Get the size in bytes of the downloaded content.

Returns
Size in bytes of the downloaded content.

◆ downloadBinaryContent()

bool ts::WebRequest::downloadBinaryContent ( const UString url,
ByteBlock data,
size_t  chunkSize = DEFAULT_CHUNK_SIZE 
)

Download the content of the URL as binary data in one operation.

The open/read/close session is embedded in this method.

Parameters
[in]urlThe complete URL to fetch.
[out]dataThe content of the URL.
[in]chunkSizeIndividual download chunk size.
Returns
True on success, false on error.

◆ downloadTextContent()

bool ts::WebRequest::downloadTextContent ( const UString url,
UString text,
size_t  chunkSize = DEFAULT_CHUNK_SIZE 
)

Download the content of the URL as text in one operation.

The open/read/close session is embedded in this method.. The downloaded text is converted from UTF-8. End of lines are normalized as LF.

Parameters
[in]urlThe complete URL to fetch.
[out]textThe content of the URL.
[in]chunkSizeIndividual download chunk size.
Returns
True on success, false on error.

◆ downloadFile()

bool ts::WebRequest::downloadFile ( const UString url,
const fs::path &  fileName,
size_t  chunkSize = DEFAULT_CHUNK_SIZE 
)

Download the content of the URL in a file in one operation.

The open/read/close session is embedded in this method.. No transformation is applied to the data.

Parameters
[in]urlThe complete URL to fetch.
[in]fileNameName of the file to create.
[in]chunkSizeIndividual download chunk size.
Returns
True on success, false on error.

◆ GetLibraryVersion()

static UString ts::WebRequest::GetLibraryVersion ( )
static

Get the version of the underlying HTTP library.

Returns
The library version.

The documentation for this class was generated from the following file: