TSDuck Developer Guide

The file tsPlatform.h contains some very low level definitions such as macros defining the environment (processor, compiler, operating system, endianness), byte and bit manipulation, etc.

C and C++ strings are made of 8-bit characters which are notoriously unable to represent international character sets. The usage of std::string with the TSDuck libraries is discouraged in favor of Unicode strings.

The class ts::UString implements Java-like Unicode strings. Each character uses 16 bits of storage. Formally, ts::UString uses UTF-16 representation. This means that all characters from all modern languages can be represented as one single character. Characters from archaic languages may need two UTF-16 values, called a "surrogate pair".

Technically, ts::UString is a subclass of std::u16string. So any operation on standard C++ strings is also available to ts::UString. But many more operations have been added to manipulate Unicode strings.

For consistency, the type ts::UChar is an alias for char16_t. The header file tsUChar.h defines some utility functions on ts::UChar. It also defines constants for most Unicode characters like ts::COLON or more complex ones such as ts::LATIN_CAPITAL_LETTER_A_WITH_ACUTE, among hundredths of others.

Some interesting features in class ts::UString are:

Unicode strings can be converted to and from DVB or ARIB (Japan) strings. Most DVB-defined character sets are implemented (see the classes ts::Charset and ts::DVBCharset) and recognized when a string is read from a descriptor. When a string is serialized into a binary descriptor, the most appropriate DVB character set is used. In practice, a few known DVB character sets are used and, when the string cannot be encoded in any of them, UTF-8 is used (UTF-8 is always a valid DVB character set).

The class ts::ByteBlock represents a raw block of bytes. It is a subclass of std::vector<uint8_t> and consequently benefits from all standard vector operations. It also adds useful methods for data serialization or deserialization in any byte order.

For data serialization or deserialization over arbitrary memory areas, the header file tsMemory.h provides low-level functions to access integer values of 8, 16, 24, 32, 40, 48 and 64 bits in any byte order.

The class ts::Buffer provides a higher-level abstraction layer over a memory area to parse or generate bitstreams. It gives access to data of any bit-size at any bit position, any endianness, either as a continuous stream or seeking at random bit positions.

The principles of the C++ class ts::Buffer were freely inspired by the Java class java.nio.ByteBuffer. There are differences between the two but the main principles are similar.

Its subclass ts::PSIBuffer provides primitives to serialize and deserialize MPEG and DVB structures such as list of descriptors, DVB, ARIB and ATSC strings or "Modified Julian Dates".

The singleton design pattern is simple in theory, but not so simple to implement correctly in practice. The TSCore library encapsulates the implementation difficulties using the two macros TS_SINGLETON() (in a header file) and TS_DEFINE_SINGLETON() (in the corresponding compilation unit).

In practice, the singleton design pattern is a specific case of global static data, variables which are defined outside any function. Using static data can be a nightmare because it is impossible to manage the initialization order of modules in C++.

In the past, how to properly managed singletons and static data was quite difficult, involving dangerous pointer and thread synchronization tricks. This has been abundantly documented by Scott Meyers (see [MEYERS-EFF]).

In modern C++, most of these difficulties have been solved in the language using local static data. In practice, the same tricks are used but they are now hidden inside the compiler and the run-time library.

All TSDuck classes use a consistent error reporting mechanism through the ts::Report abstract class.

This interface defines several levels of severity in the type ts::Severity, ranging from ts::Severity::Debug to ts::Severity::Fatal. Each instance of ts::Report defines which levels of message are reported to the user. This is usually triggered by command-line options such as --verbose or --debug.

Most classes or methods from the TSDuck libraries use a reference to an instance of ts::Report to report messages and errors. The actual reporting object is often built at application level and then propagated to all layers of code.

Some interesting subclasses of ts::Report are:

As a general rule, TSDuck prefers the usage of error reporting interface and error status over exceptions. However, for a limited number of unrecoverable conditions which should never occur in practice, exceptions are used.

All TSDuck exceptions inherit from the superclass ts::Exception. An instance of this exception is able to embed an error message and an optional system error code.

Each specific exception should be a subclass of ts::Exception. Instead of rewriting the subclass code, applications should use the macro TS_DECLARE_EXCEPTION().

An instance of the class ts::Enumeration associates a list of integer or enum values with strings. It can be used to display meaningful strings instead of integer values.

It is even more useful to decode command line arguments. When an option accepts a predefined list of values, the input string can be either an integer value or a name. When it is a name, it can even be abbreviated as long as it is not ambiguous in the corresponding ts::Enumeration. This is transparent for the application which receives the corresponding integer value.

The class ts::Args implements a generic handling of command line arguments.

Each application typically defines its own subclass of ts::Args. A plugin is always a subclass of ts::Args, through the intermediate class ts::Plugin.

A subclass of ts::Args defines the command line syntax and the corresponding help text. The superclass ts::Args automatically parses the command line, reports errors and handle common options such as --help or --version.

The value of command line options can be free strings, integer values or enumeration values. Integer values are recognized in decimal or hexadecimal form (prefix 0x) and thousands separators (‘,’) which are present for clarity are ignored. Enumeration values are handled through ts::Enumeration.

The TSCore library embeds an XML parser and several classes to handle a DOM structure.

See the class ts::xml::Node, the abstract base class of the DOM hierarchy.

The TSCore library embeds a JSON parser and several classes to handle JSON values.

See the class ts::json::Value, the abstract base class of the JSON hierarchy.

The header files tsSysUtils.h, tsFileUtils.h, tsEnvironment.h, declare utility functions on top of the operating system.

With the introduction of C++20, many of these functions have been removed in favor of new standard functions. However, a number of additional features manipulate:

The class ts::Time is a portable implementation of time (both local and UTC time).

Many operations are provided, such as:

TSDuck is heavily multi-threaded. The abstract class ts::Thread manages a thread. To define an actual thread, derive this class and implement the virtual method main().

The class ts::ThreadAttributes contains all mandatory or optional attributes of a thead. An application typically builds a ts::ThreadAttributes object and then creates threads using these attributes.

In earlier versions of TSDuck, synchronization primitives used to be implemented through specific classes (ts::Mutex, ts::Condition). They are now removed and new C++11 classes such as std::mutex and std::condition_variable are used instead.

Note that the C++11 class std::thread is not used. Its API is too limited to be useable in complex environments: it does not allow to customize the priority or the stack size before the creation of the thread. Therefore, TSDuck exclusively uses ts::Thread and ts::ThreadAttributes instead.

TSDuck relies on C++ mechanisms to track the usage of resources. Standard classes such as std::lock_guard or std::unique_lock are used to ensure that no dangling lock is lost through the guard design pattern.

In addition to multithreading, TSDuck also uses event dispatching. See chapter 4 for more details.

The class ts::ResidentBuffer implements a buffer which is locked in physical memory, preventing paging or swapping on this buffer. This is useful for large data buffers with high performance constraints.

This is a template class. The template parameter is the type of the elementary data in the buffer.

The core data of the tsp processor is a ts::ResidentBuffer<ts::TSPacket>. The incoming packets are directly written into this buffer by the input plugin. Each packet processing plugin directly reads and writes the packets here. And the output plugin reads the packet there, at the very same place they were written by the input plugin. Given that this global buffer is locked in physical memory, the best performances are guaranteed.

Note however that most operating systems require that the application has privileges to lock physical memory.

To track potential memory leaks and the impact of the application on the system, the class ts::SystemMonitor creates a background thread which reports the process metrics of the application at regular intervals.

The class ts::ForkPipe is a portable and convenient way to create a process running a specific command and creates an outgoing pipe from the calling application to the standard input of the created process, or an incoming pipe from the standard output of the created process to the calling application, or both. The pipes are open in binary mode (when it makes sense for the operating system) and can be used to pass an entire transport stream when necessary.

The classes ts::IPAddress and ts::IPSocketAddress define an IP address and a corresponding socket address (an IP address and a port number). Host name resolution and multicast are supported.

The class ts::IPAddress can hold IPv4 and IPv6 addresses. All network operations on name resolutions and TCP or UDP sockets are transparently supported for IPv4 and IPv6 addresses.

The classes ts::TCPSocket and ts::UDPSocket implement TCP/IP and UDP/IP endpoints.

The class ts::UDPSocket can be used directly to send and receive datagrams. Multicast is supported.

The class ts::TCPSocket can be used only through two subclasses. The subclass ts::TCPConnection is a TCP/IP communication endpoint, either on client or server side. It is used to send or receive data streams. The subclass ts::TCPServer is used to implement a TCP server. It accepts incoming client connections and initiates a ts::TCPConnection for each new connection. On the client side, the class ts::TCPConnection is directly used to connect to the server.

The class ts::WebRequest performs simple Web requests using HTTP, HTTPS or FTP. Using a URL, the result can be downloaded in memory or in a file. Multiple redirections and SSL/TLS are automatically handled. This class is built on top of native system libraries, libcurl on UNIX systems (Linux, macOS, BSD), WinInet on Windows.

In addition to straightforward communication classes using blocking I/O, TSDuck also uses event dispatching on non-blocking I/O ("immediate" I/O on UNIX systems or "asynchronous" I/O on Windows). See chapter 4 for more details.

The TSCore library contains classes to load shared libraries (.dll on Windows, .so on Linux and BSD, .dylib on macOS) and lookup symbols inside them in a portable way. These classes are typically used to load tsp plugins but can be used in any application.

The class ts::SharedLibrary manipulates any type of shared library.

The subclass ts::ApplicationSharedLibrary searches a shared library using TSDuck rules: if the file is not found "as it is", an optional prefix and a list of directories are used. This is how, on Windows for instance, searching the shared library named dektec will end up loading the file tsplugin_dektec.dll in the same directory as the application executable file.

Applications which interact with smart-cards shall use the PC/SC interface. PC/SC is a standard interface which was originally developped for Windows but which is also available on Linux and macOS.

The TSDuck library does not embed or hide PC/SC but it provides a few utilities like transmitting an APDU and read the response in one single function or searching a smart-card with some characteristics in the ATR from all connected smart-cards.

All these utilities are grouped in the namespace ts::pcsc.

The class ts::COM provides a portable and reliable way to make sure that the Common Object Model (COM) is properly initialized and terminated on Windows systems. This class is defined on all platforms but does nothing on non-Windows systems. It is consequently safe to use it everywhere without tedious conditional compilation directives.

Other classes manipulate Windows-specific objects and are not available on non-Windows systems.

The template class ts::ComPtr is the equivalent of a smart pointer for COM objects. The reference count of a COM object is properly incremented and decremented when the COM object is manipulated through a ts::ComPtr. The COM object is automatically released when no more reference exists.

There is little advantage to develop an intrinsicly non-portable COM object class. However, in order to access tuner devices, TSDuck needed a few custom internal COM classes to interact with the DirectShow framework. These internal classes needed some COM support functions which are available to applications (just in case…​)

The class ts::TSPacket defines a transport stream packet. It is in fact a flat structure which occupies exactly 188 bytes in memory. It is safe to use arrays or vectors of ts::TSPacket. The packets are guaranteed to be contiguous in memory.

The class ts::TSPacket also adds many operations on the TS packet to read or modify properties like the PID (type ts::PID), the continuity counters or deeper structures like PCR, DTS or PTS.

The class ts::TransportStreamId contains the identification of an MPEG/DVB transport stream.

The class ts::Service contains all possible properties of a DVB service. Not all properties need to be set at the same time. Each property can be individually set, cleared or queried.

Transport stream files are implemented by classes ts::TSFileInput and ts::TSFileOutput. They respectively read and write transport stream files with specific features such as repeating the reading of a part of the file.

The subclass ts::TSFileInputBuffered provides additional, but limited, capabilities to seek forward and backward on non-seekable files such as pipes.

The subclass ts::TSFileOutputResync adds resynchronization capabilities on continuity counters and PID’s.

The class ts::TSAnalyzer consumes all TS packets from a transport stream and analyzes virtually everything from the stream. This is the class which is used by the command tsanalyze and the plugin analyze to collect the vast amount of information it reports.

The class ts::PCRAnalyzer is a useful tool to evaluate the bitrate of a transport stream. It performs the analysis of the Program Clock Reference (PCR) which are present in the transport stream in order to evaluate the bitrate of the stream. If PCR are not found, the class can also use Decoding Time Stamps (DTS) to evaluate the bitrate. This is less precise than PCR but can be used as a backup.

The TSDuck library provides classes to manipulate PES packets and a few audio and video attributes. These features are limited to the analysis of a transport stream. There is no video or audio decoding features. Specialized libraries exist for this and are out of scope for TSDuck.

The class ts::PESPacket implements a PES packet and can manipulate its attributes, header and payload.

The class ts::PESDemux extracts PES packets from a transport stream. It can also notify the application of the changes in audio or video attributes.

The abstract class ts::AbstractAudioVideoAttributes is the root of a hierarchy of classes which contains attributes for audio or video streams. Currently, specialized classes exist for MPEG-2 video, AVC/H.264, HEVC/H.265, VVC/H.266 video, MPEG-2 audio and AC-3 audio.

The class ts::AVCParser performs the parsing of an AVC, HEVC, or VVC bitstream.

Signalization objects, sections, tables and descriptors, can be manipulated in several formats: binary objects, specialized classes and XML.

Tables in JSON format are also supported through automatic XML-to-JSON translation.

The classes ts::Section, ts::BinaryTable and ts::Descriptor implement binary forms of the signalization objects.

A binary table are made of a collection of sections. A binary table is valid when all binary sections are present. Each section contains its section number in the table and the total expected number of sections inside the table.

All sections and descriptors can be represented by the classes ts::Section and ts::Descriptor. They simply contain the complete binary content of the object and can manipulate the various components. An instance of ts::Section stores the table_id and manipulates the various components of the section header. For long sections, the final CRC32 can be checked for consistency or recomputed after modification of the section content.

Tables can be stored in binary files. The format of these files is quite simple. They just contain raw binary sections, without any encapsulation. Tables can also be stored in XML or JSON files. The class ts::SectionFile reads and writes tables or section from files, independently of the format, either a binary section file or an XML file.

Tables and descriptors can also be manipulated using specialized classes such as ts::PAT or ts::PMT for tables and ts::ContentDescriptor or ts::ShortEventDescriptor for descriptors.

All specialized classes inherit from a common abstract root named ts::AbstractSignalization. All descriptors inherit from the intemediate class ts::AbstractDescriptor. All tables inherit from the intemediate class ts::AbstractTable. Tables with long sections inherit from ts::AbstractLongTable.

Most tables and descriptors are implemented, from MPEG, DVB, ATSC, ISDB and a few private descriptors. Unimplemented descriptors shall be manipulated in binary form (or be implemented…​)

Binary tables or descriptors are converted from or to specialized classes using serialize() and deserialize() methods. The validity of a binary or specialized object can be checked using the isValid() method.

Sample deserialization code:

The deserialization can also be done in the constructor. And the validity and table_id checking is done anyway in the deserialization. So, the previous code can be simplified as:

Sample serialization:

Note that an instance of the class ts::DuckContext can store various information about the way to interpret incorrect signalization or preferences. Its default value is appropriate for a standard PSI/SI processing.

Each time the instance of ts::DuckContext is used, it accumulates information. For instance, if it is used to deserialize an ATSC MGT table, the information that the TS is an ATSC one is retained. Later, if the same instance of ts::DuckContext is used to deserialize a descriptor for which there is an ambiguity (the tag is used in two standards for instance), the ATSC version of the descriptor will be used.

It is also possible to automatically define and load command line options to preset the state of the instance of ts::DuckContext. See section 3.5.3 for more details.

Finally, specialized classes for tables and descriptors can be converted to and from XML using the methods toXML() and fromXML().

These methods are typically used by the class ts::SectionFile which represents a file containing sections and tables in binary or XML format. The class can be used to load a set of tables in XML format or to store table objects in XML format.

The class ts::SectionFile is the core of the tstabcomp utility, the tables compiler (or decompiler).

Signalization objects can be extracted from transport streams using the class ts::SectionDemux and inserted back into transport streams using the class ts::Packetizer. These two classes also have specialized subclasses.

An instance of ts::SectionDemux can extract sections or complete tables in binary form.

Tables with long sections are usually cycled. A given table with a given version number and a given table id extension is reported only once, after collecting all its sections. The same table will be reported again only when its version number changes.

On the contrary, short tables are all reported since they do not implement versioning.

It is also possible to use a ts::SectionDemux to be notified of all individual sections.

The class ts::DuckContext carries various preferences about the standards or localizations. Typically, each application has a given context. Using tsp, each plugin has it own context.

The preferences which are carried by a context include the default standard (DVB, ATSC, ISDB), the default character sets in PSI/SI, the default private data specifier (for DVB private descriptors), the HF region (for terrestrial or satellite frequency mapping)

The ts::DuckContext class can automatically define command-line arguments to explicitly specify preferences (options --atsc or --default-charset for instance). Thus, the preferences are setup from the beginning.

But preferences are also accumulated all along the execution. For instance, as soon as an ATSC table is demuxed, the fact that the transport stream contains ATSC data is stored in the context. Later, when an MPEG table (a PMT for instance) contains an ambiguous descriptor tag which is used by DVB and ATSC, then the ATSC alternative will be used.

In a multithreading architecture, each sequential activity is linearly programmed in an independent thread of execution. A server typically contains one or two threads per client connection. If the client-server protocol is synchronous, with a strict request-response pattern, each client connection is implemented using one thread. It repeatedly reads a request, processes it, writes the response and waits for the next request. On the other hand, if the protocol is asynchronous, independent messages can flow in both directions at all time. In that case, a client connection is typically implemented using two threads: one thread processes incoming messages, the other thread generates outgoing messages.

The development of that architecture is simple. The processing is implemented using linear programming, reading and writing data when necessary. The core model is "blocking-I/O". Reading or writing a message suspends the execution of the thread during the I/O, until an incoming message is received or an outgoing message is fully sent.

On the other hand, for an I/O-bound application, a multithreading architecture is costly for the two other types of resources: CPU and memory. Constant context switching between threads is costly in terms of CPU and kernel resources. The impact on memory is even worse: each thread needs an independent execution stack, while spending most of the time doing nothing but waiting for the current I/O operation.

In an event-dispatching application, there is only one thread of execution. That thread starts all I/O operations but never waits for them to complete. Instead, after starting one I/O for a client connection, for instance, it does some processing for another connection, until that client connection starts another I/O, etc. The core model is "immediate I/O" or "asynchronous I/O" (we will see the difference between the two later). The completion of each I/O is notified as an event. Processing that event means continuing the activity for the corresponding client connection, until it starts another I/O.

Because the driving principle of this architecture is "reacting" to events, the core feature of the application is named a "reactor design pattern". Object classes which are connected to the reactor and activated on reception of events are named "reactive classes".

I/O completion is one type of events but other types of events are possible: timers, used-defined events, process completions, message passing, etc.

The core reactor waits on events. Each time an event is received, an application-defined "event handler" is invoked. The event handler shall be quick. It shall not block or execute lengthy CPU-bound processing. It can only starts I/O, not wait for them to complete. Then, the event handler returns control to the reactor.

The benefit is obvious in terms of CPU and memory. There is only one thread with only one execution stack. Each client connection only requires its own contextual data. The kernel processes I/O only, not multiple threads and context switches.

However, because any benefit has a price, the programming technique is more complex. Each activity (e.g. a client connection) must be implemented as a state machine, using a specific data context and fragmented processing between I/O operations. Additionally, the sum of all CPU loads from the various activities shall not exceed the capability of one CPU.

The main practical limitation of reactors is the necessity to control absolutely all types of input/output so that no code suspends the execution of the reactor thread during an I/O. This means that third-party libraries which perform I/O under the hood cannot be used because calling a function of these libraries may block the reactor thread.

Some libraries, such as OpenSSL may have two modes: a simple one using linear programming and blocking I/O and a much more complex one where the library only performs data processing and lets the application perform the I/O. However, libraries such as libsrt and librist (which implement the SRT and RIST protocols, respectively) only have a linear API using blocking I/O. These libraries cannot be used in an application using an event dispatching architecture.

The second practical limitation of reactors is the absence of standard API and portability. The typical example of reactor-based or event-driven application is a graphical user interface (GUI). A GUI is the source of multiple events (mouse, keyboard, data sources) which can occur in any order, at any time. All GUI systems are based on the principle of event dispatching: Microsoft Windows, GTK, Qt, X-Window, etc. However, each environment defines its own API for event dispatching. They use different and incompatible APIs. To add to the confusion, other non-GUI event dispatching frameworks exist with different APIs, from the legacy Adaptive Communication Environment (ACE) to the modern libevent.

Because of the lack of a common unique, or universally accepted, standard API, it is not possible to develop portable applications. It is not possible to develop libraries which are based on a reactor API and smoothly integrate them in all event dispatching applications. As an example, libsrt proposes a "non-blocking I/O" mode but it is based on a specific libsrt API and it can be used only in applications which use libsrt as their reactor. When several libraries only accept their own specific model as event dispatcher, they cannot be used in the same application.

The typical use cases of these two architectures are:

An "immediate I/O" is …​ immediate. Either it immediately completes or immediately fails but it never blocks. When an immediate I/O fails because it is not possible to immediately complete (e.g. because no incoming data is available), the process can be notified by the kernel when an I/O becomes possible. There are two main types of I/O events: "read-ready" and "write-ready", when a read or write operation becomes possible. Note that "ready" does not mean "guaranteed". On a read-ready event, for instance, the application shall retry the read operation. It will likely immediately succeed but it may also fail again (spurious read-ready event).

With immediate I/O, there is no specific constraint on the user data buffers. After an I/O call, whether it fails or succeeds, the user buffers are never used outside these I/O calls and the buffers can be disposed at any time. A device or socket can be closed at any time without impact on the data buffers.

To make portability more difficult, distinct flavors of UNIX systems use different incompatible system calls and kernel mechanisms to handle immediate I/O event notification.

There are some limitations to the UNIX immediate I/O model:

An "asynchronous I/O" always executes in the background. An I/O is first started using a classical read or write operation. In rare cases, it completes immediately (e.g. when data are already available from a kernel buffer). However, most of the time, the I/O is "pending", it runs in the background. Later, when the I/O completes, the completion event is reported to the application.

Microsoft Windows uses the name "overlapped I/O" instead of asynchronous I/O. The event synchronization mechanism is named I/O Completion Port, or IOCP in short.

Asynchronous I/O has an important impact on the data buffer management in the application. When a read operation starts, for instance, the address and size of the buffer are remembered in the kernel. This memory buffer is then written in the background, when data arrive as part of the read operation in progress. This means that the application must not reuse the memory of that buffer before the confirmation that the I/O completed, either successfully or with an error. If the application frees a data buffer during an asynchronous I/O, the following dangerous scenario becomes possible: the freed buffer is reused by some other part of the application and some precious data are stored there; then the asynchronous I/O continues and incoming data erase the reused buffer.

Therefore, an application must be very careful about data buffers. These buffers must never be freed or reused before the confirmation of the I/O completion. This can be challenging in error paths, when a device must be prematurely closed. The application shall not simply close the device and forget about it, as it would be with classical blocking I/O. The application shall explicitely cancel all pending I/Os, then wait for the cancelation to complete (i.e. the asynchronous I/O completes with a "canceled" status), then close the device, and sometimes wait for the completion of the close operation.

The Windows asynchronous I/O model has its limitations too:

On the left side of the diagram, the hierarchy of subclasses of ts::Device contains all standard I/O classes.

The class name Device means a device with the potential of being used in non-blocking mode. By default, instances of these classes work in blocking mode and are suitable for use in linear programming. Applications can use them directly.

The three main types of I/O classes are binary files, pipes (to or from a forked process), and network sockets.

Files and pipes have a subclass for transport streams each.

TCP sockets, server and client connection, have a TLS subclass each, for encrypted servers and connections.

Files, pipes, and TCP connections expose the interface ts::StreamInterface. This interface declares methods such as writeStream(), readStream(), and endOfStream() which can be indifferently used of files, pipes, and TCP connections.

Any class which exposes ts::StreamInterface can be used by data-presentation classes such as ts::TextStream or ts::TLVStream, to read and write text lines or TLV (tag, length, value) messages. Thus, it is possible to indifferently exchange these data formats over files, pipes, TCP or TLS connections.

Class ts::WebRequest handles all forms of Web requests, based on URL (HTTP, HTTPS, FTP). This class is implemented using high-level system-specific libraries: libcurl on UNIX systems (Linux, macOS, BSD), WinInet on Windows. Because these libraries don’t expose their I/Os, class ts::WebRequest is not a subclass of ts::Device. Therefore, unlike subclasses of ts::Device, ts::WebRequest can be used in blocking mode only and an entirely different class ts::ReactiveWebRequest is available for non-blocking I/O.

On the right side of the Reactor class diagram, the hierarchy of subclasses of ts::ReactiveBase contains all "reactive classes". Each instance of a reactive class is attached to an instance of ts::Reactor. The same instance of ts::Reactor dispatches events for all reactive classes which use it.

An instance of a reactive class does not implement I/O. It uses an associated instance of a standard I/O class. The reactive class automatically sets the standard I/O class in non-blocking mode. The reactive class starts the I/O and uses the Reactor to synchronize on the I/O completion. When an I/O is completed, the reactive class calls a handler in a reactive interface.

The Reactor class diagram shows which level of reactive class uses which level of standard I/O class. It is important to understand that there is only one instance of a standard I/O class per instance of reactive class. For instance, an instance of ts::ReactiveTCPConnection is associated to one instance of ts::TCPConnection. The superclass ts::ReactiveStream uses the superclass ts::StreamInterface and the superclass ts::ReactiveDevice uses the superclass ts::Device, but all these superclasses are implemented in the same two instances of ts::ReactiveTCPConnection and ts::TCPConnection.

The reactive handler interfaces are implemented by the application-defined classes. When an application starts a non-blocking I/O, it specifies the handler which must be called upon completion. This is the address of an instance of an application class which implements the required handler interface class.

The following diagram summarizes all reactive handler interfaces, with their methods, and which reactive class calls them.

The application-defined classes need to implement the required handler interfaces, depending on which reactive class they use.

The interface ts::ReactorHandlerInterface is directly called by the Reactor. The application may use it for timers, process completions, or user events. Its methods handleReadReady(), handleWriteReady(), and handleAsynchronousIO() are used only inside the reactive I/O classes and are normally never used by an application.

Class ts::ReactiveWebRequest handles all forms of Web requests, based on URL (HTTP, HTTPS, FTP). Like class ts::WebRequest, it is based on high-level system-specific libraries: libcurl on UNIX systems (Linux, macOS, BSD), WinInet on Windows. Because these libraries don’t expose their I/Os, class ts::ReactiveWebRequest is not a subclass of ts::ReactiveDevice. It is a direct subclass of ts::ReactiveBase Similarly, ts::ReactiveWebRequest handles its own non-blocking I/Os using specific modes of libcurl and WinInet. Unlike other types of reactive I/O classes, it does not use a separate instance of ts::WebRequest.

The TSCore library contains class ts::ReactiveServer which implements a general-purpose TCP server in a Reactor environment. It is suitable to implement all kinds of servers, including SSL/TLS servers. An instance of ts::ReactiveServer automatically creates application-defined session objects when incoming clients connect and automatically deletes the session object when they disconnect. The session objects are created by an instance of ts::ReactiveServerFactoryInterface. The session object class must implement ts::ReactiveServerSessionInterface.

The TSDuck source code repository contains a sample TCP server based on ts::ReactiveServer in subdirectory sample/sample-reactor-server.

This sample application is single-threaded and uses one single instance of ts::Reactor to dispatch events in the TCP server and all client sessions.

By default, the server uses raw TCP connections. However, using the command line option --tls, it creates a TLS server and encrypted connections. This illustrates how ts::ReactiveServer can indifferently use an instance of ts::ReactiveTCPServer or ts::ReactiveTLSServer.

The global instance of ts::ReactiveTCPServer (or its TLS counterpart) uses an instance of ts::TCPServer. Similarly, each client connection uses an instance of ts::ReactiveTCPConnections (or TLS), which uses an instance of ts::TCPConnection.

The server handles text lines (it is a simplified version of an HTTP server). It illustrates how ts::ReactiveTextStream can indifferently use an instance of ts::ReactiveTCPConnection or ts::ReactiveTLSConnection.

Each client session is implemented in an instance of the application-defined class ClientConnection. Each session instance contains:

The server management is implemented in an instance of the application-defined class ServerControl. This class is used by ts::ReactiveServer for two independent purposes:

This generic application structure is illustrated in the following general diagram which applies to all server applications.

The DuckContext class is used to define and accumulate regional or operator preferences. In the TSDuck C++ programming guide, it is referred to as TSDuck execution context. Most of the time, using the default state of a new instance is sufficient.

The application sample Japanese tables, available in Java and Python, demonstrates how it can be necessary to override the defaults in specific cases.

The reporting classes (ts::Report C++ class hierarchy) are used to report logs, errors and debug. They are consistently used all over TSDuck and are required to use the high level features. There is a large hierarchy of classes in the three languages which can be classified according to two sets of criteria:

The asynchronous abstract classes can be useful to collect events, tables and sections in XML, JSON or binary / hexadecimal form in Java or Python applications when using TSProcessor or InputSwitcher. Some of the sample Java and Python applications illustrate this mechanism.

The SystemMonitor class is available in all languages, C++, Java and Python. It can be used at the top-level of an application to implement the --monitor option as found in tsp and tsswitch. An instance of a thread-safe Report class is used to report monitoring messages.

The SystemMonitor class is very simple to use. Examples are available in Java and Python.

For developers, TSDuck plugins can signal events which can be handled by the application. Each event is signalled with a user-defined 32-bit event code. An application can register event handlers in the ts::TSProcessor instance (see the class ts::PluginEventHandlerRegistry, knowing that ts::TSProcessor is a subclass of ts::PluginEventHandlerRegistry). The event handler registration can include various selection criteria such as event code value or originating plugin (see the inner class ts::PluginEventHandlerRegistry::Criteria).

C++ developers who create their own plugins can signal any kind of event that they later handle in their application. This is illustrated in a C++ sample custom application. In this sample code, everything is customized in the application: the plugin, the event it signals, the associated event data, the application handling of the event.

Since developing a TSDuck plugin is only possible in C++, Java and Python developers have more limited options. Some standard TSDuck plugins such as tables, psi or mpe provide the option --event-code. Using this option, the plugins signal event using the specified event code for each data they handle (sections or MPE datagrams depending on the plugin).

Java and Python applications can derive from class AbstractPluginEventHandler to define and register their own event handlers. Thus, binary sections or MPE datagrams can be handled directly from the plugin to the Java or Python application.

Some plugins are even dedicated to application developers and are useless on tsp command lines. This is the case of the plugin memory (both an input and an output plugin). This plugin, when used in a TSProcessor instance, performs direct transport stream input and output from and to the application using memory buffers. The memory buffers are signalled using plugin events. The memory input plugin is an example of an application-defined event handler returning data to the plugin. See this sample code in the TSDuck source code tree.

The TSDuck Java bindings are installed with TSDuck in /usr/share/tsduck/java. All classes are in a JAR file named tsduck.jar. Simply add this JAR in the environment variable CLASSPATH to use TSDuck from any Java application:

This is similar to Linux, except that instead of /usr/share, use /usr/local/share (Intel Macs) or /opt/homebrew/share (Apple Silicon Macs).

On Windows, Java bindings are optional components of the TSDuck installer. When they are selected for installation, they are installed in the TSDuck area and the environment variable CLASSPATH is modified at system level to include the JAR file of the TSDuck Java bindings.

Thus, any Java program can use TSDuck directly.

The Python bindings are installed with TSDuck in /usr/share/tsduck/python. Simply add this directory in the environment variable PYTHONPATH to use TSDuck from any Python application:

This is similar to Linux, except that instead of /usr/share, use /usr/local/share (Intel Macs) or /opt/homebrew/share (Apple Silicon Macs).

On Windows, Python bindings are optional components of the TSDuck installer. When they are selected for installation, they are installed in the TSDuck area and the environment variable PYTHONPATH is modified at system level to include the root directory of the TSDuck Python bindings.

Thus, any Python program can use TSDuck directly.

The code was initially tested with Python 3.7 and higher. Python 2.x is not supported. Intermediate versions may work but without guarantee.

There are usually two ways to call C/C++ from Python:

The second option is usually more flexible and more generic. However, the generated binary depends on the version of Python. If such an option is used, the binary installation of TSDuck would require a specific version of Python (or a specific set of versions of it). But each system has it own requirements on Python and it is difficult for a product like TSDuck to impose a specific version of Python.

Consequently, the less flexible ctypes approach was chosen. The TSDuck binary library contains C++ wrapper functions to some features of TSDuck and these carefully crafted functions are directly called from Python code using ctypes, regardless of the version of Python. Note, however, that these C++ functions are hidden inside the Python bindings and are invisible to the C++ application developer.

If you create your own third-party plugins (ie. if you are not a TSDuck maintainer), it is recommended to develop your plugins outside the TSDuck project.

Do not modify your own copy of the TSDuck project with your private plugins. This could create useless difficulties to upgrade with new versions of the project.

Consider developing your plugins in their own projects, outside TSDuck. You do not even need to get the full source code of TSDuck. It is sufficient to install the TSDuck development environment (see chapter 2).

An example of a third party plugin project is provided in the directory sample/sample-plugin.

To develop a new plugin named foo, follow these steps:

The last step is automated using the Python script scripts/build-project-files.py. This script explores the source files for all commands and plugins. It automatically generates missing project files and references them in the solution file.

This script can be run on UNIX systems (Linux, macOS, BSD) or Windows systems. On Windows, it can be easier to launch the PowerShell script scripts/build-project-files.ps1, which simply calls the Python script.

In the source file of the plugin, create a C++ class, derived from either ts::InputPlugin, ts::OutputPlugin or ts::ProcessorPlugin. If your plugin implements two capabilities (both input and output for instance), implement the corresponding two classes in the same source file.

See the class diagram of ts::Plugin for a global view of the plugin classes.

Specialized plugins which manipulate exiting tables derive from ts::AbstractTablePlugin. Examples of such plugins are pmt, pat, nit, etc. The actual plugin subclasses focus on the modification of the target table while the superclass automatically handles demuxing, remuxing and creation of non-existing tables.

Specialized descrambling plugins derive from ts::AbstractDescrambler. This abstract class performs the generic functions of a descrambler: service location, ECM collection, descrambling of elementary streams. The concrete classes which derive from ts::AbstractDescrambler must perform CAS-specific operations: ECM streams filtering, ECM deciphering, control words extraction. Most of the time, these concrete classes must interact with a smartcard reader containing a smartcard for the specific CAS.

In its constructor, each plugin receives an associated ts::TSP object to communicate with the tsp main executable. This instance of ts::TSP is a protected field named tsp which can be freely accessed by the code of the plugin.

A plugin shared library must exclusively use that tsp object for text display and must never use std::cout, printf or the like. The class ts::TSP is a subclass of ts::Report and supports all reporting methods such as info(), verbose(), error(), debug(), etc.

When called in a multi-threaded context, the supplied tsp object is thread-safe and asynchronous (the methods return to the caller without waiting for the message to be printed).

Note that the plugin instance is also a subclass of ts::Report and automatically redirects all messages to its tsp field. Therefore, the code of the plugin can transparently use its own methods info(), error(), etc. This is equivalent to calling its tsp.

A plugin can decide to terminate tsp on its own (returning end of input, output error or ts::ProcessorPlugin::TSP_END). The termination is unconditional, regardless of the state of the other plugins. Thus, if several plugins have termination conditions, tsp stops when the first plugin decides to terminate. In other words, there is an "or" operator between the various termination conditions.

The idea behind joint termination is to terminate tsp when several plugins have jointly terminated their processing. If several plugins have a "joint termination" condition, tsp stops when the last plugin triggers the joint termination condition. In other words, there is an "and" operator between the various joint termination conditions.

First, a plugin must decide to use joint termination. This is usually done in method ts::Plugin::start(), using ts::TSP::useJointTermination(bool) when the option --joint-termination is specified on the command line.

Then, when the plugin has completed its work, it reports this using ts::TSP::jointTerminate(). After invoking this method, any packet which is processed by the plugin may be ignored by tsp.

This is an optional but recommended step. One C++ module inside the tslibext_XXX library shall invoke the macro TS_REGISTER_EXTENSION as illustrated below:

Using this declaration, the extension is identified and listed using the command tsversion --extensions.

Without the declaration, the extension is loaded and functional but it is not identified.

To analyze input XML files containing tables, TSDuck uses an XML model to validate the syntax of the input XML file. There is a predefined large XML file which describes all supported tables and descriptors.

An extension may provide additional smaller XML files which describe the new tables or descriptors. See the sample extension for more details. The XML files shall be installed in the same directory as the rest of the extension (and TSDuck in general).

For each additional XML file, there must be one C++ module inside the tslibext_XXX library which invokes the macro TS_REGISTER_XML_FILE as illustrated below:

The usage rules and conventions are identical to the XML file above. The declaration macro for each names file is TS_REGISTER_NAMES_FILE as illustrated below:

Here is an example, from the sample "foo" extension, which defines additional names for a table, a descriptor and a range of CA_system_id.

If your environment defines a third-party table which is unsupported or unknown in TSDuck, you can implement it in your extension library.

First, define the C++ class implementing the table:

In the implementation of the table, register hooks for the various features you support. In this example, we register a C++ class for FooTable:

The last argument to TS_REGISTER_TABLE is a static method of the class which displays the content of a section of this table type.

The XML model for the table is included in the XML file:

for the following binary layout, using the same conventions as MPEG/DVB standards:

Similarly, it is possible to implement a third-party descriptor as follow:

In the implementation of the descriptor, we register hooks for the various features. Since this is a non-DVB descriptor with descriptor tag 0xE8, greater than 0x80, we must set the private data specifier to zero in the ts::EDID ("extended descriptor id").

The last argument to TS_REGISTER_DESCRIPTOR is a static method of the class which displays the content of a descriptor.

The XML model for the descriptor is included in the XML file:

for the following binary layout:

The command tstables (and its plugin counterpart tables) can process vast amounts of tables. To extract specific tables or sections, the command provides filtering options such as --pid, --tid or --tid-ext.

For specific sections, it is possible to define additional filtering options to the tstables command.

The extension library shall provide a C++ class implementing ts::TablesLoggerFilterInterface. The sample foo extension provide an option --foo-id which selects instances of FooTable containing specific values for some foo_id field.

See the documentation of ts::TablesLoggerFilterInterface for more details.

In the implementation of the class, we register it as a section filter for tstables:

If you work with a specific Conditional Access System, you probably manipulate confidential information that cannot be published in an open-source tool such as TSDuck. The solution is to develop a private closed-source extension.

In the extension library, you may register functions to display the structure of the ECM’s, EMM’s or private part of the CA_descriptor. The registration is based on a range of CA_system_id (here the constants CASID_FOO_MIN and CASID_FOO_MAX).

See the documentation for ts::DisplaySectionFunction, ts::LogSectionFunction and ts::DisplayCADescriptorFunction.

To register the display handlers in TSDuck: