TSDuck

TSDuck

TSDuck, The MPEG Transport Stream Toolkit

TSDuck is an extensible toolkit for MPEG/DVB transport streams. TSDuck is used in digital television systems for test, monitoring, integration, debug, lab, demo.

In practice, TSDuck is used for:

TSDuck is developed in C++ in a modular architecture. It is easy to extend it through plugins. In fact, the primary requirement for TSDuck architecture was to be able to implement any new feature within a couple of hours, which happened to be the case most of the time.

To be easily extended, TSDuck needed to be simple. TSDuck is a collection of command line tools and plugins. There is no sophisticated GUI. Each utility or plugin performs only one elementary feature but they can be combined in any order.

In short, TSDuck shall be understood as a general-purpose Toolbox for Digital TV Engineers and not as a specialized application for production operators.

TSP, the transport stream processor

The most important TSDuck tool is tsp, the transport stream processor. It is a flexible transport stream processing framework. Running tsp is a combination of elementary processing operations using plugins.

In each invocation of tsp, there are:

This is illustrated in the diagram below:

The currently existing input and output plugins can work on TS files, UDP/IP (multicast or unicast), DVB sources using cheap tuners, professional Dektec modulators and ASI devices, HiDes modulators.

Example: A CAS test bed

The following diagram illustrates a real-life example. To test a Conditional Access System and signalization, tsp is used to create a new transport stream from a live source. One real TS is received as input using a cheap USB or PCI DVB-T tuner. This TS and all its services are renamed, creating a "new TS" in the network. To test the CAS, tsp interacts with a real ECMG and a real EMMG using the standard DVB SimulCrypt protocols. One service, initially in the clear, is scrambled. tsp generates the control words, scrambles the content and inserts ECM's and EMM's. Finally, a set-top box receives all live transponders plus the new one we create.

Everything is done in one single tsp process. The command is the following. For clarity, the details such as tuning parameters, access criteria, IP addresses and ports, are represented as environment variables.

tsp -I dvb -u $UHF_INPUT                                      \
    -P tsrename -t 9 -a                                       \
    -P svrename direct8 -i 0x0901 -l 41 -n "Direct 8 Test"    \
    -P svrename bfmtv -i 0x0903 -l 42 -n "BFM TV Test"        \
    -P svrename 'i>tele' -i 0x0904 -l 43 -n "i>TELE Test"     \
    -P svrename virgin17 -i 0x0905 -l 44 -n "Virgin 17 Test"  \
    -P svrename gulli -i 0x0906 -l 45 -n "Gulli Test"         \
    -P svrename france4 -i 0x0907 -l 46 -n "France 4 Test"    \
    -P svrename 0x02FF -i 0x09FF                              \
    -P scrambler GulliTest -e $ECMG -s $SUPER_CAS_ID          \
         -p $PMT_CADESC_PRIVATE -a $AC                \
         -b $ECM_BITRATE --pid $ECM_PID               \
    -P cat -c -a $CAS_ID/$EMM_PID/$CAT_CADESC_PRIVATE         \
    -P datainject -r -s $MUX_SERVER_PORT                      \
          -b $EMM_MAX_BITRATE -p $EMM_PID             \
    -O dektec -u $UHF_OUTPUT --convolution 2/3 --guard 1/32

This command is complex but rational. The option -I on the first line specifies the input plugin and its arguments. Here, the input plugin is dvb which drives DVB tuners. Each option -P specifies a packet processor plugin and its options. The plugins tsrename and svrename respectively rename the transport stream and a service. The plugin scrambler is a DVB scrambler on one service. It generates control words, interacts with a standard ECMG and insert ECM's. It also updates the PMT of the service to declare the ECM PID. The plugin cat modifies or creates the CAT to declare a new EMM stream. The plugin datainject implements a DVB SimulCrypt EMM/PDG inserter. It is used as a server by a standard EMMG to insert EMM's, just as if it was a MUX. Finally, the option -O on the last line specifies the output plugin and its arguments. Here, the output plugin is dektec which drives a Dektec modulator, a PCIe board in the computer.

For more details, refer to the TSDuck User's Guide.

Of course, such a complex command should be maintained and edited in a shell script. Unless of course you enjoy retyping it every time.

Because of its modular structure, TSDuck is better used with a flexible shell. On Windows systems, you may use PowerShell, Cygwin or MSys (DOS command line windows should now be reserved to polymorphic deviants only).

Example: MPE injection and extraction

This example describes a test bed or demo infrastructure for MPE injection and MPE extraction. The network infrastructure is illustrated in the diagram below.

In network 1, a media server multicasts a transport stream on address 224.250.250.1, port 9000. We want to encapsulate this UDP multicast stream in an existing transport stream using MPE. This existing TS is a live satellite feed which is received using a DVB tuner.

We also change the multicast destination address for the UDP stream to 230.2.3.4, port 7000, in the MPE-encapsulated datagrams. There is no particular reason for this, we just illustrate the feasibility.

The resulting transport stream with embedded MPE is then broadcast. Here, the broadcast network is a Dektec modulator, followed by another computer using a DVB tuner.

This computer is connected to a second network. Another instance of tsp extracts the datagrams from the MPE stream and multicasts them on its network using the modified destination address.

The first tsp command adds two services in the existing transport stream. One service carries the IP/MAC Notification Table (INT). The other one carries the MPE stream. We create a fully DVB-conformant signalization for these two services.

The second tsp command extracts the MPE stream from the received TS and forwards all decapsulated UDP multicast datagrams on the local network.

This example is fully described in the TSDuck User's Guide with all details and corresponding commands.

Merging and forking transport streams

Specific plugins such as merge and fork respectively combine and duplicate transport streams. They are designed to route transport streams from and to other applications. When the "other" application is another instance of tsp, we can create complex processing graphs.

This is illustrated in the diagram below.

In addition to tsp, the command tsswitch can be used to switch inputs between a set of transport streams using the same input and output plugins as tsp. Switching between inputs can be performed automatically at the end of each input or through a remote control application. The remote control protocol is extremely simple, based on text messages in UDP datagrams.

This is illustrated in the diagram below.

The remote control can be performed using simple command lines. The bash shell provides an easy way to redirect output to an UDP message. The following sample commands send UDP messages on port 4444 to system 127.0.0.1 (the local host). This is the easiest way to use the tsswitch remote control.

echo >/dev/udp/127.0.0.1/4444 2
echo >/dev/udp/127.0.0.1/4444 next

The PSI/SI table compiler

TSDuck provides many ways to manipulate PSI/SI and other types of MPEG tables. Tables can be extracted, examined, injected or modified on the fly using specific plugins for tsp.

Tables can be manipulated in several formats:

TSDuck utilities manipulate tables in any of these formats and converts between them. Specifically, the command tstabcomp is a table compiler. It takes XML source files as input, compiles the tables they contain and output binary section files. Thus, you can create any arbitrary table from scratch and build the corresponding binary sections to inject them in a transport stream.

Sample XML source file

The expected XML input format is fully documented in the TSDuck User's Guide. Briefly, an input XML file uses <tsduck> as root node. The root node contains any number of tables. The following sample XML file contains the definition for simple (and incomplete) PAT and PMT:

<?xml version="1.0" encoding="UTF-8"?>
<tsduck>

  <PAT version="8" transport_stream_id="0x0012" network_PID="0x0010">
    <service service_id="0x0001" program_map_PID="0x1234"/>
    <service service_id="0x0002" program_map_PID="0x0678"/>
  </PAT>

  <PMT version="4" service_id="0x0456" PCR_PID="0x1234">
    <CA_descriptor CA_system_id="0x0777" CA_PID="0x0251"/>
    <component elementary_PID="0x0567" stream_type="0x12">
      <CA_descriptor CA_system_id="0x4444" CA_PID="0x0252"/>
      <ISO_639_language_descriptor>
        <language code="fre" audio_type="0x45"/>
        <language code="deu" audio_type="0x78"/>
      </ISO_639_language_descriptor>
    </component>
  </PMT>

</tsduck>

Also a decompiler

The command tstabcomp is also a table decompiler. If the input file is a binary section file, tstabcomp analyzes the tables and recreates the corresponding XML file.

One common need is to modify a binary table in a stream. Most of the time, a specialized plugin can do what you want on the fly. But if no predefined plugin exists for the precise modification you need, here is the most generic way to update a table:

Let's look at the various steps in a real example. We need to reproduce the case of a broadcast stream (satellite, terrestrial, whatever) with a specially crafted NIT. The plugin nit in tsp provides a convenient way to modify a NIT on the fly. But let's assume that the modifications we need are so specific that this plugin cannot do what we want.

First, we capture the original NIT from the live stream:

tsp -I dvb --uhf 24 \
    -P tables --pid 16 --tid 0x40 --max 1 --bin nit.bin \
    -O drop

Then, we decompile the binary table into an XML file. The input file being a binary file nit.bin, tstabcomp decompiles it into the XML file nit.xml.

tstabcomp nit.bin

Now, manually edit the XML file using any text editor and apply the specific modifications we need. Then, recompile the XML file into a binary table.

tstabcomp nit.xml

Finally, we inject the recreated nit.bin into the stream. We capture the real live stream from the antenna, we replace the NIT on PID 16 on the fly with our modified NIT file and we reinject the modified stream into a Dektec modulator on the same frequency.

tsp -I dvb --uhf 24 \
    -P inject nit.bin --pid 16 --replace --stuffing \
    -O dektec --uhf 24 --convolution 2/3 --guard 1/32

Now, we can plug a set-top box or TV set on the output of the modulator and test the result.

Even simpler, XML and binary files can be used indifferently

In fact, there is no need to explicitly decompile and recompile the table. Most TSDuck commands which manipulate tables and sections can use binary section files or XML file indifferently (see the TSDuck User's Guide for more details).

We can save the table directly in XML format:

tsp -I dvb --uhf 24 \
    -P tables --pid 16 --tid 0x40 --max 1 --xml nit.xml \
    -O drop

And, after modification of the XML file, we can directly inject it in the stream:

tsp -I dvb --uhf 24 \
    -P inject nit.xml --pid 16 --replace --stuffing \
    -O dektec --uhf 24 --convolution 2/3 --guard 1/32

Using TSDuck as a library

TSDuck can also be used as an MPEG/DVB C++ library for your own applications. Most of the TSDuck code is located into a large shareable library. The utilities and plugins are usually only small wrappers around the TSDuck library. You may use the TSDuck library to develop new plugins or create completely independent applications.

All C++ classes in the TSDuck library are documented using Doxygen.

There is a binary installer for the TSDuck development environment too. So, there is no need to rebuild TSDuck first. See more details here.

Commands and plugins

Here is a summary of all command line utilities and tsp plugins which are installed with TSDuck version 3.16-1037.

TSDuck commands

tsanalyzeAnalyze the structure of a transport stream
tsbitrateEvaluate the bitrate of a transport stream
tscmpCompare two transport stream files
tsdateExtract the date and time (TDT/TOT) from a transport stream
tsdektecControl Dektec devices
tsdumpDump and format MPEG transport stream packets
tsecmgMinimal generic DVB SimulCrypt-compliant ECMG
tsemmgMinimal generic DVB SimulCrypt-compliant EMMG
tsfixccFix continuity counters in a transport stream
tsftruncTruncate an MPEG transport stream file
tsgenecmGenerate one ECM using any DVB SimulCrypt compliant ECMG
tshidesList HiDes modulator devices
tslsdvbList DVB tuner devices
tspMPEG transport stream processor using a chain of plugins
tspacketizePacketize PSI/SI sections in a transport stream PID
tspsiExtract all standard PSI from an MPEG transport stream
tsresyncResynchronize a non-standard or corrupted MPEG transport stream
tsscanScan a DVB network
tssmartcardList or control smartcards
tsstuffAdd stuffing to a transport stream to reach a target bitrate
tsswitchTS input source switch using remote control
tstabcompPSI/SI tables compiler
tstabdumpDump PSI/SI tables, as saved by tstables
tstablesCollect PSI/SI tables from an MPEG transport stream
tsterinfoCompute or convert DVB-Terrestrial information
tsversionCheck version, download and upgrade TSDuck

TSP input plugins

dektecReceive packets from a Dektec DVB-ASI device
dvbDVB receiver device input
fileRead packets from one or more files
forkFork a process and receive TS packets from its standard output
hlsReceive HTTP Live Streaming (HLS) media
httpRead a transport stream from an HTTP server
ipReceive TS packets from UDP/IP, multicast or unicast
nullGenerate null packets

TSP output plugins

dektecSend packets to a Dektec DVB-ASI or modulator device
dropDrop output packets
fileWrite packets to a file
forkFork a process and send TS packets to its standard input
hidesSend packets to a HiDes modulator device
ipSend TS packets using UDP/IP, multicast or unicast
playPlay output TS on any supported media player in the system

TSP packet processor plugins

aesExperimental AES scrambling of TS packets
analyzeAnalyze the structure of a transport stream
batPerform various transformations on the BAT
bitrate_monitorMonitor bitrate for a given pid
boostpidBoost the bitrate of a PID, stealing stuffing packets
catPerform various transformations on the CAT
clearExtract clear (non scrambled) sequences of a transport stream
continuityCheck or fix continuity counters on TS packets
countCount TS packets per PID
datainjectDVB SimulCrypt data injector using EMMG/PDG <=> MUX protocol
decapDecapsulate TS packets from a PID produced by encap plugin
descramblerGeneric DVB descrambler
duplicateDuplicate PID's, reusing null packets
eitAnalyze EIT sections
encapEncapsulate packets from several PID's into one single PID
fileWrite packets to a file and pass them to next plugin
filterFilter TS packets according to various conditions
forkFork a process and send TS packets to its standard input
historyReport a history of major events on the transport stream
injectInject tables and sections in a TS
limitLimit the global bitrate by dropping packets
mergeMerge TS packets coming from the standard output of a command
mpeExtract MPE (Multi-Protocol Encapsulation) datagrams
mpeinjectInject an incoming UDP stream into MPE (Multi-Protocol Encapsulation)
muxInsert TS packets in a transport stream
nitPerform various transformations on the NIT
nitscanAnalyze the NIT and output a list of tuning information
patPerform various transformations on the PAT
patternReplace packet payload with a binary pattern on selected PID's
pcrbitratePermanently recompute bitrate based on PCR analysis
pcrextractExtracts PCR, OPCR, PTS, DTS from TS packet for analysis
pcrverifyVerify PCR's from TS packets
pesAnalyze PES packets
pmtPerform various transformations on the PMT
psiExtract PSI Information
reduceReduce the TS bitrate by removing stuffing packets
regulateRegulate the TS packets flow based on PCR or bitrate
remapGeneric PID remapper
rmorphanRemove orphan (unreferenced) PID's
rmspliceRemove ads insertions from a program using SCTE 35 splice information
scramblerDVB scrambler
sdtPerform various transformations on the SDT
sectionsRemove or merge sections from various PID's
sifilterExtract PID's containing the specified PSI/SI
skipSkip leading TS packets of a stream
slicePass or drop packets based on packet numbers
spliceinjectInject SCTE 35 splice commands in a transport stream
stuffanalyzeAnalyze the level of stuffing in tables
svremoveRemove a service
svrenameRename a service, assign a new service name and/or new service id
t2miExtract T2-MI (DVB-T2 Modulator Interface) packets
tablesCollect PSI/SI Tables
teletextExtract Teletext subtitles in SRT format
timeSchedule packets pass or drop, based on time
timerefUpdate TDT and TOT with a new time reference
tsrenameRename a transport stream
untilCopy packets until one of the specified conditions is met
zapZap on one service: Produce an SPTS containing only the specified service

Availability

TSDuck is open source and released under the BSD license. It is developed in C++ and designed to be portable. TSDuck is built and tested for Windows, Linux and macOS. Please see the list of useful links in the side menu.

Pre-built binary packages are available for Windows, Fedora, Ubuntu (Intel) and Raspbian (Raspberry Pi). See the Download section in the side menu.

On macOS, TSDuck is installed using the Hombrew packager. See the guidelines to install TSDuck using Homebrew.

Rebuilding TSDuck, with or without its binary installers and packages, is documented here.

Support for specialized hardware, DVB tuners, Dektec devices, HiDes modulators, is available on Windows and Linux only.

Please report problems or ask questions using the issue tracker.

References

TSDuck is referenced in the list of Open Source Software for Broadcasters by the EBU (European Broadcasting Union).

TSDuck was presented at the EBU Open Source Meetup during the IBC 2018.

License

TSDuck is released under the terms of the license which is commonly referred to as 2-Clause BSD License or Simplified BSD License or FreeBSD License.

Copyright (c) 2005-2018, Thierry Lel├ęgard
All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

This software is provided by the copyright holders and contributors "as is" and any express or implied warranties, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose are disclaimed. in no event shall the copyright owner or contributors be liable for any direct, indirect, incidental, special, exemplary, or consequential damages (including, but not limited to, procurement of substitute goods or services; loss of use, data, or profits; or business interruption) however caused and on any theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out of the use of this software, even if advised of the possibility of such damage.