TSDuck is a free open-source project which is maintained on spare time by very few developers (only one at the time of this writing). As a consequence, the maintainance of the project is driven by the lack of time and resource. To face these constraints, automation is essential.

This is briefly explained in this presentation.

For future maintainers or contributors, this page lists a few automation procedures which are used by the project.

Generation of the user's guide

The TSDuck user's guide is a large Microsoft Word file (doc/tsduck.docx). Each time the Word file is modified, a number of repetitive tasks shall be performed: updating the TSDuck version, the release date, the table of contents, creating the PDF file, etc. A Powershell script named doc/build-user-guide-pdf.ps1 automates all these tasks.

Documentation update procedure:

Double-click on doc/tsduck.docx to open it with Microsoft Word, update the documentation (new options or commands), save and quit.
Double-click on doc/build-user-guide-pdf.ps1.

If you have defined the default action for PowerShell script to run them, the script is executed, the Word file is automatically updated with all version and date information, all fields are updated, the PDF file is generated.

Be sure to commit and push the updated Word and PDF files.

Continuous integration

Each time a commit or a pull request is pushed to GitHub, the workflow .github/workflows/continuous-integration.yml is automatically run to verify that the project can be built on most supported platforms and contains no regression.

The workflow is run on Linux Ubuntu, macOS, and Windows (64 and 32 bits).
The compilation is done using two revisions of the C++ language: C++17 and C++20.
On Linux, the builds are repeated using two compilers, gcc and clang.
In each configuration of platform, compiler and language level:
- TSDuck is fully built.
- The unitary tests are run.
- The full test-suite is downloaded and run.
- The sample programs are built using a typical TSDuck installation of the binaries which were just built (Linux and macOS only).
The doxygen documentation is rebuilt. Missing documentation for new features can be found in the log. This is done on Ubuntu only.

Nightly builds

Every night, if there were any modification in the TSDuck repo during the last day, "nightly builds" are automatically produced and published on the TSDuck web site. No manual action is required, everything is automated.

The workflow .github/workflows/nightly-build.yml is automatically run every day at 00:40 UTC.

On Linux Ubuntu and Windows (64 bits only), the TSDuck binary packages are built for the current state of the master branch in the repository.
The produced binaries are installed on the CI/CD system.
The full test-suite is downloaded and run.
The complete set of doxygen documentation (a.k.a. "TSDuck programmer's guide") is built and packaged in an archive. If the installed version of doxygen is known to contain bugs in the generation of the TSDuck documentation (many were found), the source code for a known good version of doxygen is downloaded and rebuilt first. This is done to ensure that the documentation is always correct.
The Linux Ubuntu binaries, the Windows binaries and the documentation package are published as "artefacts" of the workflow. They are publicly downloadable from GitHub.

At the end of the nightly-build.yml workflow, when all build jobs are completed, the update job triggers a signal on the TSDuck web site. A PHP script is automatically run on the web site to retrieve, download and publish the latest nighly binaries and programming documentation.

Release creation

Creating an "official" TSDuck release is relatively easy. Although the build time for all binaries can be long, everything is automated and run in the background without requiring the maintainer's attention.

Since TSDuck is maintained with little resource and time, there is no product management cycle. Thanks to the continuous integration process, any state of the repository can be used as a release candidate, as long as there is no issue in the CI process and no major development or fix is in progress.

As a rule of thumb, a new release is produced every 3 to 6 months, when enough new features or fixes are available, based on the CHANGELOG file.

Building the various binaries

Binaries must be built for Windows (64 bits only) and a few major Linux distros (Fedora, RedHat, Ubuntu, Debian, and 32-bit Raspian). Additional binaries are now built for some Arm64 Linux distros.

The idea is to have a central Unix system (Linux or macOS) from which the builds are orchestrated. The script pkg/build-remote.sh can be used to build TSDuck binaries on various remote systems and collect the resulting installers.

The remote systems can be physical systems or virtual machines running on the central system. When a virtual machines is not active, it is automatically booted, the binaries are built and collected, and then the virtual machines is shut down. The currently supported hypervisors to managed virtual machines are VMware and Parallels Desktop (macOS). All systems shall be preconfigured so that the central user is authorized to ssh on each them without password (use public key authentication, not passwordless accounts!)

The maintainer shall typically maintain a personal script, outside the repository, which calls build-remote.sh on all systems.

Typical example of such a script, with one physical remote system (a Raspberry Pi) and 5 virtual machines on the local host:

$HOME/tsduck/pkg/build-remote.sh --host raspberry
$HOME/tsduck/pkg/build-remote.sh --host vmfedora --parallels Fedora
$HOME/tsduck/pkg/build-remote.sh --host vmredhat --parallels RedHat
$HOME/tsduck/pkg/build-remote.sh --host vmubuntu --parallels Ubuntu
$HOME/tsduck/pkg/build-remote.sh --host vmdebian --parallels Debian
$HOME/tsduck/pkg/build-remote.sh --host vmwindows --parallels Windows --windows --directory Documents/tsduck --timeout 20

So, building a release is as simple as running that script. After a couple of hours, all binaries and build log files are available in the pkg/installers subdirectory.

Be sure to check that all binaries are present. Check the log files if some of them are missing.

Creating the GitHub release

Once the binary installers are collected, simply run this command to create and publish the release on GitHub:

pkg/github/release.py --create

The current state of the repo on GitHub is used as base for the release. A tag is created with the version number. The release is created with a explanatory description. All binaries are published as assets of that release.

Creating the HomeBrew release

The GitHub release contains binaries for Linux and Windows. On macOS, TSDuck is distributed through HomeBrew. The binaries for all macOS versions, Intel and Arm platforms, are built inside the HomeBrew CI/CD pipeline when an updated formula is pushed in a pull request.

After the creation of the release on GitHub, create the pull request for the new TSDuck version in HomeBrew using the following command:

pkg/homebrew/brew-bump-formula-pr.sh -f

The option -f forces the creation of the pull request. Without it, it works in "dry run" mode. It can be safe to start with a dry run, as a test.

When the HomeBrew CI/CD pipeline has finished to process the pull request, the new version of TSDuck is available for all flavors of macOS.

Updating the version number

Once a release is published, the minor version number of TSDuck TS_VERSION_MINOR must be updated in the source file src/libtsduck/tsVersion.h.

This is currently not automated and shall be manually updated before the first commit following the publication of a new release.

Cleanup of long-standing issues

The issues area on GitHub is used to report problems, ask questions, and support any discussion about TSDuck. When an issue is obviously completed, because a complete answer was provided or a fixed is pushed, the issue is closed. Sometimes, a plausible response or fix is provided but some feedback is expected from the user to confirm this. When a positive feedback is provided, the issue is closed.

However, some users never provide a feedback after their problem is solved. In that case, the issue remains open forever.

To solve this, there is a label named "close pending". When a plausible response, solution or fix is provided, the maintainer of the project sets the "close pending" label on the issue. It remains open. However, if the issue is not updated in the next 150 days, it will be automatically closed.

This is achieved by the workflow .github/workflows/cleanup-issues.yml. This workflow is scheduled every week on Sunday at 02:00 UTC. It runs the Python script pkg/github/close-pending.py which automatically closes all issues with label "close pending" and no update within the last 150 days.