Make Way for a Better Build Tool: Software Construction with SCons
Originally published at SNUG Silicon Valley 2024
Table of contents
Introduction
Modern hardware development projects are complex. They involve many different tools, languages, and processes. The complexity of these projects is only increasing as the industry moves towards more advanced process nodes, more complex designs, and more advanced verification methodologies. As the complexity of these projects increases, so does the complexity of the build process, or the process of taking the source code and compiling it into a form that can be used by the rest of the project. The build process can be a simple single-line command, but is more typically a complex series of commands that must be run in a specific order with many varying configuration flags that need to be controlled. Build tools such as Make and SCons help to automate this process and return to the simple single-line command developers prefer.
SCons is an open-source alternative to the classic Make utility [1]. It is written in Python, and uses Python scripts to define the build process. SCons is a powerful tool that can be used to build any type of project, but it is especially useful for hardware development projects with many tools to manage and configurations to learn. SCons is easy to set up for a beginner, and extremely powerful for developers with Python familiarity or teams interested in improving their build efficiency.
Basic SCons: A simple example, SystemVerilog style
Every SCons project starts with a file called the SConstruct file. This file is a Python script that defines the build process. The SConstruct file is the only file required to build a project with SCons (as long as the SCons library is installed somewhere on your system, either globally or in a Python virtual environment). As it is a Python script, standard Python code is allowed in the SConstruct file. The SCons library provides additional classes and functions that provide control specific to the build process SCons executes.
The example above is a simple SCons example that contains two
instructions. The first instruction builds the command for compiling
three source files into a simv
executable, and the second
instruction runs the simulation. These instructions are built using an
SCons function inside the SCons Environment
class named
Command
. The Command
function takes three
arguments: the target file(s), the source file(s), and the command to
execute. In this case, The first command looks for three files:
dut.sv
, tb.sv
, and top.sv
. It
then executes the command vcs -o simv dut.sv tb.sv top.sv
to build the simv
executable, substituting the first
argument in the function into the $TARGET
placeholder and
the second argument into the $SOURCES
placeholder. The
second command looks for a single file: simv
, which it
attempts to execute to run the simulation.
To run this example, simply execute scons run_sim
in the
same directory as the SConstruct file and the source code files. SCons
will detect you requested the run_sim
target, determine the
run_sim
target requires the simv
target, and
determine the simv
target requires the three source files.
SCons will then execute the commands to build the simv
executable and run the simulation all in one command. If any of the
source files are missing or if simv
doesn't get generated
properly, SCons will fail with an error message. If you wanted to just
compile and simulate later, you can run the compile target directly with
scons simv
.
When running multiple times, by default, SCons will determine if the
dependencies for the requested target have changed since the last time
the target was built. If the dependencies have not changed, SCons will
not rebuild the target. This is a powerful feature that can save a lot
of time when building large projects. If you want to force SCons to
rebuild a target, you can use the -c
flag to clean the
generated files, or the -u
flag to force SCons to rebuild
all targets. In this example, if you execute scons run
a
second time, you will notice SCons will not rebuild the
simv
executable, as the source files have not changed, and
it will immediately run the simulation. The simulation runs every time
because we used a target for the instruction that never gets generated
by the simulation ( run_sim
), so SCons attempts to build
it (unsuccessfully) every time it is requested.
Intermediate SCons: Adding more features
The example in the previous section was a simple example that would only be used for the simplest of projects, such as a scratch test area. In this section, we will add some features to the example to make it more useful for a real project.
SCons distinguishes between three unique environments: the external
environment, the construction environment, and the execution
environment. The external environment is the environment most developers
are familiar with---it is typically managed through unix scripts and the
terminal. By default, SCons ignores everything in the external
environment to create a portable, self-contained build system. The SCons
developers encourage specifying all paths to tools and any other
variables necessary for the build process in the SConstruct file. The
construction environment is a set of variables that control how SCons
builds the instructions to execute. The execution environment is the
environment the built commands are executed in. If you already have a
complex and stable environment set up, it is likely desirable to copy
that into the execution environment within SCons. This can be seen in
the example above, where the SCons Environment
class is
instantiated as the construction environment, and the ENV
key holds a copy of the external environment for the execution
environment to use.
The construction environment has two additional fields added to it:
SIM_DIR
and DUT_NAME
. SIM_DIR
is
a custom variable that specifies the directory to run the simulation in.
This is useful for keeping the simulation directory (and generated
files) separate from the source code directory. It is controlled through
the AddOption
and GetOption
SCons functions,
which allow you to set the value at the command line by running
scons --sim-dir=./clean_run
, or leaving it to the default
provided. Using the DUT_NAME
variable in place of a
hard-coded dut name allows the SConstruct to be a bit more portable
between projects.
A setup
command was added for auto-generating a
synopsys_sim.setup
file. This file was then added as a
dependency of the compile command to ensure it is generated before the
compiler runs. Because we don't want to specify it on the VCS command
line, the SOURCES
list was sliced to only include the first
three files.
Setting up regressions is intuitive with SCons. By looping through a
list of tests, we can create a unique instruction for each test. In this
case, each instruction shares the simv
dependency so the
project will only need to be compiled once. The actual command is varied
to include the unique UVM_TESTNAME
argument for each test.
Finally, each instruction is added to an SCons Alias
named
regress
. This allows us to run all the tests with a single
command: scons regress
.
Finally, we used the SCons Clean
command to remove the
simulation directory when cleaning, in addition to the tracked target
files. This will take care of all the additional files and folders
created when compiling and simulating.
Using this SConstruct file, developers can compile and simulate any test or every test, clean their work area, and even easily compile into and simulate from a different work area to compare results without needing to clean the default work area. The commands are simple and intuitive:
Advanced SCons: Get the most out of your build system
The SCons User Guide [2] and MAN page [3] are large documents. A lot of the content is dedicated to using the built-in features of SCons for compiling and linking different languages, but there is still plenty of information that can be useful for hardware developers. To get the most out of SCons, being familiar with these documents will benefit you greatly. This section will cover some (but certainly not all) of the more advanced features of SCons that can be used to improve your build system even further.
Replacing env.Command
with a custom tool
All of the built-in languages SCons supports using a set of classes and
functions that are grouped under the label "tool". These tools allow
for building language-specific targets without needing to manually
construct the command. For example, to compile a simple C file, all you
might need to write is Program('hello.c')
. More advanced
configurations for C or any other supported language are configured
through the construction environment and a number of other specialized
functions.
SCons allows you to create your own tools to gain the same type of functionality for any custom build flow. For example, you could write an SCons tool that generates a command for each tool in the VCS library (vlogan, vcs, simv, verdi, etc.). Tools are intended to be fully generic and shared between multiple projects, so this would allow you to create a consistent build flow across all your projects. Then you might have an SConstruct file that looks like this:
For more information on creating a custom tool, see section 17 of the SCons User Guide [2], as well as the following resources [4] [5].
Managing multiple work libraries
Managing the synopsys_sim.setup
file on a complex project
can be a challenge. Using SCons to automatically generate a correct file
every time removes the need to manually manage the file and ensures the
file is always correct. The example in the previous section includes
instructions for indicating what the work libraries will be and
generating the synopsys_sim.setup
file. The
MarkWorkLib
function is a custom builder that creates an
empty .done
file in order to provide a static name to SCons
(SCons doesn't like to work with directories very much since they can't
be tracked for changes directly). The CreateSetup
Builder
receives all the marked work libraries and appends them all to the
generated synopsys_sim.setup
file. Based on the template
used, the final synopsys_sim.setup
file might look like
this:
Parallel builds
SCons has support for building targets in parallel, similar to Make.
This can be enabled by passing the -j
flag to SCons, or
calling SetOption('num_jobs', os.cpu_count())
, which sets
the number of threads to match the number of CPU cores on the system.
SCons will then attempt to build as many targets in parallel as
possible. This can be a powerful feature for large projects that take a
long time to build, or during regressions where you might have many
different necessary compile jobs for different hardware configurations.
To best take advantage of this, every instruction in your SConstruct
file should have a unique target list, and none of your compile steps
should target the same work library. Either compile everything in a
single step, or ensure each compile step has a unique work library to
place the results in.
Build results caching
SCons has support for caching build results by providing a path to the
function CacheDir()
. The path should be external to any
single user area and read/writeable by all users on the project. Each
time a user builds a target, the generated file will be stored in the
cache with a hash of the command and the dependencies used to build it.
When another user attempts to build the same target with the same
command and dependencies, the cached version will be retrieved instead.
This can greatly reduce compile time across the team and speed up smoke
regressions in your CI/CD pipeline as well.
Unfortunately, Getting caching to work correctly with the complex file
generation system used by some of the VCS tools can be challenging.
Caching capabilities will vary by command, and requires the command to
be a "pure function", which means the same inputs always build the same
outputs with no side effects. Caching a vlogan
build is
straightforward, as long as the 23 files it generates are properly
specified in the target of the build step. However, it is not enough to
only cache the simv
file from the vcs
command,
since running it requires libraries in the simv.daidir
directory as well. One workaround I have used is to add intermediate
steps to the build flow that generate a tar.gz
archive of
all the vcs
build results, cache the archive, then recover
the archive and extract the files before simulating. This is not ideal
but it does work, and with a long enough elaboration time it can save
significant bandwidth. Another, simpler solution is to call the
NoCache()
function on any given target to tell SCons not to
cache it even if caching is enabled.
Introducing YAML to simplify the SConstruct file and make it portable between projects
Since the SConstruct file is a Python script, it is possible to leverage additional Python packages to help build your instructions. One such package is PyYAML, which allows you to read (and write) YAML files. YAML is a human-readable data serialization language that is commonly used for configuration files. By moving project-specific information into a YAML file, you can simplify your SConstruct file and make it more portable between projects. For example, examine the following YAML file:
This would be used within an SConstruct file like this:
The potential use-cases with using a YAML file to store project-specific
build information are endless. For example, the YAML file is an ideal
central location to manage compile flags. The flexible testbench
architecture presented by Jeff Vance [6] could be managed directly in
the YAML file rather than spread out across separate .f
files. The test list in the YAML can also be augmented to include
additional metadata for each test such as command line arguments or
iteration counts. Combining these ideas together might look something
like this:
The associated SConstruct file would use this YAML file like so:
There are a lot of changes in this example:
-
The entire YAML environment is copied into the SCons environment (lines 6-9). There are many strategies for environment management, and this is one of them.
-
Multiple
simv
targets are created, one for each configuration (lines 11-15). This allows for parallel builds of each configuration. SCons will only attempt to build one of these targets when explicitly requested or when a test is requested that depends on it. They can be run by callingscons run/no_abp_ss/simv
, for example. -
The
vcs
command uses the path to the executable stored in the environment. -
The global VCS arguments are appended to the
vcs
command. -
Within the test section:
-
Each test will create an instruction for each iteration requested, each with a random seed (SCons targets need to be unique, that is why the seed value is added to the target name).
-
Each command builder references the
simv
target for the configuration requested. -
The various arguments in the global arguments section and the specific test section are all combined to create the final command.
-
Each seed of a particular test are added to an alias of the base test name, so running
scons feature_test_1
will run all the iterations requested with a single command. They are also added to theregress
alias as before.
-
All of the dependency management of SCons is still functional in this
example. The specific simv
builds will only run when a test
that uses them is called. If running a regression, both builds will
occur in parallel (if parallel builds are enabled) and the simulations
will run as requested. Running a single test will only compile the
necessary simv
target to run that test.
Implicit dependencies
Up to this point, every example has explicitly listed all dependencies
necessary to build each target. While this is the preferable approach to
ensure SCons can properly detect changes in a source file and rebuild
when necessary, there are cases where it is not practical to do so. For
example, many IP and VIP libraries include hundreds or thousands of
source files and pre-built file lists, and migrating that to a native
SCons source list would be a significant undertaking and not doable
within the time-frame of a typical project, especially when you
typically call the file list with a simple command like
vcs -f files.f
. In these cases, rather than losing the
benefits of dependency tracking, SCons provides a mechanism called a
scanner. Scanners are functions you associated with a particular file
type that search through matching sources for additional dependencies
contained within the files. The search mechanism is typically
regex-based, but any valid Python code can be used. The following
example shows a simple scanner that searches SystemVerilog files for
`include
statements and adds the included files as
implicit dependencies for the build:
With this scanner in place, any time SCons finds a .v
,
.sv
, or .svh
file in the source list of an
instruction, it will run the scanner function to search for additional
dependencies. The scanner function will search for
`include
statements and add the included files as
dependencies. The path_function
argument tells SCons where
to look for the included files. In this case, it will look in the
SVERILOG_INCDIR
environment variable, which will need to be
manually set up. Otherwise, it will default to searching in the same
directory as the scanned file. The recursive
argument tells
SCons to search the included files for additional dependencies as well.
An important consideration to be aware of when using implicit dependencies is that the reliability of SCons properly detecting changes and rebuilding when necessary is dependent on the accuracy of the scanners. If the regex doesn't catch all the files, or if the paths are not set up correctly, SCons will not detect changes in the files it doesn't know about and will not rebuild when you might expect it to. For this reason, explicitly listing dependencies is always preferable to implicit dependencies.
For more information on writing scanners, see section 20 of the SCons User Guide [2]. An example scanner is included in the appendix for scanning file lists.
Printing results and integrating with the Synopsys Execution Manager API
After running a regression, it is nice to get a summary of the results.
This is another task made easy with SCons. The third argument of the
Command()
function so far has been a string that gets
called to execute a command. However, that third argument can also be a
plain Python function. With that in mind, the following function can be
used to print a summary of the results of a regression:
Calling scons results
will glob through all the log files
in your run area and print a summary of the results. This is a simple
example, but it can easily be expanded to include more information or to
generate a more complex and better-formatted report.
Augmenting this function to optionally post to the Synopsys Execution Manager API is straightforward.
By running scons results --post-eman
, the results of each
test in the run area can be published to your project EMAN database.
This example showed a general idea about how to accomplish this, but the
exact commands will vary based on your EMAN configuration and the data
you want to publish.
Adding help text to the scons -h
command
This paper has demonstrated many advanced features of SCons that can be
used to improve your build system. However, getting a team to adopt a
new tool is never easy (for the team or the person trying to support the
tool). One way to help with this is to make use of the built-in
documentation features of SCons. By default, when running
scons -h
, you will find the available built-in options for
SCons (such as -c
or -j
) along with
descriptions. There are two methods for expanding this output and making
it more relevant to your project. First, if you are using the
AddOption
function to add new command line switches for
your build, each function includes a help
input which gets
appended to the scons -h
output.
The options above append the following to the output of
scons -h
:
The second method for adding help text is to call the SCons
Help
function. This function adds the provided block of
text to the scons -h
output. You could use this to list
targets, provide command line examples, or add anything else that would
benefit your team. The text passed in can be manipulated with any
standard Python, so you can dynamically generate it from the YAML file
to automatically keep it up to date as your target list changes! For
example:
Calling this in your SConstruct file will add the following to the
scons -h
output:
Conclusion
SCons is a powerful build system with far more capabilities than can be covered in a single paper. However, the Python-based structure and the well-written documentation make it easy to get started and learn more as you go. The examples and techniques presented in this paper summarize some of the most useful features of SCons. They are intended to help you get started and provide a foundation for you to build on. SCons provides the tools powerful and flexible enough to standardize your build process across your team and improve your build efficiency.
References
Appendix: A File List Scanner
A path function is not needed because all file paths specified in a
file list are either absolute paths or relative to the file list itself.
However, any +incdir+
paths found in the file need to be stored
and made available to the svscan
scanner. This is done by adding
them to the INCDIR_SCANPATH
variable in the environment. This
requires two additional steps not shown here. First, the
INCDIR_SCANPATH
variable needs to be added as an empty list to the
builder environment overrides for each compile builder so their paths are
not merged between compiles. Second, the svscan
scanner needs an
update to the second argument of the find_file
function call, which
takes the path list to search for the file in. The argument should include
a concatenation of path
and tuple(env.get('INCDIR_SCANPATH', []))
.