The Xyce Regression Suite, available on the Downloads page or in our github repository at https://github.com/Xyce, contains over 2000 tests intended to demonstrate correct functioning of Xyce on any platform. In addition to basic function tests, the suite also contains test cases designed to verify that particular bugs have indeed been fixed, and to ensure they don’t get re-introduced. These tests are run nightly on all our supported platforms, and also by Xyce developers in the course of their work. The needs of the team are such that the test suite has evolved into a somewhat complex package that includes a controlling script, custom verification scripts, and a system of test tagging that allows selection of subsets of tests as needed. This page is intended to give only a brief description of how to run the suite in the most basic manner (running all the tests appropriate for an open source build) to verify that the build is working correctly.
Running the regression suite is accomplished with a script called “run_xyce_regression,” contained in the “TestScripts” subdirectory of the XyceRegression directory. This script has many options, but the most straightforward way to run it uses only a few of them. Rather than document the script in its entirety, we will simply show a simple pair of command lines that will test serial and parallel builds of Xyce, with a brief explanation of what the command line means.
While the script can be used to test a build that is already installed, the approach taken by most Xyce developers is to run the test suite after building the code, but before installing it. In what follows, we will assume that you are running the test suite from within the build directory of the code you have just compiled. For the sake of example, we’ll assume that you have created two build directories, $HOME/Xyce_Build_Serial and $HOME/Xyce_Build_Parallel, and that you have performed an out-of-source build as recommended in the Building Guide. We will also assume that you have unpacked the regression suite into $HOME/Xyce_Regression-7.X, where X is the current version number.
Prerequisites
The primary test scripts that control the test suite are Perl scripts, and so you must have Perl installed on your system. One of the tests (Certification_Tests/BUG_2007/gstep) requires the Perl Digest::MD5 module. This module is present in many Perl installs on systems we’ve encountered, but we have found some that require installing an extra package to make it available.
A number of tests also use python scripts to do their work, so you must also have python installed. Many tests that use python also use the “numpy” python module, and therefore you also need to have numpy installed. A smaller number of tests in the suite requires SciPy. Perl, Python, and numpy can usually be obtained through system package managers. SciPy may be available in your system’s package manager, but on some systems (including Cygwin) it might need to be obtained from source code. If you run the test suite without having numpy or scipy, you will see about 58 test failures, most of them in the “Output” directory.
The test suite scripts that depend on Python all assume that the command “python” works and is an alias for either python2 or python3. On some systems the python package sets this up automatically, but on others a separate package is required. For example, on Debian-derived systems this package is “python-is-python3”.
To sum up, in order for all tests in the suite to work you will require:
- Perl5 with Digest::MD5 module
- Python3 with appropriate additional package so that “python” is a valid command line name to invoke the interpreter
- numpy
- scipy
If you are missing these prerequisites, some tests in the test suite will fail simply because they are looking for tools that don’t exist.
Testing a Serial Build
A serial build of Xyce may be tested with the following command line, run while the current working directory is $HOME/Xyce_Build_Serial, in which a serial build has been performed:
EXECSTRING=`pwd`/src/Xyce
eval `$HOME/Xyce_Regression-7.X/TestScripts/suggestXyceTagList.sh $EXECSTRING`
$HOME/Xyce_Regression-7.X/TestScripts/run_xyce_regression \
--output=`pwd`/Xyce_Test \
--xyce_test="$HOME/Xyce_Regression-7.X" \
--resultfile=`pwd`/serial_results \
--taglist="${TAGLIST}" \
$EXECSTRING
This command line will test the binary given in the definition of the string EXECSTRING. The --output option tells it to perform the testing and save its output in a subdirectory called Xyce_Test of the current directory. --xyce_test tells the script where to find the regression suite, itself. The --taglist tells the script which tests to run. The taglist used here is one generated by the script “suggestXyceTagList.sh” in the regression suite, that scans the capabilities of the binary and only runs those tests that are appropriate for it. The script is run and its output evaluated, which will set an environment variable named TAGLIST that we use later.
The reason for using suggestXyceTagList.sh above is to allow for the possibility of testing binaries with incomplete feature sets (such as disabling ADMS sensitivity or the models generated from Verilog-A). suggestXyceTagList.sh runs “Xyce -capabilities” to determine what features were compiled into the binary and emits a tag list to exclude tests that use features not present. Prior versions of this document had suggested a tags list directly that was only appropriate for testing complete builds directly out of their build directories, and would show a few failing tests if run on partial builds or with an installed binary.
The command should run a large number of tests, and—with the exception of tests discussed in the “Some tests are fragile” section below — any failures may be indicative of a problem in your build. It can take anywhere from 20 to more than 60 minutes to run, depending on your operating system and processor.
As the testing proceeds, the success or failure of individual tests will be printed on the console. A copy of that output will also be saved to the file named by the --resultfile option.
Detailed output of all tests and a copy of the input files used to generate them can be found in the Xyce_Test/Netlists directory that will be created in the current working directory. If tests fail, look in this subdirectory to see what happened.
Testing a Parallel Build
Only a few modifications are necessary to test the parallel code. Not all tests that are run in serial are Only a few modifications are necessary to test the parallel code. Not all tests that are run in serial are appropriate for parallel runs, and similarly not all tests appropriate for parallel will work in serial, and therefore the tags list must be modified to select a different subset. Further, one must provide the complete “mpirun” command line to the suggestXyceTagList.sh and run_xyce_regression scripts. The following command line, run from the $HOME/Xyce_Build_Parallel directory will do the job:
XYCE_BINARY="`pwd`/src/Xyce"
EXECSTRING="mpirun -np 2 $XYCE_BINARY"
eval `$HOME/Xyce_Regression-7.X/TestScripts/suggestXyceTagList.sh "$XYCE_BINARY"`
$HOME/Xyce_Regression-7.X/TestScripts/run_xyce_regression \
--output=`pwd`/Xyce_Test --xyce_test="$HOME/Xyce_Regression-7.X" \
--taglist="${TAGLIST}" \
--resultfile=`pwd`/parallel_results \
"${EXECSTRING}"You will have to enclose the mpirun command line in quotation marks so that the script recognizes it as a single argument. You may have to adjust the command line given above to add a complete path to mpirun if mpirun is not in your PATH. As before, we are using “suggestXyceTagList.sh” to generate a TAGLIST variable with the tags appropriate to the binary you have built.
There are fewer tests that will be run with this command line than with the serial build, and—with the exception of tests discussed in the “Some tests are fragile” and “Some tests only work when tested from a build directory” sections below—any failures may be indicative of a problem in your build.
Please note that, while Xyce supports parallel execution for OS X and Linux, only serial execution is supported for Windows.
Also note that the Xyce regression test suite is not generally suitable for running on large numbers of processors, and contains many circuits that are extremely small. Some are so small that they don’t work at all in parallel (which is one of the reasons that we have the “serial” and “parallel” tags system). In addition some tests are just barely large enough to work properly on two processors. So, we suggest running the parallel build over the test suite on two processors, which is how we do our own daily testing of the code.
Speeding up the test suite on systems with many cores
The run_xyce_regression commands in the previous section will work on every system, but run each regression test sequentially. When run this way it can take a long time to run. The wall clock time can be reduced if your computer has a lot of available cores by using an alternate script that runs multiple tests simultaneously.
The script “run_xyce_regressionMP” will take exactly the same command line arguments as “run_xyce_regression” but has the option of running multiple tests simultaneous runs. Simply add the command line argument --numproc=N (where N is the number of tests it may run simultaneously) to enable the simultaneous runs. If using run_xyce_regressionMP with a parallel build, remember to use a “numproc” value such that the product of numproc and the “np” value you give to mpirun is smaller than the number of cores on your system.
On Linux, BSD, and OS X we find this script can tremendously improve the time it takes to run a complete test cycle. On Windows, however, there is so much additional overhead involved in starting up processes and keeping track of the multiple processes being run that use of run_xyce_regressionMP will actually slow down the full test cycle.
Some additional tests can be run to test XDM
Approximately 100 additional tests may be run to test out the XDM netlist converter if you have also installed it. In order for these tests to work, you must have installed XDM either from one of our binary zip files or from its source code, and the directory in which its “xdm_bdl” executable is located must be in your PATH variable. In order to enable these tests, add “?xdm” to your tags list. The “suggestXyceTagList.sh” will automatically add this tag if if finds xdm_bdl in the PATH.
Some tests expose broken OpenMPI installs
If your system’s OpenMPI was built using the option --enable-heterogeneous, then it is broken and will cause a parallel build of Xyce to get a segmentation fault on 6 parallel test cases. This problem is also tested by the special test case, “MPI_Test/bug_967”. If that test case fails, you are being told that your system’s OpenMPI install is broken; this is known to be the case on Ubuntu 17.10 and 18.04, and also on Debian Stretch. This bug was also present on FreeBSD systems prior to 25 September, and may also be present on other systems that the Xyce team has not yet encountered. It is not a bug in Xyce, and cannot be fixed except by replacing the OpenMPI install with one that was built without this option (which is documented in OpenMPI’s documentation as being broken and unusable). Please consult the Xyce Building Guide for more details.
The test cases that will fail, in addition to MPI_Test/bug_967, when the system-installed OpenMPI has this problem are:
- Certification_Tests/BUG_141_SON/bug_141-empty
- Certification_Tests/BUG_520/bug520
- Certification_Tests/BUG_706_SON/rc_simple_lib
- SENS/ampSensAdj_aztec
- SENS/ampSensAdj_belos
- SENS/rc1000TranAdjGear
Some tests are fragile
The test suite will pass all tests with our supported builds, but some tests are so sensitive that they may fail in subtle ways on new platforms, especially on 32-bit architectures. This has become less of a problem since early public releases of Xyce, and in fact failures of this sort have not been observed at all in the last several releases.
There are a few tests that have been observed to fail occasionally without really being a problem. These include the FOURIER tests, which are using a fairly strict comparison criterion, and which have been observed to report a false failure on 32-bit FreeBSD 9 machines and on an Ubuntu 12.04 64-bit system. None of these failures have been observed on more recent operating systems on 64-bit architectures.
The MOS13_IC tests test the use of initial condition specifiers on the levels 1 and 3 MOSFETs using a chain of MOS inverters. Without initial conditions on some of the inverter stages, the circuit fails to obtain a DC operating point. On all of our supported platforms, the initial conditions provided in the netlists are sufficient to allow the operating point to converge, but we have found a few unsupported platforms where an additional one or two initial conditions are required. Failure of these test cases does not necessarily indicate a bad build of Xyce. If these tests fail for you and you’d like to see them pass, edit the netlists and change one or two of the final “INVERTER” subcircuits to “INVERTERON” or “INVERTEROFF” (the ons and offs must alternate, so be sure to make your added initial conditions fit the existing pattern).
The test Certification_Tests/BUG_966_SON/tran-homotopy may fail in parallel on some systems. This is because the ordering of the output variables in the raw file may vary from what the test script expects in parallel on different systems. This test should pass consistently for a serial build though.
Some of the “SAMPLING” and “EMBEDDED” tests may occasionally report failure in parallel. These tests involve random sampling, and the failure in parallel is due only to the test being too sensitive to small numerical differences that can result when Xyce compares two otherwse identical runs that may have been distributed across processors differently.
Some tests only work when tested from a build directory
As stated earlier, the previous command lines specifically assumed you were running the test suite from a build directory. It is possible to run the test suite using a binary you have already installed, but several tests will fail. These tests are all tests that build a small test program and link to the Xyce library. If you want to test a binary that you have installed (either with “make install” or one of our pre-built binaries), you can do so by changing the path at the end of the run_xyce_regression line to point to the binary you installed, but you must also add “-library” to the tags list to exclude these few tests.
If you are following the guidance above and using “suggestXyceTagList.sh” to generate your tag list, the script will have inserted this tag itself when it detects it is running a binary out of a “bin” directory instead of a “src” directory.