The Xyce Regression Suite, available on the Downloads page or in our github repostiory at https://github.com/Xyce, contains hundreds of tests intended to demonstrate correct functioning of Xyce on any platform. In addition to basic function tests, the suite also contains test cases designed to verify that particular bugs have indeed been fixed, and to ensure they don't get re-introduced. These tests are run nightly on all our supported platforms, and also by Xyce developers in the course of their work. The needs of the team are such that the test suite has evolved into a somewhat complex package that includes a controlling script, custom verification scripts, and a system of test tagging that allows selection of subsets of tests as needed. This page is intended to give only a brief description of how to run the suite in the most basic manner (running all the tests appropriate for an open source build) to verify that the build is working correctly.
Running the regression suite is accomplished with a script called "run_xyce_regression," contained in the "TestScripts" subdirectory of the XyceRegression directory. This script has many options, but the most straightforward way to run it uses only a few of them. Rather than document the script in its entirety, we will simply show a simple pair of command lines that will test serial and parallel builds of Xyce, with a brief explanation of what the command line means.
While the script can be used to test a build that is already installed, the approach taken by most Xyce developers is to run the test suite after building the code, but before installing it. In what follows, we will assume that you are running the test suite from within the build directory of the code you have just compiled. For the sake of example, we'll assume that you have created two build directories, $HOME/Xyce_Build_Serial and $HOME/Xyce_Build_Parallel, and that you have performed an out-of-source build as recommended in the Building Guide. We will also assume that you have unpacked the regression suite into $HOME/Xyce_Regression-7.1.
The primary test scripts that control the test suite are Perl scripts, and so you must have Perl installed on your system. A number of tests also use python scripts to do their work, so you must also have python installed. Many tests that use python also use the "numpy" python module, and therefore you also need to have numpy installed. A smaller number of tests in the suite requires SciPy. Perl, Python, and numpy can usually be obtained through system package managers. SciPy may be available in your system's package manager, but on some systems (including Cygwin) it might need to be obtained from source code. If you run the test suite without having numpy or scipy, you will see several test failures, most of them in the "Output" directory.
Testing a Serial Build
A serial build of Xyce may be tested with the following command line, run while the current working directory is $HOME/Xyce_Build_Serial, in which a serial build has been performed:
This command line will test the binary given as its last argument. The "--output" option tells it to perform the testing and save its output in a subdirectory called Xyce_Test of the current directory. "--xyce_test" tells the script where to find the regression suite, itself. The "--taglist" tells the script which tests to run. This particular tags list is exactly the one used in release testing by the Xyce team. The command should run a large number of tests, and—with the exception of tests discussed in the "Some tests are fragile" and "Some tests only work when tested from a build directory" sections below—any failures may be indicative of a problem in your build. It can take anywhere from 20 to more than 60 minutes to run, depending on your operating system and processor.
As the testing proceeds, the success or failure of individual tests will be printed on the console. A copy of that output will also be saved to the file named by the "--resultfile" option.
Detailed output of all tests and a copy of the input files used to generate them can be found in the Xyce_Test/Netlists directory that will be created in the current working directory. If tests fail, look in this subdirectory to see what happened.
Testing a Parallel Build
Only a few modifications are necessary to test the parallel code. Not all tests that are run in serial are appropriate for parallel runs, and similarly not all tests appropriate for parallel will work in serial, and therefore the tags list must be modified to select a different subset. Further, one must provide the complete "mpirun" command line to the script. The following command line, run from the $HOME/Xyce_Build_Parallel directory will do the job:
--output=`pwd`/Xyce_Test --xyce_test="$HOME/Xyce_Regression-7.2" \
"mpirun -np 2 `pwd`/src/Xyce"
You will have to enclose the mpirun command line in quotation marks so that the script recognizes it as a single argument. You may have to adjust the command line given above to add a complete path to mpirun if mpirun is not in your PATH.
There are fewer tests that will be run with this command line than with the serial build, and—with the exception of tests discussed in the "Some tests are fragile" and "Some tests only work when tested from a build directory" sections below—any failures may be indicative of a problem in your build.
Please note that, while Xyce supports parallel execution for OS X and Linux, only serial execution is supported for Windows.
Also note that the Xyce regression test suite is not generally suitable for running on large numbers of processors, and contains many circuits that are extremely small. Some are so small that they don't work at all in parallel (which is one of the reasons that we have the "serial" and "parallel" tags system). In addition some tests are just barely large enough to work properly on two processors. So, we suggest running the parallel build over the test suite on two processors, which is how we do our own daily testing of the code.
Some additional tests can be run for C++11-enabled builds
A small handful of tests in the test suite depend on Xyce having been built with the C++11 standard enabled. If your Xyce binary has been built in this way and you wish to run these additional tests, you may add a tag to your tags list to include them. First, check that "Xyce -capabilities" reports "Build compiler is C++11 compliant". If it does, simply add the tag "?cxx11" to your tags list in the "run_xyce_regression" to add these tests to the set that will be run.
Some additional tests can be run to test XDM
Approximately 100 additional tests may be run to test out the XDM netlist converter if you have also installed it. In order for these tests to work, you must have installed XDM either from one of our binary zip files or from its source code, and the directory in which its "xdm_bdl" executable is located must be in your PATH variable. In order to enable these tests, add "?xdm" to your tags list.
Some tests expose broken OpenMPI installs
If your system's OpenMPI was built using the option "--enable-heterogeneous", then it is broken and will cause a parallel build of Xyce to get a segmentation fault on 6 parallel test cases. This problem is also tested by the special test case, "MPI_Test/bug_967". If that test case fails, you are being told that your system's OpenMPI install is broken; this is known to be the case on Ubuntu 17.10 and 18.04, and also on Debian Stretch. This bug was also present on FreeBSD systems prior to 25 September, and may also be present on other systems that the Xyce team has not yet encountered. It is not a bug in Xyce, and cannot be fixed except by replacing the OpenMPI install with one that was built without this option (which is documented in OpenMPI's documentation as being broken and unusable). Please consult the Xyce Building Guide for more details.
The test cases that will fail, in addition to MPI_Test/bug_967, when the system-installed OpenMPI has this problem are: Certification_Tests/BUG_141_SON/bug_141-empty, Certification_Tests/BUG_520/bug520, Certification_Tests/BUG_706_SON/rc_simple_lib, SENS/ampSensAdj_aztec, SENS/ampSensAdj_belos, and SENS/rc1000TranAdjGear.
Some tests are fragile
The test suite will pass all tests with our supported builds, but some tests are so sensitive that they may fail in subtle ways on new platforms, especially on 32-bit architectures. This has become less of a problem since early public releases of Xyce, and in fact failures of this sort have not been observed at all in the last several releases.
There are a few tests that have been observed to fail occasionally without really being a problem. These include the FOURIER tests, which are using a fairly strict comparison criterion, and which have been observed to report a false failure on 32-bit FreeBSD 9 machines and on an Ubuntu 12.04 64-bit system. None of these failures have been observed on more recent operating systems on 64-bit architectures.
The MOS13_IC tests test the use of initial condition specifiers on the levels 1 and 3 MOSFETs using a chain of MOS inverters. Without initial conditions on some of the inverter stages, the circuit fails to obtain a DC operating point. On all of our supported platforms, the initial conditions provided in the netlists are sufficient to allow the operating point to converge, but we have found a few unsupported platforms where an additional one or two initial conditions are required. Failure of these test cases does not necessarily indicate a bad build of Xyce. If these tests fail for you and you'd like to see them pass, edit the netlists and change one or two of the final "INVERTER" subcircuits to "INVERTERON" or "INVERTEROFF" (the ons and offs must alternate, so be sure to make your added initial conditions fit the existing pattern).
The test Certification_Tests/BUG_966_SON/tran-homotopy may fail in parallel on some systems. This is because the ordering of the output variables in the raw file may vary in parallel, from what the test script expects, on different systems. This test should pass consistently for a serial build though.
Some of the "SAMPLING" and "EMBEDDED" tests may occasionally report failure in parallel. These tests involve random sampling, and the failure in parallel is due only to the test being too sensitive to small numerical differences that can result when Xyce compares two otherwse identical runs that may have been distributed across processors differently.
Some tests only work when tested from a build directory
As stated earlier, the previous command lines specifically assumed you were running the test suite from a build directory. It is possible to run the test suite using a binary you have already installed, but several tests will fail. These tests are all tests that build a small test program and link to the Xyce library. If you want to test a binary that you have installed (either with "make install" or one of our pre-built binaries), you can do so by changing the path at the end of the run_xyce_regression line to point to the binary you installed, but you must also add "-library" to the tags list to exclude these few tests.