Building Guide

Introduction

This guide gives directions for building Xyce from the source code using the “autotools” build that has been the primary configuration mechanism for Xyce since 2001. A new build system using CMake may also be used, which is documented in the INSTALL.md file (not INSTALL) in the Xyce source tree. (The CMake-based system is used to build our production Windows binaries, but it is considered beta for other operating systems.)

It is recommended that anyone intending to build Xyce be familiar with editing text files and compiling programs on their target computer. Prior experience using command line tools like configure, CMake and make is recommended.

This document does contain system-specific recommendations for configuring and building Xyce and its prerequisites, but also contains general guidance that should be read in its entirety before beginning the process. Most of the problems we have encountered with users having trouble building Xyce or Trilinos have resulted from skipping the general guidance sections and leaping right to the system-specific information.

NOTE: This guide is targeted primarily at users of Xyce external to Sandia. If you are an internal Sandia user of Xyce some modifications to this procedure may be appropriate. Please contact the Xyce team for additional information.

NOTE: This Guide describes the Autotools build system. The Xyce team has developed a new build system that uses CMake. We still use Autotools to build our installers, but we plan to change to CMake with the next release. To use the CMake build, read the INSTALL.md file in the repository. This file can be seen in its formatted state in the Xyce repository on GitHub. Since the CMake documentation is still being developed, this page still contains useful supplementary information.

Overview

Installing Xyce from source requires the following general steps:

• Install a complete compiler suite.
• Install all prerequisite libraries from packages or from source code.
• Build Trilinos from source code and install it.
• Download and unpack the Xyce source code.
• Build Xyce.

Most of the work is in the first three steps, and the most difficult step to get right is building and installing Trilinos.  Therefore this document spends a lot of time explaining those steps in detail.  Building Xyce itself is straightforward once the prerequisite steps are complete.

It is also recommended that after building Xyce one run the complete regression test suite. After building and testing, the code may be installed.

Most libraries that Xyce requires are available through system package managers; they are also all available in source form, which you can compile yourself. Since the majority of the work in installing Xyce involves installing the prerequisite libraries, most of this page is focused on these prerequisites.

Build Directories

It is important to note that our recommendations for building Trilinos and Xyce make use of “out-of-source” builds. This means that the executables and/or libraries are built in a directory completely separate from the directory containing the source code. This is accomplished by creating an empty directory anywhere in the file system of your computer (the “build directory”), and using appropriate arguments in “configure” scripts to tell the build system where to find the source code (the “source directory”).

Doing builds in directories separate from the source code keeps the original source tree pristine, and allows you to build many different versions (specifically, both serial and parallel versions) without changing the source tree. It also permits sharing a single source tree over a networked filesystem, and building the code for multiple operating systems without copying the source to every machine. This process is described in detail in the generic “INSTALL” file included at the top level of the Xyce source tree; this file is part of the GNU Automake system, and is not specific to Xyce.

Upgrading from a previous version of Xyce

The Xyce team has changed some recommended Trilinos build options for release 7.3, and we highly recommend rebuilding Trilinos using this new guidance.  This includes a mandatory change to C++11 and building of two optional packages that we had not previously recommended.

The build instructions in this guide assume that Trilinos has been compiled according to the guidance listed here.  A build of Trilinos you used for a previous version of Xyce might not work, and a version built with Trilinos’ default options will not work.

Prerequisites

Building Xyce requires a computer with a modern C++ compiler with C++11 support. Some of the libraries that Xyce uses also require a Fortran compiler. The Xyce development team has successfully used the Gnu Compiler Collection (gcc), the Intel Compiler Collection and Clang. Other compilers may work, but we have not tested any others. For building and running Xyce in parallel, an MPI compiler front-end is needed, such as Open MPI or MPICH.

The Automake system used by Xyce assumes a basic Unix-like command line system. Windows users may use the Cygwin system for this purpose. The autotools build does not support non-Unix development environments such as Visual Studio; some users have worked out how to get Xyce compiled in these environments, but the team can only support the build types described here.

Xyce depends on many external libraries to supply enhanced parsing, mathematical algorithms and parallel communication. Some of the required libraries may already be present on your system, and many can be obtained with package managers. Rarely, some will have to be built from source. All must be installed prior to building Xyce.

The required libraries for Xyce are listed in the table below. All of the libraries are either freely available, or have equivalents that are open source.  Please note the minimum version numbers listed.  If you try to use versions earlier than those indicated, you will likely see failures.  If shown, maximum version numbers indicate the latest version that has been tested with Xyce.  Those whose names are in red are absolutely required for a minimal build of Xyce.  Missing optional libraries may lead to configure failing (but telling you how to disable the feature that requires that library) or simply produce a Xyce with reduced functionality.  All are recommended for a fully capable build.

Licensing Note: Please be sure to understand the licenses associated with the various libraries. Most of the licenses are common and should be acceptable to most users. Inclusion of some non-open-source libraries could limit your rights to distribute binaries you produce.

Name	Version	Notes and Download URL
BLAS		May be included in your compiler’s libraries, or may require separate package installation; most systems have a version of BLAS in their package management systems. ATLAS may be used to provide optimized versions of BLAS and LAPACK. BLAS source code may be downloaded from http://www.netlib.org/blas if the package is otherwise unavailable.
LAPACK		May be included in your compiler’s libraries, or may require separate package installation; most systems have a version of LAPACK in their package management systems. ATLAS may be used to provide optimized versions of BLAS and LAPACK. LAPACK source code may be downloaded from http://www.netlib.org/lapack if the package is otherwise unavailable.
bison	3.0.4-3.8	Included in most operating systems and package management systems. Download source from http://www.gnu.org/software/bison/ and compile if it is otherwise unavailable on your system.  NOTE:  The Xyce autoconf build process accepts only versions of Bison that have actually been tested with Xyce, not “any version later than 3.0.4”. It is sometimes the case that a newer version of bison appears in package systems in between Xyce releases, which then causes the current release of Xyce to be unbuildable on those systems. If this happens to you, contact the Xyce team and we can give you directions for testing out the newer version of Bison and adjusting the build system.
flex	2.5.34 or later	Included in most operating systems and package management systems. Download source from https://github.com/westes/flex and compile if otherwise unavailable on your system.
FFT package	FFTW 3.x OR Intel Math Kernel Library	Required for Harmonic Balance analysis. Xyce is set up to use either the Intel Math Kernel Library or FFTW3 for this purpose. FFTW3 is available in most Unix-like operating systems’ package management systems. See http://software.intel.com or http://www.fftw.org. Xyce is not compatible with versions of FFTW in the obsolete 2.x series.
SuiteSparse	2.1.1 or later	Xyce may use AMD, which is part of the SuiteSparse package. If otherwise unavailable, download from http://faculty.cse.tamu.edu/davis/suitesparse.html.  Both Trilinos and Xyce may be configured without this support, but it is not recommended.
OpenMPI	1.8.8 and later	This is required for a parallel build of Xyce. Many package management systems provide Open MPI, and it is available from source at http://www.open-mpi.org. Note that some package management systems provide later versions of Open MPI. We have observed infrequent, random hangs of Xyce on exit with Open MPI 1.10 on some operating systems. On some of these same systems using a later version of OpenMPI in the 3.x range might help, but in the case of OS X we continue to see this behavior with all versions to date.  The Xyce team has therefore continued to use OpenMPI 1.8.8 on OS X. We have observed some segmentation faults inside the Zoltan library on some operating systems with pre-built OpenMPI 2.1.x and OpenMPI 3.x packages.  See the note below under “Install prerequisite libraries.”
Trilinos	12.12.1	Xyce requires Trilinos for many linear algebra and solver services. When building the serial version of Xyce one must build Trilinos without MPI support enabled. When building Trilinos for the parallel version of Xyce, then Trilinos must be built with MPI support enabled. Download from the Trilinos GitHub repository at https://github.com/trilinos/Trilinos/releases/tag/trilinos-release-12-12-1. NOTE: The current minimum version of Trilinos usable in Xyce is version 12.12.1.  However, Trilinos versions after 12.12.1 have not been rigorously tested with Xyce and may not work properly.

Important Note Regarding C++11

Xyce requires a fully-compliant C++11 compiler.

Most operating systems come with a version of a compiler that supports C++11, but some long-term support operating systems come with older versions of GCC that do not have complete C++11 support.  GCC 4.9 is the minimum version of GCC that can be used to build Xyce in a manner that passes all of its tests.  If your operating system comes with gcc 4.8, you will need to seek an alternate compiler package that provides a newer version.

Both Xyce’s and Trilinos’ build systems are capable of adding the necessary command line options to force the compiler to use the C++11 standard if it is not the default.

Building Prerequisite Libraries from Source Code

For the best instructions on building the libraries listed in the table above, see the documentation included with those libraries. On many systems, these libraries may be available as pre-compiled packages, and are installable with a package management system such as apt-get, yum, port or brew. Where a package is available for your system, it is usually preferable to use it; the package maintainer will have taken care of special porting issues associated with your system, and should also provide timely updates as those libraries have new releases.

While prebuilt packages can save one time, other considerations may require that one build from source code, instead. If you choose to do so, keep the following information in mind.

It is useful to keep the libraries needed for Xyce in a consistent location. This makes it easier to tell Xyce’s configure script where to find all the external libraries. Additionally, since some libraries require MPI to be enabled, it makes sense to keep separate library directories for MPI and non-MPI enabled code. Thus, we suggest you make two directories such as XyceLibs/Serial and XyceLibs/Parallel. Within these directories you should make lib and include directories. For the sake of consistency we will use $HOME/XyceLibs as the prototypical name for this directory in all our examples below.

Installing Prerequisite Libraries

Except for Trilinos, the libraries required for building Xyce are available in popular system package repositories, and are thus easily installed. The Xyce team recommends using these prepackaged libraries as much as possible.

Note that at this time no system provides packages for versions of Trilinos that are compatible with Xyce, so it will always need to be built from source. While some systems do provide packages for Trilinos in package management systems, these versions are either too old to be used or are not configured to have all the essential package options required by Xyce, and we do not recommend using them even if you happen to find one for your system.

Here we enumerate the packages known to exist on systems the Xyce team has built on. Other systems probably have similarly named packages. The systems for which we provide specific guidance are:

• RedHat Enterprise Linux/CentOS
• Debian Linux and Variants
• Mac OS X
• FreeBSD
• Cygwin (Windows)

Where a system does not provide a package for a given library, you will have to build that library from source. Guidelines for building required libraries from source are given in the final section below.

Note concerning OpenMPI:

It has come to our attention that some Linux distributions from between 2017 and 2020 have broken versions of OpenMPI in their package repositories.  We are not aware of continuing problems in newer releases of these Linux systems, but some of them are still on long term support and you may still be running one of them.

On these systems, OpenMPI has been compiled with the --enable-heterogeneous option, which causes it to perform certain communications operations incorrectly, making it non-compliant with the MPI standard.   Building Xyce with these broken OpenMPI libraries will result in segmentation faults on some problems (including 6 test cases in the Xyce regression suite).

Ubuntu — beginning with release 17.10 and continuing up to but not including 20.04 LTS — is one of these, and there may be others.  On such systems it is essential not to use the packaged OpenMPI to build Xyce.

You can determine whether your system’s install of OpenMPI is broken in this way by running a small test program that is present in the src/test/MPITest directory of the Xyce source tree.  This program is run automatically as part of parallel testing of Xyce, but that would require that you build and test Xyce first; waiting until then to find out that your OpenMPI install is broken is not the best approach.

If you plan to build Xyce in parallel and wish to pre-test your OpenMPI package, you can copy the file src/test/MPITest/testBUG967.c out of the Xyce source tree, then compile and run it yourself:

    mpicc -o testBUG967 testBUG967.c
    mpirun -np 2 ./testBUG967

On some systems you might have to provide full path names to the mpicc and mpirun commands.

If this produces any output with the word “BAD”, your OpenMPI install is broken and cannot be used.  You must instead build and install OpenMPI from source instead of relying on prepackaged binaries from your operating system’s repository.  You are also advised to open a bug report on your operating system’s issue tracker if one does not already exist, and ask that the package be rebuilt without --enable-heterogeneous in the future.  If a bug already exists in their issue tracker for the problem, add your name to the bug as an interested party rather than opening a duplicate bug report.

Linux

There are many different “distros” of Linux, and the Xyce team cannot provide precise guidance for building on all of them. We provide here guidance for building on two popular classes of Linux distributions.

RedHat Enterprise Linux (RHEL) and CentOS

These systems have all the prerequisite libraries available in the RPM repository. The following packages are required:

• A version of gcc, g++ and gfortran no older than GCC 4.9.1
  • Note that the default compiler on RHEL7 is too old and an alternate is required.
• blas-devel
• blas
• cmake (required to build Trilinos)
• lapack-devel
• lapack
• bison
• flex
• fftw-devel
• fftw
• suitesparse-devel
• suitesparse

If you are building Xyce from our Github repository, you will also need:

• autoconf
• automake
• libtool
• git

Install these packages using sudo yum install on each package name. In most cases, the -devel package will also cause the other package without “-devel” to be installed, so it is not strictly necessary to list all of the above.

If you are building the parallel version of Xyce you will also need:

• openmpi
• openmpi-devel

Note that the compiler tool chain available in RHEL7’s primary package repo is too old to use to build with Xyce, and the cmake available there is too old to use with Trilinos.  You will need newer versions.  Suitable compilers are available in the RHEL7 developer’s tool set, but cmake will have to be built from source.  If you do not have access to the RHEL7 developer’s tool set then you will also have to build a suitable version of gcc and g++ from source as well.

If you are installing devtoolset compilers on RHEL7, the following packages should be suitable:

• devtoolset-8-gcc-c++
• devtoolset-gcc-gfortran
• devtoolset-libquadmath-devel

Installing these packages will cause other packages to be installed as dependencies.  After installing these packages, you can access the compiler by invoking the command:

source scl_source enable devtoolset-8

After invoking this command, “gcc”, “g++”, and “gfortran” will be the newer versions of the compiler for the duration of the login session in the shell that invoked it.  This command will need to be run each time you need access to those compilers.

Debian Linux and Variants

All required packages are available in these systems’ default repositories.

• gcc (see the Trilinos note)
• g++
• gfortran
• make
• cmake
• bison
• flex
• libfl-dev
• libfftw3-dev
• libsuitesparse-dev
• libblas-dev
• liblapack-dev
• libtool

If you are building Xyce from our Github repository, you will also need:

• autoconf
• automake
• git

If you are building the parallel version of Xyce you also need:

• libopenmpi-dev
• openmpi-bin

Install these packages using sudo apt-get install, or a graphical package manager. The “-dev” packages will also pull additional packages in as dependencies.

NOTE:

As of release 17.10 of Ubuntu and continuing up to (but not including) 20.04, the packaged version of OpenMPI in the package repositories have been compiled with the --enable-heterogeneous option, which breaks MPI’s standard compliance and causes Xyce to fail on some problems.  This package cannot be used as is until it is rebuilt and repackaged without that option.  Most of these versions of Ubuntu are past their end of life, but 18.04 LTS is still active and still has the problem.

If you are running an older version of Ubuntu that has this issue, the only workaround is to uninstall the OpenMPI package and build OpenMPI from source yourself, without the --enable-heterogeneous option.  Comment 11 of the Launchpad bug report (https://bugs.launchpad.net/ubuntu/+source/openmpi/+bug/1731938/comments/11) contains instructions for how to rebuild and install OpenMPI using the Debian package building system.

Mac OS X

Building Xyce on Mac OS X will require that you obtain the Xcode package from Apple, and you must also install the XCode Command Line Tools. In addition to XCode, you also need a package manager such as MacPorts (https://www.macports.org), Fink (https://www.finkproject.org/), or Homebrew (https://brew.sh). Install one of these package managers according to the instructions provided on its web site.

It is not necessary to install blas and lapack, as XCode has those available in a system library. Bison and flex are provided by the XCode command line tools. However, in some OS X versions, the flex and Bison supplied by XCode do not work with Xyce; therefore, we recommend using the versions installed by one of the package managers. Before proceding, be sure to read the Trilinos note.

Fink

For Fink, install the following packages:

• a version of gcc (such as gcc8, gcc9, etc.)
• fftw
• suitesparse
• cmake (for Trilinos)
• flex
• bison

If you are building Xyce from our Github repository, you will also need:

• autoconf
• automake
• libtool
• git

For parallel installations, also install

• openmpi

Install packages using sudo fink install on each package.

MacPorts

For MacPorts, install:

• a version of gcc (such as gcc8, gcc9, etc. )
• fftw-3
• SuiteSparse_AMD
• cmake (for Trilinos)
• flex
• bison

If you are building Xyce from our Github repository, you will also need:

• autoconf
• automake
• libtool
• git

Install packages using sudo port install on each package.

For parallel installations, also install

• the appropriate version of Open MPI (such as openmpi-gcc6)

Homebrew

For Homebrew install the following formulae:

• fftw
• a version of gcc (such as gcc, gcc@8, gcc@9, etc. )
• suite-sparse
• cmake (for Trilinos)
• flex
• bison

If you are building Xyce from our Github repository, you will also need:

• autoconf
• automake
• libtool
• git

For parallel installations, also install

• openmpi

Install packages using brew install on each package.

FreeBSD

All libraries required by Xyce except for Trilinos are available in the FreeBSD ports system. Before proceding, be sure to read the Trilinos note.

For the serial version of Xyce, install the following ports:

• lang/gcc or any version-specific package except gcc48.  This is required even if you are building Xyce with the system-default clang compiler, because Trilinos requires a fortran compiler for some of its packages.  In the system-specific instructions below, we’ll assume you’ve installed gcc10
• math/blas
• math/lapack
• math/suitesparse
• math/fftw3
• devel/bison
• devel/cmake (needed for Trilinos)
• textproc/flex

If you are building Xyce from our Github repository, you will also need:

• devel/autoconf
• devel/automake
• devel/libtool
• devel/git

Install each of these using the standard FreeBSD ports system.

If you are building the parallel version of Xyce, you also require:

• net/openmpi

Incompatibility note:

On 4 Oct 2018, the maintainer of the “math/arpack-ng” package made MPICH an unconditional dependency, which means that as of that date, any port that requires arpack-ng will now force MPICH to be installed—this includes math/octave.  Unlike the OpenMPI ports, which install into directories that don’t intefere with each other, MPICH installs its libraries directly into /usr/local/lib, which interferes with all the OpenMPI ports at link time (the MPICH libraries are found instead of the OpenMPI libraries in /usr/local/mpi/openmpi<version>/lib).  This makes it very difficult to build Xyce with OpenMPI while MPICH is installed.   It is possible to build Xyce and Trilinos with MPICH, but we have seen some evidence that MPICH is substantially slower on some systems than OpenMPI on the same machine.  We have not yet found a workaround to make Xyce build properly with OpenMPI in the presence of an MPICH port install, and therefore recommend that you remove MPICH if it is installed (which will, of course, force all other packages that depend on MPICH to be uninstalled).

Cygwin

For Windows systems, the Xyce project team uses the Cygwin (http://www.cygwin.com/) system to provide a Unix-like environment with Unix-style tools. While it is possible to build Xyce using native Windows tools, the team does not support that approach in this release.

Install Cygwin according to its instructions. You will then need to install a series of packages in addition to the base Cygwin packages. Before proceding, be sure to read the Trilinos note.

Add:

• gcc-core
• gcc-g++
• gcc-fortran
• make
• cmake (needed for configuring Trilinos)
• liblapack0, liblapack-devel
• libsuitesparseconfig-devel
• libamd-devel
• bison
• flex
• fftw3 and libfftw3-devel
• libtool
• m4

If you are building Xyce from our Github repository, you will also need:

• autoconf
• automake
• git

The Xyce Regression Suite will also require bc, python36, python36-devel, and python36-numpy

Many of these packages will automatically install additional packages that they require.

The Xyce team has never attempted to build the parallel version of Xyce on Windows. While Cygwin does provide a version of openmpi, it is our understanding that ParMETIS does not currently build easily on Cygwin. Now that Xyce no longer requires ParMETIS, it may be possible to get it working in parallel on Cygwin. If you succeed in doing so, the Xyce team would be interested to hear about your success.

Installing Trilinos

Introduction

Trilinos is an extensive set of numerical packages for a wide range of computational problems. This section will provide an overview of building Trilinos for Xyce. However, for detailed questions on Trilinos or its build system please see the Trilinos getting started page at https://trilinos.org/getting-started/ for places to start your inquiry.

This section assumes you have installed all prerequisite libraries from packages according to the instructions in the previous section, or have built missing libraries from source.

First, download version 12.12.1 of Trilinos from the Trilinos git repository at https://github.com/trilinos/Trilinos/releases/tag/trilinos-release-12-12-1/, and unpack the source code. Xyce is not guaranteed to build properly with other versions of Trilinos.

Because of the size and complexity of Trilinos, it is recommended to build the necessary packages outside the source directories. For example, you could create two “build” directories, called trilinosSerial and trilinosParallel for the serial and parallel builds, respectively.

Trilinos has many packages that can be built, only some of which are required by Xyce. The packages and compilation options are specified using command line arguments for CMake. In the next sections we will provide examples of CMake invocations for typical serial and parallel builds of Xyce. It is important to note that any variation from the scripts given here could lead to Xyce compilation failures due to uncaptured library dependencies.

Building Trilinos involves only a few steps:

• create an empty build directory and “cd” to it.
• invoke CMake using a build script specifying all the required parameters.
• make
• make install

Because multiple versions of Trilinos might need to be built to support both serial and parallel builds of Xyce, we do not recommend installing Trilinos directly into system library and header directories (e.g. /usr/lib or /usr/include). Our instructions for building Trilinos will therefore be set up for installation in a user-specified, non-standard location, $HOME/XyceLibs/Serial and $HOME/XyceLibs/Parallel.

Building Trilinos for Serial Xyce

Serial Xyce requires a build of Trilinos that contains the NOX, LOCA, Epetra, EpetraExt, Ifpack, AztecOO, Belos, Teuchos, Sacado and Amesos packages. Amesos must be configured to use KLU. EpetraExt must be configured with BTF (block triangular factorization), experimental features, and graph reordering enabled. Trilinos must also be told where to find the AMD libraries and headers.

The CMake invocation below also adds additional packages to enable optional features in Xyce.  These are Stokhos, Amesos2, KLU2, and Basker.

After downloading a version of Trilinos no later than 12.12.1 from the Trilinos GitHub repository as described above, you will have a “tarball” of Trilinos source code called “Trilinos-trilinos-release-12-12-1.tar.gz”. We will assume for this example that the file ended up in $HOME/Downloads. Decide where you want to unpack it. For example, we will use the directory $HOME/Trilinos12.12. The commands below extract the tarball in that directory.

mkdir $HOME/Trilinos12.12
cd $HOME/Trilinos12.12
tar xzf $HOME/Downloads/Trilinos-trilinos-release-12-12-1.tar.gz

The CMake invocation below is an example that configures a version of Trilinos with the packages and options required by a serial version of Xyce.  It is very likely that Xyce will not build at all if you do not include at least these options when you build Trilinos.

There are several things to note about this example:

• This is a generic example that you will very likely have to adapt slightly for your specific operating system.  The points below describe the various issues that may need customization.
• This text should be put into a file that will be used as a shell script. This file should be placed in an otherwise empty directory (which we will call the “build directory”). We will use the name “reconfigure” as the name of this script.
• Set the variable “SRCDIR” to have the full path of the location of the Trilinos source code. If you followed the example above, this would be $HOME/Trilinos12.12/trilinos-12.12.1-Source. This variable is used in the example invocation to specify to CMake how to find the source code.
• Set the variable “ARCHDIR” to the location where you want the Trilinos libraries and headers to be installed. In this example, we use $HOME/XyceLibs/Serial. The install process will create “include” and “lib” subdirectories of this directory.  This variable is used in the CMake invocation to set the “CMAKE_INSTALL_PREFIX” parameter.  REMEMBER THIS SETTING.  After Trilinos is built and installed, you will use precisely the same setting for Xyce’s “ARCHDIR” parameter, which tells configure where to find the Trilinos you have installed.
• The example script below  makes extensive use of the shell continuation character “\” that tells the shell to ignore the following newline. The entire CMake invocation is really meant to be a single line, but for clarity we have divided it up using this continuation character. There must be NO characters on the line following the “\”, not even a space. There must also be a space before the “\”. A common source of problems is having spaces after this continuation character—in that case, the shell ignores the space after the “\” and does NOT ignore the newline at the end of the line.
• There must be no space before or after the equals sign on these lines.
• The script specifies which C, C++, and Fortran compilers to use in the lines “-DCMAKE_C_COMPILER”, “-DCMAKE_CXX_COMPILER”, and “-DCMAKE_Fortran_COMPILER” lines.
  • On most Linux systems, the C, C++ and Fortran compilers are named gcc, g++ and gfortran. Your system may be different and may have version number suffixes on the commands.
  • While Mac OS X appears to have “gcc” and “g++” commands, these are NOT the GNU Compiler Collection, but rather just aliases for the system default “Clang” compiler.  You can use this compiler if you like, but then you must also use it to build Xyce.  If you plan to use a real GCC compiler installed by a package manager, you must also use the same compiler here.  Trying to link Xyce built with GCC and Trilinos built with Clang can cause linker errors very late in the Xyce build process, and has frustrated several users.  The OS X-specific Xyce configure lines later in this document all assume you’re using GCC, and wiill have to be modified if you use the system “gcc” (clang) compiler instead.
  • With Fink on OSX, compiler names have a version suffix (e.g., gcc5, g++5 and gfortran5, though version numbers may differ from these examples on your system)
  • With MacPorts on OSX, the compiler names have additional suffixes (e.g., gcc-mp-5, g++-mp-5, and gfortran-mp-5, though specific version numbers and suffixes may differ from these examples on your system)
  • Consult your system documentation for appropriate settings if your system is not covered in the examples above.
• Flags for each compiler are specified in the “-DCMAKE_CXX_FLAGS”, “-DCMAKE_C_FLAGS”, and “-DCMAKE_Fortran_FLAGS” lines. In this example, we are using the same values for all three compilers; these are set in the “FLAGS=” line that defines a shell variable, and then all three compiler flags simply use that variable.
• We have assumed that AMD is installed in /usr/lib and its associated headers are installed in /usr/include/suitesparse, which is where they are installed by the suitesparse packages on 32-bit Linux systems. If they are installed elsewhere, modify the “-DAMD_LIBRARY_DIRS” and “-D_TPL_AMD_INCLUDE_DIRS” settings.
  • On 64-bit linux, the libraries are in /usr/lib64/ instead.
  • On OS X using Fink, the libraries are in /sw/lib and theh headers are in /sw/include/suitesparse.
  • On OS X using MacPorts, the libraries are in /opt/local/lib and the headers are in /opt/local/include.
  • On OS X using Homebrew, the libraries are in /usr/local/lib and the headers are in /usr/local/include. Newer editions of homebrew now install to /opt/homebrew/lib and /opt/homebrew/include.
  • On FreeBSD the libraries are in /usr/local/lib and the headers are in /usr/local/include/suitesparse.
  • On Cygwin, the libraries are in /usr/lib and headers are in /usr/include/suitesparse just as for Linux.
• On Cygwin you must add “-D_BSD_SOURCE -D_GNU_SOURCE” to the CXXFLAGS

Once the script is created and customized, change it to an executable script and execute it using the commands:

chmod u+x reconfigure
./reconfigure

If the script completes without error, Trilinos is configured for building.  Build it and install it with the commands:

make
make install

Example “reconfigure” script (cmake invocation):

#!/bin/sh
SRCDIR=$HOME/Trilinos12.12/Trilinos-trilinos-release-12-12-1
ARCHDIR=$HOME/XyceLibs/Serial
FLAGS="-O3 -fPIC"
cmake \
-G "Unix Makefiles" \
-DCMAKE_C_COMPILER=gcc \
-DCMAKE_CXX_COMPILER=g++ \
-DCMAKE_Fortran_COMPILER=gfortran \
-DCMAKE_CXX_FLAGS="$FLAGS" \
-DCMAKE_C_FLAGS="$FLAGS" \
-DCMAKE_Fortran_FLAGS="$FLAGS" \
-DCMAKE_INSTALL_PREFIX=$ARCHDIR \
-DCMAKE_MAKE_PROGRAM="make" \
-DTrilinos_ENABLE_NOX=ON \
  -DNOX_ENABLE_LOCA=ON \
-DTrilinos_ENABLE_EpetraExt=ON \
  -DEpetraExt_BUILD_BTF=ON \
  -DEpetraExt_BUILD_EXPERIMENTAL=ON \
  -DEpetraExt_BUILD_GRAPH_REORDERINGS=ON \
-DTrilinos_ENABLE_TrilinosCouplings=ON \
-DTrilinos_ENABLE_Ifpack=ON \
-DTrilinos_ENABLE_AztecOO=ON \
-DTrilinos_ENABLE_Belos=ON \
-DTrilinos_ENABLE_Teuchos=ON \
-DTrilinos_ENABLE_COMPLEX_DOUBLE=ON \
-DTrilinos_ENABLE_Amesos=ON \
  -DAmesos_ENABLE_KLU=ON \
-DTrilinos_ENABLE_Amesos2=ON \
 -DAmesos2_ENABLE_KLU2=ON \
 -DAmesos2_ENABLE_Basker=ON \
-DTrilinos_ENABLE_Sacado=ON \
-DTrilinos_ENABLE_Stokhos=ON \
-DTrilinos_ENABLE_Kokkos=ON \
-DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES=OFF \
-DTrilinos_ENABLE_CXX11=ON \
-DTPL_ENABLE_AMD=ON \
-DAMD_LIBRARY_DIRS="/usr/lib" \
-DTPL_AMD_INCLUDE_DIRS="/usr/include/suitesparse" \
-DTPL_ENABLE_BLAS=ON \
-DTPL_ENABLE_LAPACK=ON \
$SRCDIR

Building Trilinos for Parallel Xyce

Building Trilinos for use in Parallel Xyce requires enabling some additional Trilinos packages (Zoltan and ShyLU). It also requires using the compiler wrappers provided by your system’s MPI package (ex. Open MPI) instead of the compilers (gcc, g++, gfortran) directly. These wrappers are typically called “mpicc,” “mpic++”, and “mpif77” for the C, C++, and Fortran compilers, respectively, but your system may install them with different names or special paths.  “mpicxx” is a common alternate name for the C++ wrapper.  It is also common on systems with case-sensitive file systems to install something called “mpiCC” for the C++ compiler, but we caution against relying on this—you might find yourself building later on a system with a case-insensitive file system, and “mpiCC” will actually be the same as “mpicc”, i.e. the C compiler instead of C++.  This will result in very confusing behavior.

The CMake invocation below is an example invocation that would be appropriate on a Linux system that provides AMD through SuiteSparse. The notes in the serial build instructions also apply to this CMake invocation.

Once the script is created and customized, run it. If it completes without error, Trilinos is configured for building. Build with make and install with make install.

System specific notes:

• On OS X using MacPorts, the OpenMPI wrappers may have additional suffixes such as mpicc-openmpi-gcc5, mpicxx-openmpi-gcc5, and mpif77-openmpi-gcc5.  Other OS X package managers may give these wrappers different names.

Example “reconfigure” script for Parallel Trilinos:

#!/bin/sh
SRCDIR=$HOME/Trilinos12.12/Trilinos-trilinos-release-12-12-1
ARCHDIR=$HOME/XyceLibs/Parallel
FLAGS="-O3 -fPIC"
cmake \
-G "Unix Makefiles" \
-DCMAKE_C_COMPILER=mpicc \
-DCMAKE_CXX_COMPILER=mpic++ \
-DCMAKE_Fortran_COMPILER=mpif77 \
-DCMAKE_CXX_FLAGS="$FLAGS" \
-DCMAKE_C_FLAGS="$FLAGS" \
-DCMAKE_Fortran_FLAGS="$FLAGS" \
-DCMAKE_INSTALL_PREFIX=$ARCHDIR \
-DCMAKE_MAKE_PROGRAM="make" \
-DTrilinos_ENABLE_NOX=ON \
  -DNOX_ENABLE_LOCA=ON \
-DTrilinos_ENABLE_EpetraExt=ON \
  -DEpetraExt_BUILD_BTF=ON \
  -DEpetraExt_BUILD_EXPERIMENTAL=ON \
  -DEpetraExt_BUILD_GRAPH_REORDERINGS=ON \
-DTrilinos_ENABLE_TrilinosCouplings=ON \
-DTrilinos_ENABLE_Ifpack=ON \
-DTrilinos_ENABLE_Isorropia=ON \
-DTrilinos_ENABLE_AztecOO=ON \
-DTrilinos_ENABLE_Belos=ON \
-DTrilinos_ENABLE_Teuchos=ON \
-DTrilinos_ENABLE_COMPLEX_DOUBLE=ON \
-DTrilinos_ENABLE_Amesos=ON \
 -DAmesos_ENABLE_KLU=ON \
-DTrilinos_ENABLE_Amesos2=ON \
 -DAmesos2_ENABLE_KLU2=ON \
 -DAmesos2_ENABLE_Basker=ON \
-DTrilinos_ENABLE_Sacado=ON \
-DTrilinos_ENABLE_Stokhos=ON \
-DTrilinos_ENABLE_Kokkos=ON \
-DTrilinos_ENABLE_Zoltan=ON \
-DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES=OFF \
-DTrilinos_ENABLE_CXX11=ON \
-DTPL_ENABLE_AMD=ON \
-DAMD_LIBRARY_DIRS="/usr/lib" \
-DTPL_AMD_INCLUDE_DIRS="/usr/include/suitesparse" \
-DTPL_ENABLE_BLAS=ON \
-DTPL_ENABLE_LAPACK=ON \
-DTPL_ENABLE_MPI=ON \
$SRCDIR

Download and unpack Xyce source code

All versions of Xyce from release 6.0 to the latest release may be downloaded through this web site on the Downloads page.  To use these, simply download the “.tar.gz” file for the release and unpack it with “tar xzf Xyce-7.6.tar.gz”, executed in an appropriate location on your system.  THIS IS OUR RECOMMENDED DOWNLOAD METHOD FOR MOST USERS.

Beginning with Xyce 6.11, the team is also maintaining a GitHub repository where not only the released version, but interim, possibly unstable versions may be obtained with git.  This repository is a truncated view of the repository used for Xyce development, and contains only the master branch and tagged releases.  Only releases are put through the QA steps needed to certify Xyce for internal use, but the master branch will only contain commits that at least pass all of the test suite on all of our internally supported platforms at the time the merge happened.  Only release versions of Xyce should be used for production work.

Note that if you download Xyce from GitHub instead of our web site, the tar files or git clone you obtain will not have a “configure” script.  This script must be created using the “bootstrap” script present in the top level of the Xyce source tree, which requires that autoconf, automake, and libtool be installed. 

Obtaining Xyce through GitHub

The Xyce team has provided a public GitHub repository to allow more rapid distribution of updated Xyce code than our 6-month release process would allow.  We do not recommend this approach unless you are familiar with git and are willing to accept the consequences of using code that has not been put through the full quality assurance process that Xyce releases require.

You may obtain an initial copy of Xyce from Github by cloning the repository.

cd /some/place/appropriate
git clone https://github.com/Xyce/Xyce.git

This will initially check out the most recent state of the “master” branch of the repository, not any particular release.  The code will be in “/some/place/appropriate/Xyce”

The git repository does not have a copy of the “configure” script that the “tarball” download contains, and in order to proceed to the build you must first “bootstrap” the source code to create it.  This operation requires that you have GNU Autoconf, Automake, and Libtool installed.

 cd /some/place/appropriate/Xyce

./bootstrap

Bootstrap will create the configure script and various “Makefile.in” files needed to build Xyce.  Once this process is complete, you can build Xyce just as described in the following section in the same manner as if you had unpacked it from a tarball.

 NOTE:  It is also possible to download “tarballs” of Xyce source code from the Github “Releases” tab, but these tarballs are not identical to the ones you could download from our web site.  They are just packaged versions of a git repository state, and therefore also lack the “configure” script.  If you download a tarball from Github instead of doing a clone, you still have to run “bootstrap” as described above.

As noted, the initial clone you made was checked out from the latest state of the master branch.  To check out a specific release, you can use the release tag:

   cd /some/place/appropriate/Xyce

  git checkout Release-7.6.0

  ./bootstrap

Each Xyce release has a tag named “Release-<major version>.<minor version>.<patch level>”.  The GitHub repository contains only releases beginning with 6.11.

From time to time you should update your clone from GitHub:

  cd /some/place/appropriate/Xyce
  git checkout master
  git pull
  ./bootstrap

This will cause git to copy all changes that have happened since your last pull.

“bootstrap” is not always required every time you switch branches or update the repository, but it is recommended that you do it anyway just in case the build system has changed in any way.

Building Xyce

Xyce is built using the standard GNU Autoconf/Automake configure system. Most of the work getting Xyce configured involves getting configure to find the required libraries and headers. For Trilinos built according to the directions in this guide, this is primarily accomplished by passing the “ARCHDIR” parameter in to configure, which is used by configure to locate the Trilinos libraries and headers you have installed, in addition to a few extra files that Trilinos creates that simplify linkage.

It may also be necessary to tell configure the location of additional libraries and headers through the LDFLAGS and CPPFLAGS variables.  You also may need to specify which compiler to use, or any special libraries it should link in.

If you download Xyce from GitHub instead of our web site, the tar files or git clone you obtain will not have a “configure” script.  This script must be created using the “bootstrap” script present in the top level of the Xyce source tree, which requires that autoconf, automake, and libtool be installed. 

The Xyce team recommends compiling Xyce “out-of-source.” This is accomplished by creating an empty directory (the “build directory”), making that directory the current working directory, and invoking the Xyce configure script with a complete path name and appropriate options. Once configure is complete, Xyce can be built with “make” and installed with “make install.” The complete process looks like this:

mkdir my_build
cd my_build
/path/to/Xyce/configure [configure options]
make
sudo make install

It is only necessary to use “sudo” with “make install” if you do not have write privileges to the directory into which Xyce will be installed.  The default install location for Xyce will be into subdirectories of /usr/local.  This may be changed by adding the --prefix= option to configure and providing a prefix to use instead of “/usr/local”.  We recommend using a non-default prefix for Xyce installs, so that you can have multiple serial and parallel versions of Xyce installed to different locations.  If you use the default prefix, each install would overwrite files from previous installs.  In the system-specific sections we will assume a target prefix of ‘$HOME/XyceInstalls/Serial’ or ‘$HOME/XyceInstalls/Parallel’ as appropriate.  These assumptions are just suggestions.

For simplicity and repeatability, we recommend that you put the full configure invocation into a file and execute that file as a shell script rather than typing it directly.  We recommend calling this file “reconfigure”, just to make clear what it does: it reruns “configure” for you. Create the file, put “#!/bin/sh” as its first line, then place the configure commands in it.  Make the script executable and run it:

chmod u+x reconfigure
./reconfigure

We recommend that the test suite be run in between the “make” and “make install” steps, to assure that your build is working properly. Consult the Running the Xyce Regression Test Suite page for instructions on running these tests.

The remaining sections below detail the configure options most likely to be required. A full list of configure’s options may be obtained by typing:

/path/to/Xyce/configure --help

In the first section below provides example configure invocations for selected systems the Xyce team has built on.  In the next section, we describe the basic options that can be used to control configure should the example configurations not be exactly what you need. If your system is one of the ones for which we have examples, you may want to skip the section on basic options.

NOTE:  Some open-source users have noticed that the Xyce source tree contains a set of “CMakeLists.txt” files and have tried to use CMake instead of autoconf to build Xyce.  These are used for a new, experimental CMake build that will replace autotools in a future release.  If you prefer to use CMake, please stop reading this document and read the INSTALL.md file in the Xyce source tree, which documents the alternative build system.  The autotools build remains the primary recommended build system for this release as it is the most thoroughly tested at this time.

Note on memory requirements

Building Xyce can require a great deal of memory and disk space when compiled with full optimization.  Generally speaking, when it comes to memory available for compiling Xyce, more is better.

The most challenging part of Xyce to compile is the contents of the “src/DeviceModelPKG/ADMS” directory, which contains some device models that have some very large, complex functions, and therefore take a great deal of processor time and memory to optimize fully. 

Note:  The Verilog-A compiler used to generate the models in the src/DeviceModelPKG/ADMS directory was refactored for the 7.2 release of Xyce, and produces code that requires far less time and memory to compile than in all previous releases.  Xyce 7.2 has been compiled without resorting to the solutions below on a Raspberry Pi with only 2GB of RAM.  We are retaining the text below for completeness, but they should now seldom be needed.

This can be a problem when attempting to build on very small systems, such as Raspberry Pi, which have no more than a gigabyte of memory (and far less in older versions). The ideal solution for such a system, resulting in fully optimized, fully capable builds of Xyce, would involve cross-compiling Xyce on a larger system with the smaller system as target. This solution can be hard to realize, and the Xyce team can’t help you with it.

Even without the extreme restrictions of tiny systems such as Raspberry Pi, compiling Xyce can be very demanding of memory. The requirements are very much dependent on the compiler in use.

We have found that with a modern laptop with 16GB of ram and eight cores, Xyce will compile reasonably quickly using the GCC 5.4.0 compiler suite using “make -j8”, but on the same system using the Intel C++ compiler (version of 2018) some of the files in the ADMS directory can cause the compiler to consume nearly 4GB each, and one of the files (the BSIM-CMG 110) takes nearly an hour to compile with this compiler.  All of the systems on which the Xyce project builds its binaries with the Intel compiler have much more RAM than this.

However, if you need to build Xyce on a system without sufficient memory, there are ways to reduce the memory usage at compile time. Doing so will either lead to increased compile times or result in a less optimized build of Xyce, but may be the only path forward for users of very small systems.

If you find your Xyce build process aborting due to insufficient memory, there are some techniques you can try to get around the memory limitation.  The suggestions below are listed roughly in order of increasing impact on the quality of the final build, and the more severe your memory restriction the more likely you’ll have to go further down the list to find the one that works.

• Reduce the number of jobs used by make
  If your computer has many cores, you may be tempted to run make with a “-j” option telling it to compile as many files simultaneously as you have cores.  In some cases you may then find your compilation crashing (or being killed by a Unix signal) during the processing of the ADMS directory, because the extreme memory footprint of the full optimization of even one of these files could be consuming all the memory on your system.  If this is the case, you can often get past the problem by reducing the number of jobs requested by “make -j”.   This can be accomplished in two ways, listed here in order of increasing impact on build time:
  • Build the ADMS directory with “make -j1” (or using no -j option)
    After running configure, simply “cd” into the src/DeviceModelPKG/ADMS subdirectory of your build directory and run make there with the smaller (or no) -j option.  This will cause each file in the ADMS directory to be compiled sequentially instead of all at once, and if your system has enough memory to compile even one of these files, then your build won’t crash, and this option will still result in your having a fully optimized Xyce at run time.  Once the ADMS directory is finished compiling, you can return to the top level of your build directory and run “make -j” with a larger number of jobs.
    
    The Xyce team has observed this technique to be adequate to build Xyce with GCC 4.8 on a laptop with as little as 4GB of RAM. 
  • Use “make -j1” for all of your build
    If you do not want to do your build in two “make” invocations, you could simply run make with no “-j” or with “-j1” for the entire build process.  This will result in significant delay in compiling the many files present in the Xyce source tree, but it will work.  Since the part of Xyce that has the highest probability of running out of memory is the ADMS directory, you should certainly try the previous option first, as it will be much faster than this option.
• Reduce the optimization level of models derived from Verilog-A sources
  Xyce contains a number of modern device models that have been translated from Verilog-A to C++. These models have some very large functions that are intended for use in sensitivity analysis. 
  
  Optimization of these models using very aggressive optimization levels such as “-O3” can consume a tremendous amount of RAM. To deal with this, the build system provides a means of reducing the optimization level of models contained in the ADMS subdirectory of the DeviceModelPKG directory. This is done by setting the “ADMS_CXXFLAGS” variable at configure time to something that differs from the normal CXXFLAGS used for the rest of the code. For example, if you have CXXFLAGS=”-O3″ on your configure line, you can force the ADMS directory to use a different optimization level by adding ADMS_CXXFLAGS=”-O1″ to the configure line. Most of the code will be compiled with the -O3 optimization level, and the ADMS devices will be compiled with only -O1. 
  
  Prior to the refactor of the ADMS Verilog-A compiler, this change was observed to be enough to get Xyce compiled on systems with around 2GB of RAM so long as only “-j1” is given to make.  Since that refactor, we have observed that building the ADMS directory with -O1 may reduce the memory needs of the ADMS directory enough even to compile on systems with less than 2GB of RAM.
• Turn off analytic sensitivity derivative calculation for Verilog-A-derived devices
  Xyce supports sensitivity analysis using analytic derivatives if the models have that capability.  If models do not provide analytic derivatives, Xyce will use finite differences to compute the derivatives needed for sensitivity analysis.  Finite differences will usually be less accurate than analytic derivatives. 
  
  Unfortunately, providing these analytic derivatives in models derived from Verilog-A requires generation of special purpose functions that are much larger than those used for other analyses, and drastically increases memory usage at build time. It also significantly increases compile time as the compiler struggles to optimize these huge functions.
  
  If simple reduction of compiler optimization level is not sufficient to get Xyce built on your small-memory system, you can try disabling analytic derivatives for these models. This is done by adding --disable-adms_sensitivities to your configure command line. The result will be that all the code that computes analytic sensitivities will be ifdeffed out, and the compilation will generally proceed much faster and with much lower memory use. This will have no impact on you if you are not using sensitivity analysis using these models, but may have impact on accuracy and speed if you are using this analysis.  Two of the tests in the Xyce Regression suite will fail because they are explicitly written to require analytic derivatives (and you can ignore those failures).
  
  This method has been successfully applied to get Xyce to build on a Raspberry Pi 2, which has only 1GB of RAM.
• Reduce the optimization level for all of Xyce
  If neither of the above approaches reduces the memory usage at compile time enough to get Xyce to build on your system, and the compilation aborts due to memory exhaustion in something other than the ADMS subdirectory, you might have to reduce optimization level for everything. This is done by inserting the appropriate optimization level flag into CXXFLAGS on the configure line. Doing this will definitely impact performance of the code. It will not help at all if the compilation is aborting while compiling the ADMS subdirectory and the previous two approaches didn’t help. In that case, your only option is to build on a machine with more memory.

Example configure invocations for selected systems

Here we present example invocations of configure for selected systems.

Because multiple versions of Trilinos might need to be built to support both serial and parallel builds of Xyce, we do not recommend installing Trilinos directly into system library and header directories (/usr/lib, /usr/include, etc.). Therefore, in each example we have assumed that Trilinos (and any other handbuilt libraries) have been installed into a directory called $HOME/XyceLibs/Serial/lib, and their associated headers into $HOME/XyceLibs/Serial/include. In the case of Trilinos, that can be accomplished by setting “ARCHDIR” to $HOME/XyceLibs/Serial in the CMake invocation script described earlier.  In the examples below, you should set ARCHDIR to the location where Trilinos was installed, i.e. the same value you used for ARCHDIR when building and installing Trilinos.

The configure invocations below are presented separately for serial and parallel builds.

Serial builds:

• Linux
• OS X
• FreeBSD
• Cygwin

Parallel builds

• Linux
• OS X
• FreeBSD

Serial Builds

Linux

If all packages have been installed as recommended, configuring on Linux is fairly simple. This is because the package manager installs all of its packages in /usr/lib and /usr/include, and these are searched by configure and the compilers with no extra options required.

The example configure invocation below should be appropriate for any distribution of Linux, though it has only been tested specifically on Red Hat, Ubuntu, and Linux Mint variants.  Minor modifications may be necessary on other distros.

/path/to/Xyce/configure \
CXXFLAGS="-O3" \
ARCHDIR="$HOME/XyceLibs/Serial" \
CPPFLAGS="-I/usr/include/suitesparse" \
--enable-stokhos \
--enable-amesos2 \
--prefix=$HOME/XyceInstall/Serial

Here, we have not had to specify anything more than the extra directories to search for SuiteSparse headers, and the location into which Trilinos was installed.  We have added two commands to enable optional features that will soon become default features in a future release.

Mac OS X

If you have installed XCode, Fink, and all the Fink packages we have recommended, and installed Trilinos in /home/me/XyceLibs/Serial (per our example), building Xyce on Mac OS X in serial can be accomplished with the following configure invocation:

/path/to/Xyce/configure \
ARCHDIR="$HOME/XyceLibs/Serial" \
CXXFLAGS="-O3" \
LDFLAGS="-Wl,-framework,Accelerate -L/sw/lib" \
CPPFLAGS="-I/sw/include/suitesparse -I/sw/include" \
CXX=g++9 \
CC=gcc9 \
F77=gfortran9 \
--enable-stokhos \
--enable-amesos2 \
--prefix=$HOME/XyceInstall/Serial

Note:  The exact name of gcc, g++, and gfortran may vary depending on what package you installed. 

For a MacPorts installation, use the following:

/path/to/Xyce/configure \
ARCHDIR="$HOME/XyceLibs/Serial" \
CXXFLAGS="-O3" \
LDFLAGS="-Wl,-framework,Accelerate -L/opt/local/lib" \
CPPFLAGS="-I/opt/local/include" \
CXX=g++-mp-10 \
CC=gcc-mp-10 \
F77=gfortran-mp-10 \
--enable-stokhos \
--enable-amesos2 \
--prefix=$HOME/XyceInstall/Serial

Note:  The exact name of gcc, g++, and gfortran may vary depending on what package you installed. 

Finally, if you are using homebrew as your package manager, you can configure Xyce with:

/path/to/Xyce/configure \
ARCHDIR=$HOME/XyceLibs/Serial \
CXXFLAGS="-O3" \
LDFLAGS="-Wl,-framework,Accelerate -L/usr/local/lib -L/usr/local/opt/flex/lib -L/usr/local/opt/bison/lib" \
CPPFLAGS="-I/usr/local/include -I/usr/local/opt/flex/include -I/usr/local/include" \
LEX=/usr/local/opt/flex/bin/flex \
BISON=/usr/local/opt/bison/bin/bison \
CC=/usr/local/bin/gcc-10 \
CXX=/usr/local/bin/g++-10 \
F77=/usr/local/bin/gfortran-10 \
--enable-stokhos \
--enable-amesos2 \
--prefix=$HOME/XyceInstall/Serial

Note:  The exact name of gcc, g++, and gfortran may vary depending on what package you installed.  The paths given above for homebrew were the names installed by the gcc package at the time of this writing, but may change.

We have had reports that newer editions of homebrew now install to /opt/homebrew/lib and /opt/homebrew/include. If you have one of these newer installs you must replace /usr/local with /opt/homebrew in the configure invocation above.

The -Wl,-framework,Accelerate option in LDFLAGS tells configure to link in the optimized libraries provided by XCode. These libraries include BLAS and LAPACK, so when configure starts to look for those routines it will find them without having to search for libraries like libblas or liblapack.

While we have used gcc in the above examples, the system clang compiler could also be used to compile Xyce (assuming it was used to compile Trilinos as well). If clang is used to build Xyce then gfortran must still be installed and used, since clang does not have a native Fortran compiler. (If clang is used, replace Wl,-framework,Accelerate with -framework Accelerate.)NOTE: While numerous Xyce developers have built the code with exactly the configure options shown above, it has been observed that in some combinations of package managers and XCode versions the build process has problems finding the Fortran versions of BLAS and LAPACK. When this happens, there will be a large number of link errors at the end of the build, reporting “undefined symbols for architecture” referenced by functions in Teuchos (a Trilinos package). Should this occur, one can add the --disable-fortran_blas option to the configure invocation to make it use C versions of BLAS and LAPACK library functions instead. Re-run configure with this option added and recompile.

FreeBSD

If all libraries and packages have been installed as described above, serial Xyce can be configured on FreeBSD with the following configure invocation.

/path/to/Xyce/configure \
CXXFLAGS="-O3" \
LEX=/usr/local/bin/flex \
ARCHDIR="$HOME/XyceLibs/Serial" \
LDFLAGS="-L/usr/local/lib -Wl,-rpath=/usr/local/lib/gcc10" \
CPPFLAGS="-I/usr/local/include" \
CXX=clang++ \
CC=clang \
F77=gfortran10 \
--enable-stokhos \
--enable-amesos2 \
--prefix=$HOME/XyceInstall/Serial

In this example, we have used the system Clang compiler.  Because the system contains no Fortran compiler, it is necessary to use the gfortran10 compiler and specify an “rpath” in the CXXFLAGS so that the resulting binary will always reference the correct Fortran standard library installed by the  “gcc10” port.

Xyce will also build with the port-provided gcc10 and its associated gfortran10.  Make the obvious change to the rpath above if you choose to use it.

Cygwin

If all libraries and packages have been installed as described above, the following configure invocation will configure Xyce to be built under Cygwin:

/path/to/Xyce/configure \
ARCHDIR="$HOME/XyceLibs/Serial" \
CXXFLAGS="-O2" \
CPPFLAGS="-D_BSD_SOURCE -D_GNU_SOURCE" \
CXX=g++ \
CC=gcc \
--enable-stokhos \
--enable-amesos2 \
--prefix=$HOME/XyceInstall/Serial

While it is tempting to let configure guess the C++ and C compilers as with Linux, doing so causes problems in the link stage, when for some reason libtool is invoked using the C compiler as linker instead of C++ (it appears that the reason for this is that configure finds “CC” as a valid C++ compiler, but when this name is passed to libtool it is interpreted to mean “C mode”). This results in numerous inexplicable “undefined references” to standard C++ library functions. Explicitly specifying g++ and gcc as the compilers avoids this problem.

NOTE: if you are creating a “reconfigure” script file containing the configure invocation above, it is essential that you do so using a Unix-style editor under Cygwin (vim, nano, emacs, etc.) and not a Windows text editor such as Notepad. Notepad will create a file using DOS style line breaks (“”CR/LF”), and this will confuse the bash shell when running the script, and the error messages it emits will not indicate the true source of the problem.

Parallel Builds

Most of the complexity of building Xyce in parallel on any platform is in getting the third-party libraries built. If you have all the required packages for Trilinos built, and have properly built Trilinos using MPI compiler wrapper scripts, then configuring Xyce is usually just a matter of providing the same wrapper scripts for compilers and adjusting the configure line to point at the parallel versions of libraries.

In addition to naming the MPI compiler wrappers and identifying library directories, one must add --enable-mpi to the configure options.

Linux

The example configure invocation below should be appropriate for any distribution of Linux, though it has only been tested specifically on Red Hat, Ubuntu, and Linux Mint variants.  Minor modifications may be necessary on other distros.

/path/to/Xyce/configure \
CXXFLAGS="-O3" \
ARCHDIR="$HOME/XyceLibs/Parallel" \
CPPFLAGS="-I/usr/include/suitesparse" \
--enable-mpi \
CXX=mpicxx \
CC=mpicc \
F77=mpif77 \
--enable-stokhos \
--enable-amesos2 \
--prefix=$HOME/XyceInstall/Parallel

Mac OS X

Fink:

/path/to/Xyce/configure \
CXXFLAGS="-O3" \
ARCHDIR="$HOME/XyceLibs/Parallel" \
LDFLAGS="-Wl,-framework,Accelerate -L/sw/lib" \
CPPFLAGS="-I/sw/include/suitesparse -I/sw/include" \
--enable-mpi \
CXX=mpicxx \
CC=mpicc \
F77=mpif77 \
--enable-stokhos \
--enable-amesos2 \
--prefix=$HOME/XyceInstall/Parallel

MacPorts:

/path/to/Xyce/configure \
CXXFLAGS="-O3" \
ARCHDIR="$HOME/XyceLibs/Parallel" \
LDFLAGS="-Wl,-framework,Accelerate -L/opt/local/lib" \
CPPFLAGS="-I/opt/local/include" \
--enable-mpi \
CXX=mpicxx-openmpi-gcc6 \
CC=mpicc-openmpi-gcc6 \
F77=mpif77-openmpi-gcc6 \
--enable-stokhos \
--enable-amesos2 \
--prefix=$HOME/XyceInstall/Parallel

Change the compiler name above to match the one installed by MacPorts.  The command above is only an example.

Homebrew:

Note that, with Homebrew, Open MPI wrapps the native clang compiler.

$HOME/Downloads/Xyce/configure \
CXXFLAGS="-O3" \
ARCHDIR="$HOME/XyceLibs/Parallel" \
LDFLAGS="-Wl,-framework,Accelerate -L/usr/local/lib -L/usr/local/opt/flex/lib" \
CPPFLAGS="-I/usr/local/include -I/usr/local/opt/flex/include" \
LEX=/usr/local/opt/flex/bin/flex \
BISON=/usr/local/opt/bison/bin/bison \
--enable-mpi \
--enable-stokhos \
--enable-amesos2 \
CXX=/usr/local/bin/mpicxx \
CC=/usr/local/bin/mpicc \
F77=/usr/local/bin/mpif77 \
--prefix=$HOME/XyceInstall/Parallel

FreeBSD

/path/to/Xyce/configure \
CXXFLAGS="-O3 -Wl,-rpath=/usr/local/lib/gcc10" \
LEX=/usr/local/bin/flex \
ARCHDIR="$HOME/XyceLibs/Parallel" \
LDFLAGS="-L/usr/local/lib" \
CPPFLAGS="-I/usr/local/include" \
--enable-mpi \
--enable-stokhos \
--enable-amesos2 \
CXX=/usr/local/mpi/openmpi/bin/mpicxx \
CC=/usr/local/mpi/openmpi/bin/mpicc \
F77=/usr/local/mpi/openmpi/bin/mpif77 \
--prefix=$HOME/XyceInstall/Parallel

Note the required “rpath” is still there for FreeBSD, because the MPI wrappers will have the same gfortran shared library dependence that the serial build does.  It is assumed here that you have installed gfortran from the GCC 10 compiler collection prior to building OpenMPI, and that your OpenMPI is built with the default compiler your version of FreeBSD will choose for them (which will be generally be clang on FreeBSD).

Note: It has been observed that in recent versions of FreeBSD, OpenMPI, and certain versions of hwloc2, a nonprivileged user will get a “permission denied” warning from mpirun announcing “Failed to initialize libpciaccess with pci_system_init()”.  This warning is coming from the hwloc package and is due to that package trying to perform a full scan of the PCI system for your computer’s topology, and on BSD this requires administrator privilege.  This issue is a known issue in hwloc on BSD as reported here.  The warning is harmless and the run will work, but there may be some performance hit at startup.  One can avoid the hwloc probe and this warning by getting a system administrator to create an XML file of the system topology, then setting an environment variable to make hwloc use the XML file instead of the libpciaccess.  This workaround is documented in the github issue linked to earlier in this paragraph.  (In the week prior to publication of this document the FreeBSD ports system was updated to hwloc 2.5.0, which no longer attempts the full PCI system scan and does not produce this warning).

Generic Configure Options

If the selected example configurations above are not sufficient to get Xyce built on your system, you may need to tell configure a little more about your system’s specifics.  The general information below documents how to tell configure about some typical system-specific information that may impede portability.

Specifying compilers and helper programs

“configure” usually tries to find the C, C++, and Fortran compilers from a well-known list of standard compiler names. If your system does not have one of the standard-named compilers, or you wish to use some other compiler, you can specify them with the configure variables CC, CXX and F77. It is important to use the same compilers to compile Xyce that you used to compile the Trilinos libraries. Additional variables also allow you to specify particular versions of helper programs, like the Flex lexical analyzer generator (using the LEX variable). For example:

/path/to/Xyce/configure \
CC=strangeC \
CXX=strangeC++ \
F77=strangeF77 \
LEX=/usr/local/bin/flex

Here we have used the shell’s continuation mechanism, and placed a backslash at the end of each line other than the last. It is important that no spaces appear after the backslashes.  There must also be a space before the “\”.

Specifying Xyce-specific library locations

We recommend installing Trilinos, built especially according to the directions in this guide, into a separate directory.  In the earlier section on building Trilinos, we instructed you to use a variable called “ARCHDIR” in your CMake invocation for Trilinos.  Xyce’s configure script also has a variable called “ARCHDIR” that can be used not only to find these libraries and headers, but also some extra files that help determine interdependencies between Trilinos packages and link in all required Trilinos libraries.  Simply use the same install directory you gave to the Trilinos CMake script (ARCHDIR) as the value of Xyce’s configure ARCHDIR:

/path/to/Xyce/configure ARCHDIR=$HOME/XyceLibs/Serial [other configure options]

This will result in $HOME/XyceLibs/Serial/lib being added to LDFLAGS and $HOME/XyceLibs/Serial/include being added to CPPFLAGS, as well as ensuring the interdependent Trilinos packages are properly linked.

We make this recommendation because Trilinos must be built differently for parallel builds of Xyce than it must be for serial builds.  Keeping these two builds in completely separate installation directories simplifies the process of having both versions of Xyce built on the same system.

Specifying compiler flags

Each compiler has a configure variable for passing it parameters. These variables are CXXFLAGS for C++ compiler flags, CFLAGS for C compiler flags, and FFLAGS for Fortran compiler flags:

/path/to/Xyce/configure \
CC=gcc9 \
CXX=g++9 \
F77=gfortran9 \
LEX=/usr/local/bin/flex \
CXXFLAGS="-O3" \
CFLAGS="-O3"

Specifying alternate installation directory

By default, configure will set up the Xyce build process so that “make install” installs Xyce components into subdirectories /usr/local: the executable will go in /usr/local/bin, associated libraries into /usr/local/lib, and header files into /usr/local/include.  This behavior can be changed by adding the --prefix option to configure:

path/to/Xyce/configure \
--prefix=/opt/Xyce \
...

The example above will cause “make install” to install into /opt/Xyce/bin, /opt/Xyce/lib, and /opt/Xyce/include.

If you do not have root privileges on your system, this procedure can allow you to install Xyce into a directory that you own instead.  Otherwise, use “sudo” with “make install” to install Xyce into system directories.

The Xyce Team STRONGLY recommends that you always specify a --prefix for Xyce builds, and not use the default “/usr/local.”  Installing both serial and parallel versions of Xyce directly into /usr/local will cause the two versions to overwrite some components and leave one or the other unusable.

Specifying additional library search directories

If you have prerequisite libraries that are installed in any directory that the linker will not search by default, you can direct configure to search them by adding “-L” flags to the variable “LDFLAGS”:

/path/to/Xyce/configure \
LDFLAGS="-L/some/random/directory/lib -L/home/me/archdir/ZipChip2000/lib"

Specifying additional include search directories

It is sometimes the case that system packages install headers in subdirectories of /usr/include or /usr/local/include. The C preprocessor (and therefore configure) will not search for include files outside standard locations. If header files are installed in any directory that configure will not search by default, you can direct configure to search them by adding “-I” flags to the variable “CPPFLAGS”:

/path/to/Xyce/configure \
CPPFLAGS="-I/usr/local/include/obscurelib"

Usage Tracking Code in Xyce

If you browse the Xyce source code, you will see that beginning with version 6.2 Xyce has code to enable usage tracking.  Some users may be concerned about this, so this section documents the code, how to enable it, how to disable it, and what it does.

Usage tracking was implemented in Xyce so that we could track our internal users’ use of the code to better serve them, and to report to management on how heavily our software is being used in-house.

Usage tracking is disabled by default

The usage tracking code in Xyce is not compiled in unless specifically requested, and when a URL for usage data collection provided.  The open-source binary that we provide on the external web site does not have usage tracking enabled.

Only binaries intended strictly for internal use have usage tracking turned on.  In these binaries, “Xyce -capabilities” will report “Metrics reporting via cURL”.

What usage data is collected?

No open source binaries have usage tracking turned on, and usage tracking is not built by default.  No usage data is collected from these builds.

If your binary is one that was intended for purely internal use at Sandia, “Xyce -capabilities” will report “Metrics reporting via cURL,” and each run of Xyce will result in a small packet of data about your run sent to an internal web server.  This web server is inside Sandia’s firewall and cannot receive data from outside Sandia.

The data packet contains

• Your user ID.
• The name of the machine on which you are running,
• The hardware type (x86, amd64, etc.),
• Your OS (Windows, Linux, etc.) as reported by a system call,
• The version number of Xyce you are running,
• The total count of devices in your circuit,
• A breakdown of the number of devices of each type in your circuit,
• Counts of operations as reported in the console output summary (number of linear solves, number of time steps, number of residual and jacobian loads, etc.),
• The start and end times of the run.

Your user ID is used to determine your Org number and Sandia UID by the web server that collects the data, and this information is added to the packet.

Turning off usage reporting

If you are running a binary that includes usage reporting and you wish to opt out of this reporting, set the environment variable “XYCE_NO_TRACKING” to any value.  If this environment variable is set, Xyce will skip sending the usage data.  Remember that no open source builds have this feature enabled, and that you can check if the feature is enabled with “Xyce -capabilities”.

On Unix-like systems

export XYCE_NO_TRACKING="anything at all"

On Windows

set XYCE_NO_TRACKING="anything"

If you are running Xyce at Sandia, we ask that you only opt out of usage reporting if absolutely necessary.  We will use this information to document how widely Xyce is used inside Sandia to management. 

Turning on usage tracking

The usage tracking URL at Sandia is inaccessible outside Sandia, and therefore we cannot collect information from open source users.

If you are building Xyce for institutional use and wish to collect usage data yourself, you can turn on usage tracking easily.  You must provide your own web server and collection script.  If you are a Sandia user building Xyce from source, please contact the Xyce team and ask for the URL to use to enable usage tracking.  Counting uses of Xyce inside Sandia is important to the Xyce team.

• To turn on usage tracking, you must have the cURL library installed on your system.  Most Linux systems have this package available in their package repositories.  You will require both the library package and the “development” package to build Xyce with this feature.  On Windows, you must build cURL from source.  See the cURL web site for details if building cURL from source.
• add “--enable-curl” to your configure options when building Xyce
• set the TRACKING_URL variable to the URL of your tracking script on the configure command line:
  • /path/to/configure […options…] --enable-curl  TRACKING_URL=”http://mysite.org/directory/cgi-bin/trackingscript.pl” […more options…]

The data sent to the tracking URL is a single line of JSON-encoded data. 

A trivial script that simply accepts the data and stores it in a file can be used, or you can write a more elaborate one yourself:

#!/usr/bin/env perl
use CGI;

my $query=CGI->new;
my %params=$query->Vars;
my $json_text=$params{POSTDATA};
open(STATSFILE,">>/path/to/writable/statsfile");
print STATSFILE "$json_text\n";
close(STATSFILE);

print $query->header(),$query->start_html("Gotcha");
print $query->h1("got it");
# This line is useful for debugging, but if the run has a lot of data to report
# might create an unreasonably large response.  Comment out for production.
#print $query->p($json_text);
print $query->end_html();