SCONS
|
|
SCONS
|
|
| |
| Working groups |
| Blessed plots and figures |
| Approving new results and publications |
| Approval web pages - new results |
| Approval web pages - new publications |
| Mu2e Acronyn Dictionary |
| Fermilab Meeting Rooms |
| Fermilab Service Desk |
| ReadyTalk : Home |
| ReadyTalk : Help |
| ReadyTalk : Toll Free Numbers |
> scons -j 4 --mu2elevel=debugthe other option is "prof" which is the standard non-debug build. This will change in mid-July 2014. Pay attention for the announcement.
One particularly nice feature is that SCons, itself, keeps track of dependencies on included header files whereas make and its derivatives required a separate tool for this. From the SCons web site:
SCons is an Open Source software construction tool that is a next-generation build tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software.We have also considered using cmake instead of scons. This is discussed below.
> scons -j n
where n is number; at the times during the build when it is possible for several things to be done in parallel, scons will use up to n threads. Our experience is that, on a lightly used machine, n should be about about 1.5 to 2 times the number of available cores. Why more than 1 times the number of coress? This keeps the cores busy while some of the jobs are waiting for disk IO to complete. Typical desktop machines these days have 2 cores; mu2egpvm* machines have 4 cores and detsim has 32 cores. In the Mu2e instructions it suggests using -j 4 on mu2egpvm* or detsim; this smaller number was chosen because you will rarely be the only user on the machine. In particular, if we run a tutorial and 10 people all decide to bulid at the same time, the machine will slow to a crawl. Even two people doing scons -j 16 will make detsim unusable by anyone else.
If you change any header files you should do a full rebuild: scons -j 4
There is really only one safe case for doing partial rebuilds. Suppose that you already have a full, clean build and are repeatedly editing several files in MyPackage/src. For example,
MyPackage/src/My_module.cc MyPackage/src/File1.cc MyPackage/src/File2.ccNote that there are no header files in this list. You can build just these files by issuing the command:
scons -j 4 lib/libMyPackage.so lib/libMy_module.soThis will only consider the following files for rebuilding
sconsscons must traverse the entire code base to determine which files need to be recompiled and which libraries need to be rebuilt. When you run
scons target_namescons only needs to check the dependency tree of target_name.
You can also do a partial clean up,
scons -c -j 4 lib/libMyPackage.so lib/libMy_module.so
This section describes a feature that is very useful when used correctly but very dangerous when used incorrectly. Read to the end of this section to learn about when the dangers arise.
When you run scons, the first thing that it does is to scan the code base to build a dependency tree. This dependency tree encodes the information: suppose I change a particular file, what other files need to be recompiled or relinked? When scons is finished, it saves the dependency tree in the .sconsign.dblite file (found in the same directory as the SConstruct file).
The default behaviour of scons is that it will not trust the cached dependency tree and will always recompute it. It does this every time that your run scons. If you know that your changes did not change any of the dependency information, then you can tell scons to trust it's cached dependency tree. For example,
scons -j 4 --implicit-deps-unchangedYou can also combine this with partial builds. Suppose that you have editted only Analyses/src/ReadBack_module.cc, you can rebuild it with:
scons -j 8 --implicit-deps-unchanged lib/libmu2e_Analyses_ReadBack_module.soYou can read more about this at: http://www.scons.org/doc/2.3.4/HTML/scons-user/ch06s04.html. You can use the Prev and Next buttons on that page to read more about managing dependencies in scons.
Here are the results of some real-life tests for builds in which everything was up to date so scons did not have to build anything, just check for work to do. The table below gives results for running on mu2egpvm04, using code very close to Offline v5_4_6 and for running on detsim using Offline v5_4_6:
mu2egpvm04 scons -j 8 105s scons -j 8 --implicit-deps-unchanged 45s detsim scons -j 20 84s scons -j 20 --implicit-deps-unchanged 44s scons -j 4 --implicit-deps-unchanged 41s scons -j 1 lib/libmu2e_Analyses_ReadBack_module.so 19s scons -j 1 lib/libmu2e_Analyses_ReadBack_module.so --implicit-deps-unchanged 12s scons -j 20 lib/libmu2e_Analyses_ReadBack_module.so 19s scons -j 20 lib/libmu2e_Analyses_ReadBack_module.so --implicit-deps-unchanged 12sThe bottom line is that --implicit-deps-unchanged offers a speedup between about a factor of 1.5 and 2.5. But the times are still slow enough to be noticeable.
What is the dangerous situation? If you have made edits that modify the dependency tree, you use must not use the --implicit-deps-unchanged option. If you do, you may get an inconsistent build and your code may not execute with subtle, hard to detect errors. As a rough guideline, you must not use this option if you have done any of the following:
After you check out Mu2e software, in the top level directory you will find a file named SConstruct. This defines rules that are common to building all parts of the Offline software.
In each Mu2e PackageName/src directory you will find a file named SConscript. This provides additional information that is needed to build the files in that src/ directory; usually this additional information is only the list of link-time library dependencies, which differ from library to library. Most of the SConscript files are identical or differ only by their link lists; one notable exception is Mu2eG4/src/SConscript which contains the rules needed to find the G4 header files, the G4 compiler -D options and the G4 libraries.
In the following, pay careful attention to whether the sentence is describing the SConstruct file ( the unique file at the top level ) or the many SConscript files.
The SConstruct file defines the information that is needed globally, such as the include paths (-I), the link library paths (-L), compiler switches, linker switches, and the commands needed to build dictionaries. Most of the SConscript files only specify the first order link dependencies of the libraries to be built. A few SConscript files also specify compiler flags, link flags, additional -L and additional -I flags that are needed only for that particular directory.
Some of you may recognize that the SConstruct and SConscript files are python scripts (python is a widely used scripting language). Most Mu2e users will not need to learn python to use SCons; we can think of the SConstruct and SConscript files as just configuration files with a small grammar that we need to learn.
import os
import sys
# Check that the release-specific setup has been run.
if not os.environ.has_key('MU2E_BASE_RELEASE'):
sys.exit('You must setup a Mu2e base release before running scons.\nExiting.')
# Extract information from the shell environment.
art_inc = os.environ['ART_INC']
art_lib = os.environ['ART_LIB']
base = os.environ['MU2E_BASE_RELEASE']
boost_lib = os.environ['BOOST_LIB']
boost_inc = os.environ['BOOST_INC']
clhep_inc = os.environ['CLHEP_INC']
clhep_lib = os.environ['CLHEP_LIB_DIR']
The first two lines tell scons to import information about the local operating
system and operating environment; this includes all of the environment variables defined
at the time of the call to scons. The next line checks that one of the required environment
variables is present; if you are using the standard mu2e setup scripts this is a sufficient
check that you have run the correct scripts. The remaining lines copy some environment variables
into variables that are local to SConstruct; these lines have been truncated.
The section that has an if branch for test releases will be discussed later.
The next major section of SConstruct has the following structure. Some of the long lists have been truncated to make it easier to see the gross structure:
env = Environment( CPPPATH=[ cpppath_frag,
base,
base+'/BaBar/include',
art_inc,
... deleted lines
],
LIBPATH=[ libpath_frag,
base+'/lib',
art_lib,
... deleted lines
],
ENV={ 'LD_LIBRARY_PATH': os.environ['LD_LIBRARY_PATH'],
'GCC_FQ_DIR' : os.environ['GCC_FQ_DIR'], # For GCCXML
... deleted lines
}
)
... lots of stuff deleted
Export('env')
The command env=Environment(...) creates a new python variable named env that contains
environment information; think of this as a list of environment variables as is available in
a standard unix shell. Initially the variable env is local to SConstruct. The command
Export('env') makes env visible to all of the SConscript files.
The above code fragment adds three enviroment variables to env,
CPPPATH, LIBPATH and ENV. Note the env and ENV are distinct.
CPPPATH and LIBPATH have special meaning to SCons and both have the meaning that you
might expect from previous experience with make: CPPPATH is a list of directories that
the C-Preprocessor (CPP) will use to resolve #include directives LIBPATH is a list of
directories in which the linker will look to find libraries. The variable ENV contains
copies of some information from the environment in the shell that called SCons; it does
not have special meaning to SCons but will be used by the SConscript files.
The philosophy behind ENV is to discourage writers of SConscript files from directly accessing the external environment but instead to get information about the calling environment via ENV. In this way, SConstruct can augment or override information found in the calling environment.
SConscript files may add elements to CPPPATH and LIBPATH and the syntax for this will be discussed later. Actually the SConscript files can delete or totally redefine CPPPATH or LIBPATH but I cannot imagine a situation in which that would be useful.
Before describing the CPPPATH in detail, you need to know one more thing: scons issues all of its commands from the directory from which you invoked scons. When scons compiles, for example, the file EventGenerator/src/ConversionGun.cc, it will issue the command:
> g++ -o EventGenerator/src/ConversionGun.os {flags} {-Ixxx -Iyyy ... } EventGenerator/src/ConversionGun.cc
What is a .os file? SCons anticipates that some people will want to build both static (.a) libraries
and shared (.so) libraries from the same source files. Object files destined for the two libaries usualy
require different compiler options ( such as -fpic for shared libaries ); to keep the bookkeeping straight,
if an object file is destined for a .a library, it is named .o; on the other hand, if an object file
is destined for a .so library, scons names it .os.
Now look in a little more detail at CPPPATH:
CPPPATH=[ cpppath_frag,
base,
base+'/BaBar/include',
art_inc,
mesfac_inc,
fhicl_inc,
cetlib_inc,
cpp0x_inc,
boost_inc,
clhep_inc,
The environment variable CPPPATH is a list of directory names; For each name in the list, scons will add a -I
element to the compile command. Most of the lines defining CPPPATH are local variables within the SConstruct
python script and must be defined. The exception is cpppath_frag which is itself a list of directory names;
if that list happens to be empty cpppath_frag adss nothing to CPPPATH; if it non-empty it adds all of its
elements to CPPPATH.
The meaning of cpppath_frag is the following. If one checks out and builds a complete release of Offline, then the environment variable MU2E_BASE_RELEASE is defined but MU2E_TEST_RELEASE is empty. If on the other hand, one is working in a test release, then MU2E_TEST_RELEASE points to the current working area (the test release) while MU2E_BASE_RELEASE points to a complete build of Offline (the base release) against which the test release builds. When compiling code from a test release, the command line must first have the required -I elements from the test release, followed by the -I elements from the base release, followed by the remaining -I elements. If one is compiling the code from a base release, there is, by definition, no test release so the first -I elements must be those of the base release. They gymnastics with cpppath_frag ensures that the -I elements for the test release are present if and only if they are required.
Hopefully the above explanation is enough to understand the LIBPATH variable as well.
One few final note about this section: in python it is perfectly OK to have an apparently superfluous comma at the end of a list; it can be there or not but the usual style is to put it there.
The next main block of SConstruct is:
genreflex_flags = .... lots deleted
aa= ... lots deleted
genreflex = Builder(action=aa)
env.Append(BUILDERS = {'DictionarySource' : genreflex})
These lines tell scons how to generateroot dictionary and map files for the data products;
this operation uses the root genreflex program.
The next few lines are:
SetOption('warn', 'no-fortran-cxx-mix')
env.MergeFlags('-g')
env.MergeFlags('-O2')
env.MergeFlags('-rdynamic')
env.MergeFlags('-Wall')
These tell scons to add various flags to the compile and link commands; scons is smart enough to figure out
which commands are part of the compile command, which are part of the link command and which are part of both.
The next fragment extracts the version of the g++ compiler and adds it to env. And we have already discussed the Export command.
ff = os.popen('g++ --version'); ll = ff.readline(); ff.close()
gcc_version = ll[10:13]
env.gcc_ver=gcc_version.replace('.','')
Export('env')
The last fragment in SConstruct searches the directory heirarchy to find SConscript
files. This is stored in the local variable ss. The line env.SConscript(ss) tells
SCcons to execute each of these SConscript scripts.
ss=[]
for root,dirs,files in os.walk('.'):
for file in files:
if file == 'SConscript': ss.append('%s/%s'%(root[2:],file))
pass
pass
env.SConscript( ss )
At present our evaluation is that scons is more forgiving to less sophisticated users, at a small enough cost to more advanced users. When no new files have been added, both systems do a build with a single command, make or scons. When a new file has been added, however, cmake requires that your first rerun cmake before running make. On the other hand, with scons, you only need to issue the scons command.
One weaknesses of scons is that the scons command must be issued from the top level directory, which requires that scons scan the full work tree to discover than only a subset need be done. With cmake, on the other hand, the cmake and make commands may be issued from any directory and they will rebuild only the subset of the project that decends from that directory. While the choice of scons makes work slower for advanced users, it makes it much, much safer for novice users who might not be aware when it is unsafe to issue the cmake command from a lower level directory.
We believe that we can teach scons to do out-of-source builds but we have not yet tried.
|
|
| Security, Privacy, Legal |
|