Separate builds for various combinations of architecture and branch are supported. All data private to a build (ports tree, src tree, packages, distfiles, log files, bindist, Makefile, etc) are located under the ${arch}/${branch}/builds/${buildid}/ directory. The most recently created build can be alternatively referenced using buildid latest, and the one before using previous.
New builds are cloned from the latest, which is fast since it uses ZFS.
The scripts/dopackages.wrapper script is used to perform the builds.
# dopackages.wrapper ${arch} ${branch} ${buildid} [-options]
Most often, you will be using latest for the value of buildid.
[-options] may be zero or more of the following:
-keep - Do not delete this build in the future, when
it would be normally deleted as part of the latest - previous cycle. Do not forget to clean it up manually when you
no longer need it.
-nofinish - Do not perform post-processing once the
build is complete. Useful if you expect that the build will need to be restarted
once it finishes. If you use this option, do not forget to cleanup the
clients when you do not need the build any more.
-finish - Perform post-processing only.
-nocleanup - By default, when the -finish stage of the build is complete, the build data will
be deleted from the clients. This option will prevent that.
-restart - Restart an interrupted (or non-finished) build from the beginning. Ports that failed on the
previous build will be rebuilt.
-continue - Restart an interrupted (or non-finished) build. Will not rebuild ports that failed on the
previous build.
-incremental - Compare the interesting fields of the
new INDEX with the previous one, remove packages
and log files for the old ports that have changed, and rebuild the rest. This
cuts down on build times substantially since unchanged ports do not get rebuilt
every time.
-cdrom - This package build is intended to end up on
a CD-ROM, so NO_CDROM packages and distfiles should
be deleted in post-processing.
-nobuild - Perform all the preprocessing steps, but
do not actually do the package build.
-noindex - Do not rebuild INDEX during preprocessing.
-noduds - Do not rebuild the duds file (ports that are never built, e.g., those marked IGNORE, NO_PACKAGE, etc.) during
preprocessing.
-nochecksubdirs - Do not check the SUBDIRS for ports that are not connected to the build.
-trybroken - Try to build BROKEN ports (off by default because the amd64/i386™ clusters are fast enough now that when doing
incremental builds, more time was spent rebuilding things that were going to
fail anyway. Conversely, the other clusters are slow enough that it would be a
waste of time to try and build BROKEN ports).
Note: With
-trybroken, you probably also want to use-fetch-originaland-unlimited-errors.
-nosrc - Do not update the src tree from the ZFS snapshot, keep the tree from previous
build instead.
-srcvcs - Do not update the src tree from the ZFS snapshot, update it with a fresh
checkout instead.
-noports - Do not update the ports tree from the ZFS snapshot, keep the tree from previous
build instead.
-portsvcs - Do not update the ports tree from the ZFS snapshot, update it with a fresh
checkout instead.
-norestr - Do not attempt to build RESTRICTED ports.
-noplistcheck - Do not make it fatal for ports to
leave behind files after deinstallation.
-nodistfiles - Do not collect distfiles that pass
make checksum for later uploading to ftp-master.
-fetch-original - Fetch the distfile from the
original MASTER_SITES rather than any cache such as on ftp-master.
-unlimited-errors - defeat the "qmanager threshhold"
check for runaway builds. You want this primarily when doing a -restart of a build that you expect to mostly fail, or
perhaps a -trybroken run. By default, the threshhold
check is done.
Unless you specify -restart, -continue, or -finish, the
symlinks for the existing builds will be rotated. i.e, the existing symlink for previous will be deleted; the most recent build will have its
symlink changed to previous/; and a new build will be
created and symlinked into latest/.
If the last build finished cleanly you do not need to delete anything. If it was
interrupted, or you selected -nocleanup, you need to
clean up clients by running
% build cleanup ${arch} ${branch} ${buildid} -full
When a new build is created, the directories errors/, logs/, packages/, and so forth, are cleaned by the scripts. If you are short of space, you can also clean out ports/distfiles/. Leave the latest/ directory alone; it is a symlink for the webserver.
Note: dosetupnodes is supposed to be run from the dopackages script in the
-restartcase, but it can be a good idea to run it by hand and then verify that the clients all have the expected job load. Sometimes, dosetupnode cannot clean up a build and you need to do it by hand. (This is a bug.)
Make sure the ${arch} build is run as the ports-${arch} user or it will complain loudly.
Note: The actual package build itself occurs in two identical phases. The reason for this is that sometimes transient problems (e.g., NFS failures, FTP sites being unreachable, etc.) may halt a build. Doing things in two phases is a workaround for these types of problems.
Be careful that ports/Makefile does not specify any empty subdirectories. This is especially important if you are doing an -exp build. If the build process encounters an empty subdirectory, both package build phases will stop short, and an error similar to the following will be written to ${arch}/${branch}/journal:
don't know how to make dns-all(continuing)
To correct this problem, simply comment out or remove the SUBDIR entries that point to empty subdirectories. After doing
this, you can restart the build by running the proper dopackages command with the -restart option.
Note: This problem also appears if you create a new category Makefile with no SUBDIRs in it. This is probably a bug.
Example 6. Update the i386-7 tree and do a complete build
% dopackages.wrapper i386 7 -nosrc -norestr -nofinish
Example 7. Restart an interrupted amd64-8 build without updating
% dopackages.wrapper amd64 8 -nosrc -noports -norestr -continue -noindex -noduds -nofinish
Hint: it is usually best to run the dopackages command inside of screen(1).
You may need to manipulate the build data before starting it, especially for experimental builds. This is done with the build command. Here are the useful options for creation:
build create arch branch [newid] - Creates newid (or a datestamp if not specified). Only needed when bringing up a new branch or a new architecture.
build clone arch branch oldid [newid] - Clones oldid to newid (or a datestamp if not specified).
build srcupdate arch branch buildid - Replaces the src tree with a new ZFS snapshot. Do not forget to use -nosrc flag to dopackages later!
build portsupdate arch branch buildid - Replaces the ports tree with a new ZFS snapshot. Do not forget to use -noports flag to dopackages later!
Sometimes there is a need to rebuild a single package from the package set. This can be accomplished with the following invocation:
# path/qmanager/packagebuild amd64 7-exp 20080904212103 aclock-0.2.3_2.tbz