Producing New Sage Packages

If you are producing code to add new functionality to Sage, you might consider turning it into a package (an spkg file) instead of a patch file. If your code is very large (for instance) and should be offered as an optional download, a package is the right choice; similarly, if your code depends on some other optional component of Sage, you should produce a package.

If you’re not sure whether to build an spkg file or a patch file, ask for advice on sage-devel.

Creating a New spkg File

Sage packages are distributed as .spkg files, but an .spkg file is just a .tar.bz2 file (or a tar file), but named with .spkg to discourage confusion. In particular, you can type

tar jxvf mypackage-version.spkg

to extract one and see what is inside. If you want to create a new .spkg file, you should start by looking at some old ones. In a source code distribution of Sage, the standard ones can be found in SAGE_ROOT/spkg/standard, or you can download .spkg files from http://sagemath.org/download-packages.html.

Here’s how to make your own .spkg file:

First, make a directory, e.g., mypackage-0.1. The name of the directory should be a lower-case string with no dashes, followed by a dash, followed by a version number.

Directory structure

  • Put your files in that directory. If you are porting another software package, then the directory should contain a subdirectory “src”, containing an unaltered copy of the package. Every file not in “src” should be under version control – that is, checked into an hg repository.

    More precisely, the directory should contain the following:

    • src: this directory contains vanilla upstream code, with a few exceptions: for example, when the spkg shipped with Sage is in effect upstream, and development on that code base is happening in close coordination with Sage. See John Cremona’s eclib spkg, for instance.

    • .hg, .hgignore, and .hgtags: The Sage project uses Mercurial for its revision control system (see Producing Patches with Mercurial). The hidden directory .hg is part of the standard Sage spkg layout. It contains the Mercurial repository for all files not in the src directory.

      The files .hgignore and .hgtags also belong to the Mercurial repository. The file .hgtags is optional, and is frequently omitted. You should make sure that the file .hgignore contains “src”, since we are not tracking its content. Indeed, frequently this file contains only a single line,

      src
      
    • spkg-install: this file contains the install script - see below for more information and a template.

    • SPKG.txt: this file describes the spkg in wiki format. Each new revision needs an updated changelog entry or the spkg will get an automatic “needs work” at review time. See below for a template.

    • spkg-check: this file runs the test suite. This is somewhat optional since not all spkgs have test suites. If possible do create such a script since it helps isolate bugs in upstream packages

    • patches: this directory contains patches against src. Each file requiring changes (e.g., foo.c) needs to have a diff against the original file (e.g,. foo.c.patch) for easy rebases against new upstream. Updated files should be copied into the right place in src at the start of spkg-install. Please document all patches in SPKG.txt, i.e. what they do, if they are platform specific, if they should be pushed upstream, etc.

The file spkg-install

  • The script spkg-install is run during installation of the Sage package. In this script you may make the following assumptions:

    • The PATH has the locations of sage and python (from the Sage installation) at the front. Thus the command

      python setup.py install

      will run the correct version of Python with everything set up correctly. Also, running gap or Singular, for example, will run the correct version.

    • The environment variable SAGE_ROOT points to the root directory of the Sage installation.

    • The environment variable SAGE_LOCAL points to the SAGE_ROOT/local directory of the Sage installation.

    • The environment variables LD_LIBRARY_PATH and DYLD_LIBRARY_PATH both have SAGE_ROOT/local/lib at the front.

    The spkg-install script should copy your files to the appropriate place after doing any build that is necessary. Here is a template:

    #!/usr/bin/env bash
    
    if [ "$SAGE_LOCAL" = "" ]; then
       echo "SAGE_LOCAL undefined ... exiting";
       echo "Maybe run 'sage -sh'?"
       exit 1
    fi
    
    cd src
    
    ./configure --prefix="$SAGE_LOCAL"
    if [ $? -ne 0 ]; then
       echo "Error configuring PACKAGE_NAME."
       exit 1
    fi
    
    make
    if [ $? -ne 0 ]; then
       echo "Error building PACKAGE_NAME."
       exit 1
    fi
    
    make install
    if [ $? -ne 0 ]; then
       echo "Error installing PACKAGE_NAME."
       exit 1
    fi

    Note that the first line is /usr/bin/env bash; this is important for portability. Next, the script checks that SAGE_LOCAL is defined, to make sure that the Sage environment has been set. After this, the script may simply run cd src and then call either python setup.py install or the autotools sequence ./configure && make && make install, or something else along these lines.

    Often, though, it can be more complicated. For example, it is often necessary to apply the patches from the patches directory. Also, you should first build (e.g., with python setup.py build, exiting if there is an error, before installing (e.g., with python setup.py install) – this way, you won’t overwrite a working older version with a non-working newer version of the spkg.

The file SPKG.txt

  • The SPKG.txt file should follow this pattern:

    = name of spkg =
    
    == Description ==
    
    Describe the package here.
    
    == License ==
    
    Describe the package's license here.
    
    == SPKG Maintainers ==
    
    List the maintainers here
    
    == Upstream Contact ==
    
    Provide information for upstream contact.
    
    == Dependencies ==
    
    List the dependencies here
    
    == Special Update/Build Instructions ==
    
    List patches that need to be applied and what they do
    
    == Changelog ==
    
    Provide a changelog of the spkg here.

When the directory (say, mypackage-0.1) is ready, the command

sage -pkg mypackage-0.1

will create the file mypackage-0.1.spkg. As noted above, this creates a compressed tar file; running sage -pkg_nc mypackage-0.1 creates an uncompressed file.

When your spkg is ready, you should post about it on sage-devel; if people there think it’s a good idea, then post a link to the spkg on the Sage trac server (see The Sage Trac Server: Submitting Patches and Packages) so it can be refereed. (Don’t post the spkg itself to the trac server – just provide a link.) If it gets a positive review, it might be included into the core Sage library, or it might become an optional download from the Sage web site, so anybody can automatically install it by typing sage -i mypackage-version.spkg.

Note

There are usually a number of things to do for all spkgs:

  • Make sure that the hg repo contains every file outside the src directory, and that these are all up-to-date and commited into the repo.
  • ensure that make install is non-parallel, i.e. do export MAKE=make
  • Include an spkg-check file if possible (see trac ticket #299).
  • Include md5sums for spkgs (see trac ticket #329).
  • Set LDFLAGS on OSX (see trac ticket #3349).

Note

  • If your package depends on another package, say boehmgc, then you should check that this other package has been installed. Your spkg-install script should check that it exists, with the code like the following:

    BOEHM_GC=`cd $SAGE_ROOT/spkg/standard/; ./newest_version boehm_gc`
    if [ $? -ne 0 ]; then
        echo "Failed to find boehm_gc.  Please install the boehm_gc spkg"
        exit 1
    fi
  • Caveat: Do not just copy to e.g. SAGE_ROOT/local/lib/gap\*/ since that will copy your package to the lib directory of the old version of GAP if GAP is upgraded.

  • External Magma code goes in SAGE_ROOT/data/extcode/magma/user, so if you want to redistribute Magma code with Sage as a package that Magma-enabled users can use, that’s where you’d put it. You’d also want to have relevant Python code to make the Magma code easily usable.