External dependencies and SDF.extern

The SDF format is intended for long-term use and low-maintenance. Thus, relying on third-party libraries is a problem. So think twice if you really need this fancy new library, since it may be dead in a year and cause problems to SDF in two years.

Libraries with a large community and relatively secure long-term stability like Numpy are probably fine, so keeping them as external dependencies in setup.py is a valid approach here. But relying on niche libraries can be a problem, so we keep copies of such libraries in SDF.extern and thus ensure that SDF installations don’t break with updates to these libraries.

How to add a package mymodule to SDF.extern

  1. Check if you can handle it

    • If mymodule has many dependencies that also need to be added, think twice before starting to add them all

    • If mymodule or one of its dependencies contains a C-extension or other non-Python parts, adding it would

      probably complicate the SDF installation process. Think again before you continue.

  2. Copy files to SDF/extern

    • Go to PyPI page of the package mymodule

    • Download as archive (e.g. tar.gz) and extract files

    • Usually the base directory contains (among others) a file called setup.py and a directory with the package name. That directory contains the source code and has to be copied to SDF/extern

  3. Fix imports

    • Often, packages consist of multiple files and import themselves. In this case, change all imports to import from SDF.extern.mymodule, or to relative imports

  4. Add dependencies of mymodule to SDF.extern (same procedure)

How to import modules from SDF.extern

  • Directly needed module

    • For such modules it may be useful for developers to be able to quickly switch to locally installed versions

    • In this case, add a switch like use_system_mymodule to SDF.CONFIG

    • In code outside of SDF.extern use checks like if CONFIG.use_system_mymodule to switch between from mymodule import ... and from SDF.extern.mymodule import ...

    • In SDF.extern, such checks are not necessary, as this code will only be executed from SDF.extern anyway

  • Dependencies of modules in SDF.extern

    • Such packages do usually not need a switch to local installations, so from SDF.extern.mymodule import ... will suffice