Packaging Python projects for Debian / Raspbian with dh-virtualenv

This article aims to explain some things to developers which don’t use Python a lot, and might struggle with some of the concepts otherwise.

I highly recommend the lecture of the following article as an introduction to the concepts discussed here:

• https://www.dabapps.com/blog/introduction-to-pip-and-virtualenv-python/

pypi.org

pypi.org is an official repository of software for the Python programming language. It includes libraries we want to use in our projects, which are not shipped by default with Python.

The problem

Our goal is to package an application written in Python (targetting Python3) as a .deb package for Raspbian (which is, all things considered, really equivalent to Debian).

Our application has dependencies on several Python libraries.

For instance, we would like to use the requests library. The requests library itself depends on other libraries, e.g. chardet, urllib3, etc.

Manually managing these dependencies and manually shipping the required library code is inherently impractical.

pip

Python has it’s own package / dependency management system, called pip

pip allows you to install python libraries and applications, so that your own code can use these libraries.

Here there are two problems:

• we want to ship the package as a Debian package, for end users to install easily. We can safely assume that on a significant number of target systems the appropriate python libraries will not have been installed by the user
• if we try to install these libraries system-wide on behalf of the user, we might break compatibility with some other packages (e.g. because of API changes in between versions of the libraries)

Therefore we need a solution.

virtualenv

virtualenv will allow us to create virtual environments for Python.

In these virtual environments we have:

• our own Python interpreter
• copies of the required standard libraries
• our own pip
• our own “system-wide” library

Practical part: Trying out virtualenv

Install virtualenv (as part of dh-virtualenv, which we will get to in a little while) by doing:

sudo apt install dh-virtualenv

As requests is already installed system-wide on my test box, we will use a different package as an example. randstr will allow you to generate a random string.

Check if randstr is installed system-wide:

pip3 show randstr

This should return nothing. Indicating that this package is not installed systemwide:

Now we will create a virtual environment. The virtual environment folder does NOT have to live inside your code folder, but it can. You should ignore it in GIT.

let’s say we have a directory /home/pi/tutorial

Change into this directory, and create a new environment folder env:

cd /home/pi/tutorial

which python3

virtualenv -p /usr/bin/python3 env

Note: you should specify the Python interpreter if you want Python 3, as otherwise Python 2 will be used. I am using Python 3. (Running virtualenv with interpreter /usr/bin/python2)

which python3 will tell you where the python3 binary resides, adjust the path for virtualenv accordingly.

activate the environment:

source env/bin/activate

(This command path assumes that you stayed in the tutorial directory). Please also note, that this is not an executable script – you have to use source to use it with your bash terminal.

Notice, how (env) is prepended before your command prompt:

This means that the environment is active. Now you can run pip3 again, but this time pip3 will perform it’s work on the virtual environment:

pip3 show randstr

pip3 show requests

should still yield nothing. The second line will also show nothing – showing that (if requests is installed system-wide) we are indeed in a virtual environment.

Now install randstr:

pip3 install randstr

and check again, if it is installed:

pip3 show randstr

This time it shows that randstr is installed. It has been installed into env/lib/python3.5/site-packages:

While the environment ist still activated, let’s write a small Python sample application:

nano sample.py

paste the following code, generating a random string using the new library we just installed:

from randstr import randstr
print(“hello world”)
print(randstr())

and save.

Run the code using python3:

python3 sample.py

while the environment is still active, this will work:

Now it’s time to exit the environment, to see whether the code will still run:

deactivate

Your prompt will return back to normal. And

python3 sample.py

will throw an error message, about no module ‘randstr’ existing on your system. This is exactly the behaviour we want!

You can have a look at the env folder, seeing how it ships all the necessary things to run your Python code, including libraries, the python executable, pip, and more.

Debian virtualenv packaging with dh-virtualenv

OK, now with the basics out of the way, what we want is to package a virtualenv alongside with our code, so that our application will run reliably and without disturbing other applications on our user’s computers.

This is where dh-virtualenv enters the picture. dh-virtualenv is an addition to the Debian building scripts, which allows you to package virtual environments.

We need to create a basic package structure for our new package. This is where cookiecutter will help.

cookiecutter

cookiecutter creates new projects for you from templates, thus reducing the overhead and time you spend on developing and getting your head around the many files which are required.

https://cookiecutter.readthedocs.io/en/latest/

sudo apt install cookiecutter

Note: the package python3-cookiecutter will provide a Python module for Python 3. What we want is the application itself, which is installed with the cookiecutter package as seen above. This might pull in Python 2, but – oh well.

cookiecutter is not necessary for your end users, it is only used to create several files for your package.

create an example project directory, into which we will apply the template (I am still in the tutorial directory):

mkdir sampleproject

cd sampleproject

Using a special mold for dh-virtualenv we can set up many of the necessary files:

https://github.com/Springerle/dh-virtualenv-mold

cookiecutter https://github.com/Springerle/dh-virtualenv-mold.git

This will ask you several questions.

Note, that the folder should be named debian, even when packaging for Raspbian – therefore keep the name as debian.

This will install the following files:

Now you need to run the following commands, as per the instructions under the dh-virtualenv-mold:

sudo apt-get install build-essential debhelper devscripts equivs

sudo mk-build-deps --install debian/control

To build the package later on, you will run the follwoing command from your project’s top level directory:

dpkg-buildpackage -uc -us –b

Right now, though, the command will fail because setup.py is missing. We will get to setup.py and requirements.txt in a brief while:

/usr/bin/python3: can't open file 'setup.py': [Errno 2] No such file or directory

Information about the individual files

changelog

This file will contain your release and version information for the package. You can update it using a special tool, dch.

This file is not a changelog for your application, but just a changelog for the packing of your application.

compat

This file includes a special magic number “9”. (for compatibility issues, it’s fine to leave it as it is)

control

This is the main file to set package settings, and dependencies on a Debian level. If your application depends, for example, on the omxplayer being installed, it will need to go in here as a dependency:

Source: sampleproject
Section: contrib/python
Priority: extra
Maintainer: Maximilian Batz <???>
Build-Depends: debhelper (>= 9), python, python-dev, dh-virtualenv (>= 0.10), tar
Standards-Version: 3.9.5
Homepage: https://picockpit.com

Package: sampleproject
Architecture: any
Pre-Depends: dpkg (>= 1.16.1), python2.7 | python3, ${misc:Pre-Depends}
Depends: ${python:Depends}, ${misc:Depends}
Description: An example package to demonstrate virtualenv Debian packaging with dh-virtualenv
     .
     This is a distribution of “sampleproject” as a self-contained
     Python virtualenv wrapped into a Debian package (“omnibus” package,
     all passengers on board). The packaged virtualenv is kept in sync with
     the host’s interpreter automatically.
     .
     See https://github.com/spotify/dh-virtualenv for more details.

You will also want to edit the long Description.

cookiecutter.json

Includes your initial settings. Not required for packaging.

copyright

Your copyright file, with your license (e.g. MIT). It’s pre-filled with your name, year, e-Mail and “some rights reserved”.

rules

This is a Makefile, to actually build your package. It is prefilled by the cookiecutter template.

sampleproject.links

This file allows you to create links during installation . The filename will include your project name instead of sampleproject.

Ref: https://stackoverflow.com/questions/9965717/debian-rules-file-make-a-symlink

sampleproject.postinst

This file allows you to run additional setup steps after installation (e.g. activating your python script as a service). The filename will include your project name instead of sampleproject.

sampleproject.triggers

As far as I understand it this is for dh-virtualenv to install a newer python interpreter into the virtual environment to ship, if the system’s (your development box) one is updated. The filename will include your project name instead of sampleproject.

This might or might not be the case, I have not tested it.

setup.py

As mentioned above, we need a setup.py file. This is the standard way applications are packaged under Python (they ship with a setup.py, which is executed for various tasks related to packaging).

A simple setup.py can be seen on this page:

https://github.com/benjaminirving/python-debian-packaging-example/blob/master/setup.py

Not all entries in this files are necessary. For instance, you can leave out the classifiers. I have the following setup.py:

Note: the version number and other information inside here are for your Python package (which will be created in the course of building a Debian package with your virtualenv).

Structure for your code

Your code will now live in a subdirectory of the main sampleproject directory:

The subdirectory has the same name as the main directory in this case.

Note the __init__.py

The sample.py is the one we used above, but a bit reworked to look like this:

Additionally, I set up a virtual environment for testing while developing inside the main (top level) sampleproject directory:

virtualenv -p /usr/bin/python3 sp_env

called sp_env for sample project environment

Entering the environment:

source sp_env/bin/activate

Install the randstr library into this environment:

pip3 install randstr

and now we can run the code:

python3 sampleproject/sample.py

OK. Exit the environment (! important !), and try to build the package again:

deactivate

dpkg-buildpackage -us –uc

By default (because of the cookie cutter), your package will be created for installation in /opt/venvs/:

New python executable in /home/pi/tutorial/sampleproject/debian/sampleproject/opt/venvs/sampleproject/bin/python3

This can be changed in the debian/rules file.

Now the package has been built, and deposited outside of the sampleproject top directory (inside the tutorial directory):

You can install the .deb file using dpkg:

sudo dpkg -i sampleproject_0.2.1+nmu1_armhf.deb

See the installation results with

tree /opt/venvs

127 directories and 945 files were installed.

Now you can try to execute the sampleproject using

/opt/venvs/sampleproject/bin/sampleproject

requirements / dependencies

The application will not run as (maybe) expected, but throw an error about randstr not being present. What gives?

The reason is that dh-virtualenv does not know about the requirement we have, to bundle the package randstr during the build process. This is the final piece of the puzzle.

The requirements need to be added to the setup.py, as a way to let pip know which additional packages need to be installed in the virtual environment which is being created.

add the following to setup.py:

install_requires=[

        ‘randstr’

]

The whole file then will look like this:

In my tests the requirements.txt placed at top-level did not influence the behaviour of dh-virtualenv and the Python packages actually included in the Debian package. If you still want it, here’s how:

pip offers you a way to put out the exact dependencies you used for developing in the virtual environment. Enter the virtual environment:

source sp_env/bin/activate

And create the requirements.txt file:

pip3 freeze > requirements.txt

Now you can have a look at that file:

cat requirements.txt

Even though this file seems not to be used by dh-virtualenv (maybe it needs to go in a different subdirectory), it is a good reference for yourself, to see which packages you should put as dependencies into the install_requires part of setup.py.

Note that the file sampleproject.links offered us a convenient way to link this into the /usr/bin folder, I will therefore remove the comment sign for the next build:

With that out of the way, it’s time to purge the sampleproject, and build it again, then install it. Don’t forget to deactivate first!

deactivate

sudo apt purge sampleproject

dpkg-buildpackage –us -uc

cd ..

sudo dpkg –i sampleproject_0.2.1+nmu1_armhf.deb

This time, the package should be accessible using sampleproject:

which sampleproject

sampleproject

Ref: https://packaging.python.org/discussions/install-requires-vs-requirements/

Bonus tip: changing the installation location

/opt/venvs is not a very Debian “place” to put a package.

The specifications are here:

http://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html

https://unix.stackexchange.com/questions/10127/why-did-my-package-get-installed-to-opt

• /usr/local is not for packages
• /opt is not for Debian packages proper (it’s for user-installed third-party software)

http://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s06.html

Applications may use a single subdirectory under /usr/lib. If an application uses a subdirectory, all architecture-dependent data exclusively used by the application must be placed within that subdirectory.

https://wiki.debian.org/HowToPackageForDebian

https://www.debian.org/doc/debian-policy/ch-opersys.html#file-system-hierarchy

“The FHS requirement that architecture-independent application-specific static files be located in /usr/share is relaxed to a suggestion. In particular, a subdirectory of /usr/lib may be used by a package (or a collection of packages) to hold a mixture of architecture-independent and architecture-dependent files. However, when a directory is entirely composed of architecture-independent files, it should be located in /usr/share.”

The best location to me seems to be /usr/share

We need to edit two files:

in debian/rules DH_VIRTUALENV_INSTALL_ROOT must be changed to /usr/share, like this:

and debian/sampleproject.links must be changed:

Also, we will make a new entry into the changelog:

dch –i

simply edit the text, to reflect a new version number, and an entry for the changelog.

Then the project can be build again:

dpkg-buildpackage -us –uc

After installation, you can verify that indeed the new location is used:

The package will take up about 16 MB on your target system, thanks to the virtual environment and everything which is shipped with it. This probably could be reduced by some parameters, but that exercise is left to the reader.

Have fun packaging!