Motivation

pidoctor is written in Python (as there was no easy way to get Crystal to work on ARMHF / musl).

This means a dependency on Python – which adds overhead.

I suspect that this overhead is the reason that pidoctor will not run on 256 MB Raspberry Pi’s – because the RAM is exhausted by all the packages which are required.

Furthermore, it is desirable to have the distribution be as small a download as possible! This means Python has to be removed from the equation somehow, the package should produce a binary (as would be the way with Crystal).

Luckily, there is nuitka.

And – lucky you – I have worked out some kinks of using nuitka on Alpine Linux, ARMHF (maybe it will work for x86 too).

Nuitka

nuitka is a fantastic piece of software, which will take a Python script and compile it to an executable, thereby speeding it up dramatically.

nuitka does all the heavy lifting, you don’t even need to adjust anything in your source code (methinks). You can run it like this:

python3 -m nuitka –follow-imports –standalone pidoctor.py

Note the –standalone flag, which will compile the necessary shared libraries into your <projectname>.dist folder – thereby the installation will be able to run without any dependency on Python!

Please note: WordPress unfortunately fucks up the double dashes and makes them into single dashes. the –m has a single dash, the –follow-imports and the –standalone have double dashes.

Installation of Nuitka on Alpine Linux (specifically arhmf, might work on other targets)

apk add python3-dev

apk add chrpath

pip3 install –U nuitka

optionally, if you are running in diskless mode (to add nuitka to the files which should be persisted):

lbu add /usr/bin/nuitka3-run

lbu add /usr/bin/nuitka3

lbu add /usr/lib/python3.6/site-packages/nuitka

lbu commit –d

Note: nuitka’s source is in Python.

Check that nuitka runs:

python3 –m nuitka

should execute without showing any syntax errors.

Fixing nuitka to run with Alpine Linux armhf

Important: WordPress unfortunately badly mangles the syntax. Please refer to the GitHUB issue I opened, where I also attached the patch files for a reference.

nuitka calls several support binary utilities on your system, and expects them to return the results in a certain fashion. This is, where nuitka initially stumbles.

First error

pidoctor:/opt/pidoctor# python3 -m nuitka –follow-imports –standalone pidoctor.py

Traceback (most recent call last):

File “/usr/lib/python3.6/site-packages/nuitka/__main__.py”, line 188, in <module>

main()

File “/usr/lib/python3.6/site-packages/nuitka/__main__.py”, line 182, in main

MainControl.main()

File “/usr/lib/python3.6/site-packages/nuitka/MainControl.py”, line 846, in main

Plugins.considerExtraDlls(dist_dir, module)

File “/usr/lib/python3.6/site-packages/nuitka/plugins/Plugins.py”, line 102, in considerExtraDlls

for extra_dll in plugin.considerExtraDlls(dist_dir, module):

File “/usr/lib/python3.6/site-packages/nuitka/plugins/standard/ImplicitImports.py”, line 372, in considerExtraDlls

uuid_dll_path = locateDLL(“uuid”)

File “/usr/lib/python3.6/site-packages/nuitka/utils/SharedLibraries.py”, line 71, in locateDLL

return dll_map[dll_name]

KeyError: ‘libuuid.so.1.3.0’

Reason behind the error: nuitka calls ldconfig –p and expects it to list a list of cached shared libraries, in a format like this:

libICE.so.6 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libICE.so.6

on Alpine Linux, ldconfig –p fails – as there is no such option. ldconfig on Alpine Linux is incapable of producing a list of the form which Nuitka wants.

Fix:

in /usr/lib/python3.6/site-packages/nuitka/utils/SharedLibraries.py

add the following code immediately after the import statements at the top:

def find_alpine(name,paths):
     for path in paths:
         for root, dirs, files in os.walk(path):
             if name in files:
                 return os.path.join(root,name)

and in locateDLL, before import subprocess, add the following code:

if os.path.isfile(‘/etc/alpine-release’):
     return find_alpine(dll_name,[“/lib”,”/usr/lib”,”/usr/local/lib”])
else:
     import subprocess

    (…)

The (…) above indicates that the remainder of the code of locateDLL should be indented under the else.

What this code does: if the file /etc/alpine-release exists – indicating that we are running on Alpine Linux, we search for the shared library in three predefined directories:

• /lib
• /usr/lib
• /usr/local/lib

and return the first matching file. Note, we do not handle the situation in case the file is not found – in this case nothing is returned.

Second error

Traceback (most recent call last):
   File “/usr/lib/python3.6/site-packages/nuitka/__main__.py”, line 188, in <module>
     main()
   File “/usr/lib/python3.6/site-packages/nuitka/__main__.py”, line 182, in main
     MainControl.main()
   File “/usr/lib/python3.6/site-packages/nuitka/MainControl.py”, line 859, in main
     standalone_entry_points = standalone_entry_points
   File “/usr/lib/python3.6/site-packages/nuitka/freezer/Standalone.py”, line 1169, in copyUsedDLLs
     used_dlls = detectUsedDLLs(source_dir, standalone_entry_points)
   File “/usr/lib/python3.6/site-packages/nuitka/freezer/Standalone.py”, line 1062, in detectUsedDLLs
     assert os.path.isabs(dll_filename), dll_filename
AssertionError: ldd

Reason behind the error: nuitka calls ldd with each shared library (“dll_filename”) to be added to your .dist folder. The purpose of this is to find the dependencies of these shared libraries themselves. (A recursive search if you will).

For example:

pidoctor:/opt/pidoctor# ldd /usr/lib/libexpat.so.1
         ldd (0x76f43000)
         libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x76efd000)
         libc.musl-armhf.so.1 => ldd (0x76f43000)

As you see, libexpat.so.1 has two dependencies:

• libgcc_s.so.1 –> to be found at /usr/lib/libgcc_s.so.1
• libc.musl-armhf.so.1 –> to be found at ldd

and this is precisely where nuitka has problems. It wants to copy a file with an absolute path, and is given the name “ldd” instead. Again, something which is specific to Alpine Linux.

Therefore the assertion fails, and nuitka does not continue.

Here’s how to find out which file it actually references:

pidoctor:~# which ldd
/usr/bin/ldd

pidoctor:~# ls -alh /usr/bin/ldd
lrwxrwxrwx    1 root     root          28 Jan  1  1970 /usr/bin/ldd -> ../../lib/ld-musl-armhf.so.1

Note that ../../lib/ld-musl-armhf.so.1 is a relative path, which actually refers to /lib/ld-musl-armhf.so.1 on my system. And on yours, probably, too.

Fix:

search for libutil.so. in the file /usr/lib/python3.6/site-packages/nuitka/freezer/Standalone.py

There is a line with result.add(filename) below it. Just above this line, insert:

if filename == ‘ldd’:
     filename = os.path.join(os.path.dirname(‘/usr/bin/ldd’),os.readlink(‘/usr/bin/ldd’)) #e.g. ‘/lib/ld-musl-armhf.so.1’ fix for Alpine

Third error

Error, needs ‘chrpath’ on your system, due to ‘RPATH’ settings in used shared

Fix:

Did you install chrpath as advised above?

apk add chrpath

That’s it – now nuitka should perform it’s magic without hiccups. I will send the author of nuitka the patches and this blog article, maybe he can include them in nuitka.

Factory version

Update 2.2.2019: The author of Nuitka added my patches (in a slightly modified form). In the current “factory” version, the –standalone compilation works for me. Please refer to this github page for details:

https://github.com/Nuitka/Nuitka/issues/237

http://nuitka.net/doc/factory.html