Integrating the BSEC in CircuitPython (A work in progress)

Some pointers on using the USER_C_MODULES build flag

This blog post will touch upon two different topics. On the one hand, I’ll talk about the general approach towards including and testing your own C code in your custom build of CircuitPython . On the other hand, I’ll be talking about specific challenges in using the BME688 Breakout Board, based on the BME688 sensor by Bosch, with the specific breakout board being designed by us, and their BSEC library on the RP2040. This library allows advanced analysis of the sensor data provided by the BME688 and calculations based on that sensor data.

The benefits of wrapping either your own C code or static libraries in CircuitPython are quite obvious. A lot of micro-controller boards, in this case the Raspberry Pi Pico which is based on the RP2040, allow the execution of C code. Writing pure C can be daunting, especially for programmers who haven’t spent a lot of time with that language yet. CircuitPython on the other hand provides easy access via the Python syntax, a lot of well written libraries and a great community to help you out.

Our goal was to test whether it is theoretically possible to wrap the BSEC library. This library is provided for a multitude of platforms and compilers, including the Cortex M0+ and the arm-none-eabi-gcc compiler.

While working around the BSEC might be specific to our case, I’ll provide some more general pointers on how to include a custom written C-Module in CircuitPython to quickly test its functionality, how to link an additional binary library and how to build CircuitPython so both the C-Module and the library are included. There are some minor pitfalls along the way, which I hope this blog post will cover.

If you want to write your own CircuitPython C module, there currently exist two different approaches. You are able to read about the first one in the official guidelines on the Adafruit website which are currently out of date but still contain a lot of useful information. Therefore the recommended approach is to look at the source code of CircuitPython and follow it as an example. Without going into too much detail, this is based around placing your header and source code files into the ‘shared-bindings’ and ‘shared-module’ directories respectively while using the provided wrapper methods.

If your goal is long term usage or contributing to the official release, you should follow those guidelines. If you first want to see if the code runs on a specific platform or do some quick testing, the approach described here is a little easier and less work intensive.

How to use USER_C_MODULES?

First we go through the process of building CircuitPython by using the official instructions which can be found here. After making sure everything works as intended, CircuitPython builds and runs on your target platform, we can start work on including our own code. This is done via the USER_C_MODULES build flag. This functionality is inherited from MicroPython. If you are interested, you can take a look at the official MicroPython documentation which seems to be applicable to CircuitPython as well.

First we create a new folder in the CircuitPython directory and fill it with the necessary files:

circuitpython
├──circuitpython/
|       ├──data/
|       ├──devices/
|       ├──docs/
…       …       
|
└──externalCmodules/
        └──BSEC_Test
                ├── bsec_datatypes.h
                ├── bsec_interface.h
                ├── libalgobsec.a
                ├── micropython.mk
                └── testingBSEC.c

In this case ‘bsec_datatypes.h’, ‘bsec_interface.h’ and ‘libalgobsec.a’ are provided by Bosh. We only need to worry about mircopython.mk and testingBSEC.c.

As it turns out CircuitPython already includes a C (And C++) example and its respective Makefile inside the source code. You can find it at:

circuitpython/examples/usercmodule/cexample/

We can simply copy the code over and edit it for our own use. To do this we include the header files of the library we want to use (In our case #include “bsec_interface.h”) and adapt the code within. The ‘examplemodule.c’ is a well chosen example, since it contains a very basic function which takes integer input and produces integer output. Editing it to our demands was mostly a case of changing the amount of input variables and renaming some functions. If your goal is only to test the viability of a large scale integration this should probably be sufficient. This can be easily achieved by returning error codes and some sample output and checking whether it looks as expected. Otherwise adding new functions or making more fundamental changes to existing ones isn’t very difficult and well described in the MicroPython documentation.

You can already leave it at that. Depending on the complexity of your test case it might be necessary to add further functions, but in our case the main goal was simply to see if the code runs.

Additionally we modify the Makefile – micropython.mk – to look something like this:

BSEC_TEST_DIR := $(USERMOD_DIR)
# Add our C file to SRC_USERMOD.
SRC_USERMOD += $(BSEC_TEST_DIR)/testingBSEC.c
# Adding the pre-compiled library to SRC_USERMOD.
SRC_USERMOD += $(BSEC_TEST_DIR)/libalgobsec.a

We then simply go ahead and build CircuitPython like described in the official documentation:

• 1. Go into the directory of the port you want to build for (In our case circuitpython/ports/raspberrypi)
• 2. make BOARD=raspberry_pi_pico USER_C_MODULES=../../../externalCmodules

The build flag points to the directory we put our code and libraries into.

Problems during the build process

If you followed the instructions pretty closely so far, you will run into the first issue. The build system doesn’t find the library (libalgobsec.a).

It seems the build system looks for it at two different places during the build process. In our case at the following two places:

circuitpython
├──circuitpython/
|       ├──data/
|       ├──devices/
|       ├──docs/
|       └──BSEC_Test
|               └── libalgobsec.a
...
|
└──externalCmodules/
        └──BSEC_Test
                ├── bsec_datatypes.h
                ├── bsec_interface.h
                ├── libalgobsec.a
                ├── micropython.mk
                └── testingBSEC.c

This obstacle apparently has its origin in the way the build system interacts with the USER_C_MODULE buildflag. Even after spending quite some time looking at it, every solution I tried fell short in some way or introduced further complexity. Should someone read this and find a cleaner solution, I would appreciate you sharing it! I’ll edit the text accordingly in that case.

Thankfully the problem can be easily avoided by duplicating the library and placing it at the second location as well. While this obviously isn’t the cleanest solution, there is no size difference in the final build.

If you are trying to get your own code to work, this should be all you need to allow further testing. By circumventing this little problem your own code and libraries should already compile, be linked properly, build and be callable.

But what about our test case, the BSEC?

If you follow this process you’ll quickly be disappointed. You’ll find yourself confronted by error messages about undefined functions:

/home/hanno/System/bin/ARM_GCC/gcc-arm-none-eabi-10-2020-q4-major/bin/../lib/gcc/arm-none-eabi/10.2.1/../../../../arm-none-eabi/bin/ld: IaqEstimator.c:(.text+0xea): undefined reference to `fminf'
/home/hanno/System/bin/ARM_GCC/gcc-arm-none-eabi-10-2020-q4-major/bin/../lib/gcc/arm-none-eabi/10.2.1/../../../../arm-none-eabi/bin/ld: IaqEstimator.c:(.text+0x112): undefined reference to `fmaxf'
/home/hanno/System/bin/ARM_GCC/gcc-arm-none-eabi-10-2020-q4-major/bin/../lib/gcc/arm-none-eabi/10.2.1/../../../../arm-none-eabi/bin/ld: IaqEstimator.c:(.text+0x162): undefined reference to `fmaxf'
/home/hanno/System/bin/ARM_GCC/gcc-arm-none-eabi-10-2020-q4-major/bin/../lib/gcc/arm-none-eabi/10.2.1/../../../../arm-none-eabi/bin/ld: IaqEstimator.c:(.text+0x190): undefined reference to `fmaxf'
/home/hanno/System/bin/ARM_GCC/gcc-arm-none-eabi-10-2020-q4-major/bin/../lib/gcc/arm-none-eabi/10.2.1/../../../../arm-none-eabi/bin/ld: IaqEstimator.c:(.text+0x198): undefined reference to `fminf'
........

Well, as it turns out, there currently exists one major obstacle. Since CircuitPython is meant to run on microcontrollers and memory is often quite limited on those, the developers wrote and use their own math library. While this might be no problem depending on your personal use case, the BSEC uses the functionality provided by the standard C math library. And while there exists a buildflag to build CircuitPython with the standard C math library (circuitpython/ports/raspberrypi/mpconfigport.mk → INTERNAL_LIBM), it currently seems to be defunct. Making it work again would require additional time, a good knowledge of the build system and went a little too far for our small test case.

But there is a (somewhat ‘hacky’) solution to at least get the BSEC to initialize and give out some basic information, proving its usage is probably possible to integrate it in theory. We either simply wrap or reimplement the missing functions in our C code by using the built-in math functions that the internal_libm provides. This can look something like the following:

float fminf ( float x, float y ) {
    if (isnan(x))
		return y;
	if (isnan(y))
		return x;
	// handle signed zeros, see C99 Annex F.9.9.2 
	if (signbit(x) != signbit(y))
		return signbit(x) ? x : y;
	return x < y ? x : y;
}

In most cases the code was taken or slightly adapted from the arm-none-eabi-gcc compiler which is open source and can be looked at without problem.

Now CircuitPython builds, the BSEC is callable and returns the appropriate values. Despite that some problems persist if trying to use certain functionality. It seems likely that these are the result of the CircuitPython build process, because they disappear if the BSEC is used natively with C. To get the BSEC to run in its entirety and to be easily callable from inside CircuitPython additional work is necessary and probably some coordination to get access to a correct version of the BSEC (Which currently is build using the 9-2019-q4-major release of the ARM GNU tool chain, while CircuitPython uses the 10-2020-q4-major release) and to slightly modify and adapt the build process of CircuitPython. Whether this will be done by us in the future has to be seen, but I hope this blog post might provide some direction to developers who are interested in testing their own code by using the USER_C_MODULES buildflag and want to avoid some pitfalls.

Should there be an update to our progress, this page will be updated accordingly.

If you have further questions that I didn’t cover in enough detail (or at all) feel free to either leave a comment or reach out via Email! I’ll be happy to respond.