brainsteam.co.uk/brainsteam/content/posts/2017-08-06-.md at c8229cc56febe9854c2bef36ed42a597304cd3a9

4.8 KiB

Raw Blame History

title

author

type

date

draft

url

medium_post

Using Cython modules from Python

Cython compiles into a binary library that can be loaded natively with an import statement. However, getting it compiled is the tricky bit.

When you’re doing quick and dirty dev work and re-running your code to see if it will work every few minutes, I’d recommend making use of the pyximport library that comes with Cython. This module makes importing cython libraries really convenient by wrapping the build process and making the import statement look for and build .pyx files. All you need to do to get it working is run:

import pyximport; pyximport.install()

Then you can literally just import your library. Imagine your Cython file is called test.pyx, you can just do:

import test

and off you go.

If, like me, you’re a big fan of Jupyter notebooks and using importlib reload to bring in new versions of models you’re developing, Cython and pyximport offer a hack that supports this. When you import pyximport, add reload_support=True to the install function call to enable this.

import pyximport; pyximport.install(reload_support=True)

I found this to be very hacky and that reloading often failed with this method unless preceeded by another import statement. Something like this usually works:

from importlib import reload
import test
reload(test)

Optimising and Understanding Cython Code

Remember that Cython code is first “re-written” or “transpiled” to C code and then is compiled to machine readable binary by your system’s C compiler. Well written C is still one of the fastest languages you can write an application in (but also complex and easy to cause a crash from). Since Python is an interpreted language that lives inside a virtual environment, each operation – such as adding together two numbers – actually translates to several C expressions.

Well written Cython code can be compiled down to a small number of instructions but badly optimised Cython will just result in lines and lines of C code. In these cases, the benefit you’re going to be getting from having written the module in Cython is likely to be negligible over standard interpreted Python code.

Cython comes with a handy tool which generates a HTML report showing how well optimised your code is. You can run it on your code by doing

cython -a test.pyx

What you should now have is a test.c file and a test.html file in the directory. IF you open the HTML file in the browser you’ll see your Cython code and yellow highlights. It’s pretty simple: the brighter/more intense the yellow colouring, the more likely it is that your code is interacting with normal Python objects rather than pure C ones and ergo the more likely it is that you can optimise that code and speed things up*.

*Of course this isn’t always the case. In some cases you will want to be interacting with the Python world like in code that passes the output from a highly optimised C function back into the world of the Python interpreter so that it can be used by normal Python code.

If you’re trying to squeeze loads of performance out of Cython, what you should be aiming for is to get to a point where all your variables have a C type (by using cde****f to declare them before you use them) and by only applying C operations and functions wherever possible.

For example the code:

i = 0
while i < 99:
    i += 1

will result in

4.8 KiB Raw Blame History Unescape Escape

Using Cython modules from Python

Optimising and Understanding Cython Code

4.8 KiB

Raw Blame History