Exporting from Go to other environments like Python’s FFI.

We wanted to take a string masking/unmasking package that we’d written in Go for our backend systems, and use it on our data science team. To accomplish this, we needed to accomplish a couple of goals: 1) ensure correctness, such that a data science app could always decode a given string created in the Go back-end if they had the correct decryption key, and 2) that we would get good performance in this operation, as it would potentially be called on millions of strings for large data sets.

This was our first attempt at using Cgo, and also our first attempt at importing a C extension library in Python. Both were interesting learning experiences! We initially were going to rewrite the same string encrypt/decrypt key scheme in pure Python, but reusing our Go package and just writing binding code proved to be easier, and require less testing and code duplication.

We used Cgo to export our Go library and its public API to an auto-generated C shared object. We then used Python’s C FFI utilities to map this into a Python package. With not much extra effort, it gave us a good standard of performance (since the Go library was exported as a C shared library from the Python POV, so performance is on part with a C extension) and known correctness, both of which were important requirements for our use case.

I’ll be producing an example repo that shows something very similar: A Go package, a generated C shared object, a Python wrapper around the C object using FFI, and the final Python package for the user. I can be reached on GitHub and Twitter at @damienstanton.

Tech Stack

  • Python

About the Engineer

Damien is a full-stack developer, often writing in Go, Rust, Java or other systems languages. Most of his work is in distributed or embedded machine learning systems.

company logo
Damien Stanton
Full-stack ML/AI & Android engineer, systems programmer, deep learning specialist.

Comment test