Week 6: The “Four’s” of Python/C++

Very technical blog today because this week contained a special occasion. On the 4th day of the week, it was the 4th day of the 4th month of 2024. Whatever you were doing at 4:44 is now your lucky activity (or extremely unlucky, possibly death-inducing, if you believe in the unlucky number 4). To celebrate this miraculous string of 4’s, I will explain 4 methods today: 2 ways to use C++ in Python and vice versa and 2 ways to set up a debugger in the Terminal.

Debugging in the Terminal

While VSCode provides extensions to set up debuggers, ultimately all the work happens in the Terminal; VSCode extensions simply provide ease of access. However, in my case, those extensions are giving me issues because of bugs in the version of VSCode I have that hasn’t been resolved on Linux yet, so I’m going to try relying only on the Terminal instead.

Method 1: GDB (GNU Debugger)

GDB is a command-line tool for debugging C, C++, Fortran, etc. To use it, compile a C++ executable by running this command to include debugging information

g++ -g -o filename filename.cpp

Then, activate GDB with the command

gdb filename

And run the program

run

If you want to attach it to a running process, you can also activate GDB with this command instead

gdb -p your_process_id

To use GDB with Python files, you need to set up the python-gdb extension. Then, build your Python file with debug mode by adding the option 

--with-pydebug

After launching GDB, attach it to your running Python process that uses your extension module. The process is explained in full detail here.

Method 2: Valgrind

Valgrind is a toolset for catching memory leaks. You can set it up on your computer, then run tools like Memcheck by using the command

valgrind --leak-check=yes filename

To use it in tandem with GDB, you can get vgdb, an embedded GDB server. Launch Valgrind with the option

--vgdb=yes --vgdb-error=0

This turns on the embedded GDB server and tells Valgrind to wait for 0 errors to occur before waiting for a connection from GDB.

Then, in another window, launch GDB with your program and run

target remote | vgdb

Combining Python and C++

Method 1: Extension modules

The Python/C API is a well-defined process for either extending Python with C modules (extension modules) or using Python as an embedded component in a larger application. To define an extension module, you need to include this at the top of your C file

#define PY_SSIZE_T_CLEAN

#include <Python.h>

Then, you need an init function for Python to recognize your C file as an extension module.

PyMODINIT_FUNC

PyInit_your_module_name (void) {

    // some code

}

In C, Python objects are represented by a single datatype, PyObject. When calling a function from the extension module in Python, the parameters need to be pointers to a PyObject. In order to read the PyObject, there are built-in functions like 

PyObject_Length(PyObject *your_arr_name)

PyList_GetItem(PyObject *your_arr_name)

PyLong_AsLong(PyObject *your_int_or_long_name)

You can read more about the API on the official documentation. NumPy even has its own guide to using extension modules.

Method 2: pybind11

pybind11 is a simplified version of Boost.Python based on C++11 features, a library used to expose C++ code to Python and vice versa. The goal of Boost.Python is to minimize boilerplate (near repetitive) code in extension modules. Since pybind11 is a simplified version of Boost.Python, it’s an even more simplified version of extension modules. To use pybind11, you need to include this header file (you might need more header files based on the complexity of your program). 

#include <pybind11/pybind11.h>

To make things easy, you can shorten the namespace name by writing

namespace py = pybind11;

Like with extension modules, you need an initialization function for Python to recognize your code. 

PYBIND11_MODULE(your_module_name, m) {

    // some code

}

To add a function, write it like you normally would above this function, then inside the PYBIND11_MODULE function, add this line

m.def(“your_function_name_in_python”, &your_function_name_in_c++);

To add a class, write the class, then in PYBIND11_MODULE, add this line

py::class_<your_class_name>(m, “your_class_name_in_python”)

    .def(py::init())

If you want to add functions to the class, just add another .def().

You can read more about pybind11 in its official documentation.

There is also a way to directly cast C++ data into NumPy arrays using a buffer object.

Sources:

https://docs.python.org/3/howto/gdb_helpers.html#gdb 

https://valgrind.org/docs/manual/quick-start.html 

https://docs.python.org/3/c-api/intro.html 

https://numpy.org/doc/stable/user/c-info.how-to-extend.html 

https://pybind11.readthedocs.io/en/stable/basics.html 

https://pybind11.readthedocs.io/en/stable/advanced/pycpp/numpy.html