Week 7: Mesh of Languages

Last week, I introduced several C++ debuggers and extension module techniques. This week, I didn’t end up using the C++ debuggers because while I was hoping to trace the calls to the C++ code from Python, my gdb had problems connecting to the process, so I traced the Python code using the Python debugger (pdb) instead.

 

PDB

To use PDB, you can either import pdb at the beginning of the file and put pdb.set_trace() wherever you want a break in your code, or use breakpoint() to skip the import statement. When you run your Python program, it will automatically open the PDB command line.

Some commonly used commands are:
c to continue to next breakpoint
s to step into the current line
whatis some_var to get the data type of some_var
p some_var to print the contents of some_var
until linenumber to continue until that line number; continues to end of current function if the line number is greater than the last line number of the function

To get the process ID of your running Python script, use the os.getpid() function:

Connect GDB to the running process using gdb -p your_pid:

I had a problem with how something was linked as you can see from the colorful message on the bottom, so this didn’t connect GDB, but normally that process would be correct.

The other commands in opensfm_run_all

In the blogs of previous weeks, I’ve only been talking about the reconstruction part of the process, but there’s in fact code to process before and after as well. When you run opensfm_run_all, it executes a bash file with these commands

While reconstruct is one of the lengthiest processes, I needed to pick a command with less action so that I could start out with a test of a complete process with few changes. All of the commands invoke a call to opensfm_main.py, which initializes a DataSet to store the input images, then calls the respective command with the DataSet object. Since the DataSet object stores its data as matrices defined by the C++ Eigen class, which has built-in implementation for conversion to NumPy, I may have to create a list of types in a header file to direct the initialization of these matrices to vectors of BsiAttribute’s.

The current types.h file:

extract_metadata formats all the metadata in each image into attributes of the DataSet object.
detect_features detects features in images using OpenCV algorithms like SIFT (Scale-Invariant Feature Transform), SURF (Sped Up Robust Features), AKAZE (Accelerated KAZE), ORB (Oriented FAST and Rotated BRIEF), and HAHOG (Histogram of Oriented Gradients).
match_features generates similar pairs for matching using strategies based on GPS distance, time, triangulation, Bag of Words, and VLAD.
create_tracks creates a track manager that links matches into tracks using union find.
reconstruct calls the incremental reconstruction process.
mesh adds the Delaunay meshes to the reconstruction using the corresponding perspective transformations.
undistort exports the reconstruction to NVM_V3 format from VisualSfM.
compute_depthmaps finds the angle between neighboring images to merge depthmaps.

Out of these, I decided to test mesh because it has one of the higher level linear algebra operations, inverse, and a lower level dot product, ignoring the operations in the cameras that aren’t perspective, which is what my phone takes pictures in. Since dot product was already implemented in BsiAttribute, I started out with a small testcase using pybind11 to make sure I knew how to use it.

pybind11 example

After writing the function and adding it to pybind11 in your C++ code,

You need to add these lines to created a shared library using your pybind11 module

Here, you can see that pybsi was generated along with the other .so files when running setup.py.

Now you can call it from your Python code.

Now that I know this pybind11 structure works, I need to implement inverse with BsiAttribute. To avoid having to run setup.py every time I want to test whether it works in Python, I’ll start off by testing it separately in C++. I turned to using CLion because I was already familiar with the debugger.

Inverse

NumPy uses the sgesv LAPACK routine for inverse. LAPACK (Linear Algebra Package) is a collection of high level linear algebra operations written in Fortran. It looks a lot like pseudo code because it’s mostly used for mathematical computation.
sgesv is a routine that solves for X, the solution to A * X = B, where A is a n-by-n matrix and X and B are both n-by-m matrices, using LU decomposition with partial pivoting (only row interchanges). This is the equivalent of using Gaussian elimination to find the inverse.
LU decomposition breaks down a matrix A into

A = P * L * U.

In the case of inverse,
P is the permutation matrix that stores the row interchanges,
L is the lower (all 0’s above the diagonal) unit (has 1’s along the diagonal) triangular matrix that stores the elementary elimination to get to row-echelon form, and
U is the upper triangular matrix that stores the elementary elimination to get to reduced row-echelon form.

By solving for the right hand side using an identity matrix, you get the inverse.

So far, I’ve set up a skeleton for the main functions and written and tested a compare function that compares the elements at the same index of a BsiAttribute.

The testing code:

Next week, I’ll probably be continuing to write the inverse function since there are several small functions in between that haven’t been implemented yet, and I need to run testcases on all of them to make sure they’re accurate for development.

Sources:
https://docs.python.org/3/library/pdb.html
https://www.netlib.org/lapack/