Week 10: Levenberg Marquardt

Before I get into compute_depthmaps, I looked a little deeper into the math behind bundle adjustment and dug up some concepts that heavily rely on Linear Algebra and Multivariable Calculus. 

Minimization Problem

Last week, I said that bundle adjust uses a linear solver but never specified what the linear problem was. Bundle adjustment is actually a nonlinear least squares problem that solves the optimization problem of minimizing 

where F(x) is vector of m functions in n-dimensional space. 

The solution to the vector x can be approximated by taking step 𝚫x each iteration. In other words, the minimization problem can be written as 

Gradient Descent Method

The gradient descent method takes steps in the direction of steepest descent using step size determined by a positive scalar ⍺. ⍺ multiplied by the Jacobian matrix is the update to F(x). With the proper selection of ⍺, this method always converges, but may take a long time doing so.

Gauss-Newton Method

For moderately-sized problems, the Gauss-Newton method converges much faster than the gradient descent method, but there’s a chance that it diverges.

Using the Taylor series expansion,

If is small, the higher order terms can be ignored, yielding this linear function

with an error on the order of (x – 𝚫x)2.

is the Jacobian matrix J(x). This final expression is called the Jacobian linearization

Solve for 𝚫x for a number of iterations.

Levenberg-Marquardt Method

To make sure F(x + 𝚫x) converges, the Levenberg-Marquardt method introduces a regularization term to dampen the Hessian matrix (matrix of 2nd degree partial derivatives). 

𝜆 is a regularization parameter that is larger at the beginning to make 𝚫x smaller. As the iterations progress and the solution improves, 𝜆 is decreased, and the solution accelerates toward convergence as the Levenberg-Marquardt method becomes more like the Gauss-Newton method.

compute_depthmaps

Now that I’ve finished introducing the minimization problem in bundle adjustment, it’s time to look at the algorithm behind compute_depthmaps. The entire command looks at the data for depth of points in the images and compiles a ply file of the points. To check which image has the most accurate location for each point, PatchMatchUpdatePixel() uses Normalized Cross Correlation (NCC). The equation for NCC is the cosine of the angle between the images. 

For each of the pixels, the function checks the neighboring pixels and 6 random planes for the same pixel and takes the position with the highest cross correlation score. Since the function checks several random planes, the position might be slightly different every time compute_depthmaps is run. 

Source:

https://grail.cs.washington.edu/projects/bal/bal.pdf 

https://people.duke.edu/~hpgavin/ExperimentalSystems/lm.pdf