Stanford EE364A Convex Optimization I Stephen Boyd I 2023 I Lecture 13

Linear Algebra

• Linear algebra is the foundation of the course, and everyone should be familiar with its concepts.
• Direct methods for solving sets of equations, such as Gaussian elimination, factor the coefficient matrix into easily invertible matrices.
• Sparse matrix factorization deals with efficient methods for factorizing and solving sparse matrices.
• The Cholesky factorization is used to solve positive definite systems of equations efficiently.
• Sparse Cholesky factorization is used to solve positive definite systems of equations efficiently.
• Permuting the matrix before factorization can significantly affect the sparsity of the resulting L factor.
• Finding a good permutation is crucial for achieving efficient factorization and solving times.
• Cholesky factorization fails if the matrix is not positive definite.
• LDL transpose factorization is a method for solving non-singular symmetric matrices.
• The sure complement method is used to solve matrix equations efficiently in certain cases.
• Sparse matrices with a banded structure and dense rows and columns can be solved efficiently using block elimination.
• The sparsity pattern of a matrix can indicate the complexity of solving the associated system of equations.
• The Matrix Inversion Lemma provides a way to efficiently invert a matrix that is a perturbation of a diagonal matrix.
• Un-elimination can be an effective technique for solving certain types of linear systems.
• Sparse solvers can efficiently solve large systems of linear equations, making many traditional methods obsolete.

Unconstrained Minimization

• Unconstrained minimization involves finding the minimum value of a smooth, twice continuously differentiable function without any constraints.
• Iterative methods are used to solve unconstrained minimization problems since analytical solutions are generally not available.
• Stopping criteria are used to determine when the iterative process should stop.
• Descent methods are widely used for convex optimization and involve finding a descent direction and choosing a step length.
• Gradient descent is the most intuitive iterative method for unconstrained minimization.
• Gradient descent exhibits linear convergence, meaning that each iteration reduces the error by a constant factor.