Modified Newton method | Backtracking Armijo | Theory and Python Code | Optimization Techniques #5

☕️ Buy me a coffee: https://paypal.me/donationlink240 🙏🏻 Support me on Patreon:   / ahmadbazzi   In this one, I will show you what the modified newton algorithm is and how to use it with the backtracking search by Armijo rule. We will approach both methods from intuitive and animated perspectives. The difference between Damped and its modified newton method is that the Hessian may run into singularities at some iterations, and so we apply diagonal loading, or Tikhonov regularization at each iteration. As a reminder, Damped newton, just like newton’s method, makes a local quadratic approximation of the function based on information from the current point, and then jumps to the minimum of that approximation. Just imagine fitting a little quadratic surface in higher dimensions to your surface at the current point, and then going to the minimum of the approximation to find the next point. Finding the direction towards the minimum of the quadratic approximation is what you are doing. As a matter of fact, this animation shows you why in certain cases, Newton's method can converge to a saddle or a maximum. If the eigenvalues of the Hessian are non positive - in those cases the local quadratic approximation is an upside down paraboloid. Next, we talk about the line search we are going to use in this tutorial, which is the Armijo backtracking method. This is achieved by the Armijo condition, which sufficiently decreases our function ! Of course, looking at the Armijo condition equation as is might not reveal any insights, but geometrically looks beautiful, let me show you how. ⏲Outline⏲ 00:00 Introduction 00:57 Modified Newton Method 03:44 Backtracking by Armijo 06:41 Python Implementation 24:41 Animation Module 40:12 Animating Iterations 43:32 Outro 📚Related Courses: 📚 Convex Optimization Extended Course    • Convex Optimization   📚 Python Programming Extended Course    • Python Programming   📚 Convex Optimization Applications Extended Course    • The Transshipment Problem in Decision Maki...   📚 Linear Algebra Extended Course    • Linear Algebra   📚 Python projects course    • Python   🔴 Subscribe for more videos on CUDA programming 👍 Smash that like button, in case you find this tutorial useful. 👁‍🗨 Speak up and comment, I am all ears. 💰 If you are able to, donate to help the channel Patreon -   / ahmadbazzi   This lecture contains many optimization techniques. #python #optimization #algorithm