Why are most Lagrange multipliers zero in the SVM solution?
When solving your SVM problem, you'll be optimizing a Lagrangian subject to KKT conditions. Specifically, something like:
$$L(x)=f(x)\sum_k \lambda_k c_k(x),$$
where your constraint satisfies $c_k(x)\geq 0$ and $\lambda_k\geq 0$. The optimum is achieved when the gradient of the above lagrangian is equal to 0 and $\lambda_i\geq 0$ and $\lambda_i c_i(x)=0$ for all $i$. Specifically, when $\lambda_i\neq 0$, the constraint is said to active, whereas if $\lambda_i=0$, then you can freely move out of the constraint region while preserving the optimum. This is why we demand $\lambda_i>0$.
Related videos on Youtube
Arnomoonens
Updated on August 01, 2022Comments

Arnomoonens over 1 year
I read everywhere that a nonzero Lagrange multiplier $\lambda_i$ signifies that the corresponding point $x_i$ is a support vector, but I can't see how a support vector and a nonsupport vector have a different value for the Lagrange multiplier.
Can you please explain how the process of optimizing the Lagrangian leads to some Lagrange multipliers being zero and some nonzero?

Alex R. over 7 yearsI think you'll find this helpful: engr.mun.ca/~baxter/Publications/LagrangeForSVMs.pdf

Arnomoonens over 7 years@AlexR. In 4.1 (Example 4): First it is stated that $x^21 \geq 0$ but later on I see that, after deriving w.r.t. $\lambda$, we get $x^21=0$. How is this possible? Can't this value be greater than zero?

iRestMyCaseYourHonor over 3 yearsI think you can find this answer helpful. stats.stackexchange.com/questions/54976/…


Arnomoonens over 7 yearsI see that it can move freely around when $c_i(x)=0$, because then the constraint $\lambda_i c_i(x)=0$ is already satisfied. But why would it become greater than zero? Is it because then the Lagrangian is minimized?