To solve this problem, let's first express the function $f(x)$ and then compute its gradient.

Step 1: Understanding the Function

Given: $f(x) = \frac{1}{2} \|Ax - b\|^2 + \|x\|^2$

Here:

• $A \in \mathbb{R}^{3 \times 4}$
• $b \in \mathbb{R}^3$
• $x \in \mathbb{R}^4$

The Euclidean norm squared, $\|v\|^2$, is given by $v^T v$.

So, we can rewrite the function as: $f(x) = \frac{1}{2} (Ax - b)^T (Ax - b) + x^T x$

Step 2: Compute the Gradient of $f(x)$

Let's compute the gradient of $f(x)$ with respect to $x$.

1. First term: $\frac{1}{2} \|Ax - b\|^2$

   $\|Ax - b\|^2 = (Ax - b)^T (Ax - b)$

   To find the gradient of this term with respect to $x$, we use the identity: $\nabla_x \left( \frac{1}{2} (Ax - b)^T (Ax - b) \right) = A^T(Ax - b)$

2. Second term: $\|x\|^2 = x^T x$

   The gradient of $x^T x$ with respect to $x$ is: $\nabla_x (x^T x) = 2x$

Putting these together, the gradient of $f(x)$ is: $\nabla f(x) = A^T (Ax - b) + 2x$

Step 3: Evaluate the Gradient at Any Point $x$

The expression for the gradient $\nabla f(x) = A^T (Ax - b) + 2x$ is valid for any point $x$.

Step 4: Determining the Uniqueness of the Critical Point

The critical points of $f(x)$ are found by solving: $\nabla f(x) = 0$ $A^T(Ax - b) + 2x = 0$

This can be rewritten as: $(A^T A + 2I)x = A^T b$

Here, $(A^T A + 2I)$ is a $4 \times 4$ matrix. To determine if the solution for $x$ is unique, we need to check if this matrix is invertible.

• $A^T A$ is a symmetric $4 \times 4$ matrix.
• Adding $2I$ (where $I$ is the $4 \times 4$ identity matrix) shifts all eigenvalues of $A^T A$ by 2.

Since $A^T A$ is positive semi-definite, all its eigenvalues are non-negative. Adding 2 ensures that all eigenvalues of $(A^T A + 2I)$ are strictly positive, making it positive definite.

A positive definite matrix is always invertible. Hence, the matrix $A^T A + 2I$ is invertible, implying that the equation $(A^T A + 2I)x = A^T b$ has a unique solution for $x$.

Conclusion

• The gradient $\nabla f(x) = A^T(Ax - b) + 2x$.
• The critical point where $\nabla f(x) = 0$ will be unique because $A^T A + 2I$ is positive definite and therefore invertible.

Would you like more details or have any questions?

Related Questions:

1. What are the eigenvalues of the matrix $A^T A + 2I$?
2. How would the solution change if $A$ were a different size matrix?
3. Can you provide a specific example with numerical values to solve for $x$?
4. What does it mean for a matrix to be positive definite?
5. How do we determine if a matrix is invertible in general?

Tip:

Always check if a matrix is positive definite or positive semi-definite when determining the uniqueness of solutions in optimization problems.