To diagonalize the given matrix using an orthogonal transformation, we will follow these steps:

Given matrix $A = \begin{pmatrix} 3 & -1 & 1 \\ -1 & 5 & -1 \\ 1 & -1 & 3 \end{pmatrix}$, we need to find an orthogonal matrix $P$ and a diagonal matrix $D$ such that:

$P^T A P = D$

Where $D$ is a diagonal matrix containing the eigenvalues of $A$, and $P$ contains the orthonormal eigenvectors of $A$.

Steps:

1. Find the eigenvalues of the matrix $A$: Solve the characteristic equation:

   $\det(A - \lambda I) = 0$

   Where $\lambda$ represents the eigenvalues, and $I$ is the identity matrix.

2. Find the eigenvectors corresponding to each eigenvalue: For each eigenvalue $\lambda_i$, solve the equation:

   $(A - \lambda_i I) \mathbf{v} = 0$

   to find the corresponding eigenvector $\mathbf{v}_i$.

3. Normalize the eigenvectors: Normalize the eigenvectors to form an orthonormal set. These eigenvectors form the columns of the orthogonal matrix $P$.

4. Construct the diagonal matrix $D$: The diagonal matrix $D$ will have the eigenvalues on its diagonal.

Let me calculate the eigenvalues and eigenvectors for this matrix.The eigenvalues of the matrix $A$ are:

$\lambda_1 = 6, \quad \lambda_2 = 2, \quad \lambda_3 = 3$

The corresponding eigenvectors (normalized) are: