To compute the integral of $x^5 e^{-x^2}$, we will use substitution and integration by parts where necessary.

Integral:

$\int x^5 e^{-x^2} \, dx$

Step-by-step solution:

1. Substitute: Let $u = x^2$, so that $du = 2x \, dx$, or $\frac{du}{2} = x \, dx$.

   Now rewrite $x^5$ as $x^4 \cdot x$, so we can substitute $x \, dx$ from the expression: $\int x^5 e^{-x^2} \, dx = \int x^4 \cdot x e^{-x^2} \, dx = \int x^4 e^{-x^2} \cdot \frac{du}{2}$

2. Separate powers of $x$: Since $u = x^2$, we have $x^4 = u^2$. This converts the integral to: $\int u^2 e^{-u} \cdot \frac{du}{2}$ Simplifying: $\frac{1}{2} \int u^2 e^{-u} \, du$

3. Integration by parts: Now we apply integration by parts to solve $\int u^2 e^{-u} \, du$. Let: $v = u^2 \quad \text{and} \quad dw = e^{-u} \, du$ Then $dv = 2u \, du$ and $w = -e^{-u}$.

   Using the formula for integration by parts, $\int v \, dw = vw - \int w \, dv$, we get: $\int u^2 e^{-u} \, du = -u^2 e^{-u} + \int 2u e^{-u} \, du$

4. Repeat integration by parts: Now solve $\int 2u e^{-u} \, du$ using parts again: Let $v = 2u$ and $dw = e^{-u} \, du$, so $dv = 2 \, du$ and $w = -e^{-u}$. Applying the integration by parts formula again: $\int 2u e^{-u} \, du = -2u e^{-u} + \int 2e^{-u} \, du = -2u e^{-u} - 2e^{-u}$

5. Combine results: Now substitute this back into the original integral: $\int u^2 e^{-u} \, du = -u^2 e^{-u} + (-2u e^{-u} - 2e^{-u})$ Simplifying: $\int u^2 e^{-u} \, du = -(u^2 + 2u + 2) e^{-u}$

6. Final answer: Multiply the result by $\frac{1}{2}$, and substitute back $u = x^2$: [ \int x^5 e^{-x^2}