3

(a) For a critical point (equilibrium point) we need $x = y$ and $x^2 + y^2 = 2 x^2 = 2$. Therefore $x=y=1$ or $x=y=-1$. The Jacobian matrix is

$$J = \pmatrix{2x & 2y \cr 1 & -1\cr}$$

At the equilibrium point $(-1,-1)$ the Jacobian matrix $$\pmatrix{-2 & -2 \cr 1 & -1\cr}$$ has characteristic polynomial $P(r) = (-2-r)(-1-r) +2 = r^2 + 3 r + 4$. The eigenvalues are the roots of this, namely $(-3 \pm \sqrt{7} i)/2$. They are complex with negative real part, so this is a spiral attractor (asymptotically stable spiral).

At the equilibrium point $(1,1)$ the Jacobian matrix $$\pmatrix{2 & 2\cr 1 & -1\cr}$$ has characteristic polynomial $P(r) = (2-r)(-1-r) - 2 = r^2 - r -4$. The eigenvalues are the roots of this, namely $(1 \pm \sqrt{17})/2$. One is positive and the other negative, so this is a saddle point, and unstable.

(b)