x' = x*cos(a) - y*sin(a)
y' = x*sin(a) + y*cos(a)
as the following figures illustrate.
In the first figure, the blue triangle has vertex angle a. If r is the radius of the circle, then the radial side of the blue triangle is r cos(a) and its circumferential side is r sin(a).
Therefore in the following figure, the red, pink and magenta triangles are all similar. The pink one is obtained from the red one by a scaling factor of cos(a). The magenta one is obtained from the pink one by combining (1) a rotation through a right angle around one of its vertices and (2) a suitable scale change. The ratio of the size of the magenta one to the red one is sin(a).
Click and drag on the nodes to change (x, y) and a.
If we choose (x, y) to be of length 1, say
x = cos(b)
y = sin(b)
then this gives us the formula for the cosine and sine of a + b.
cos(a + b) = cos(a) cos(b) - sin(a) sin(b)
sin(a + b) = sin(a) cos(b) + cos(a) sin(b)