The algorithm used to randomly create the rotation matrix is due to R Salomon (see reference). No guarantee is given that the generated rotation matricies are uniformly distributed in any sense.
Dimension of desired rotation matrix.
A random d \times d rotation matrix.
Salomon R. Re-evaluating genetic algorithm performance under coordinate rotation of benchmark functions. A survey of some theoretical and practical aspects of genetic algorithms. Biosystems. 1996;39(3):263-78.