A function $g: \R^d \to \R$ is radial or invariant by
rotations if it happens that $g(R(x)) = g(x)$ for all $x \in \R^d$
and all rotations $R: \R^d \to \R^d$ (linear orthogonal
transformations with determinant 1). The set of real or complex radial
functions is clearly a vector space, and we may consider the
projection of a function to it. What is its expression?
More precisely, consider $L^2$ the usual real or complex Hilbert space
of square-integrable functions on $\R^d$, with the inner product
$\langle{ \cdot,\cdot }\rangle$ and norm denoted by $\|\cdot\|$. Call $\mathcal{R}$ the
subspace of $L^2$ formed by all functions which are invariant by
rotations. What is the expression of the orthogonal projection to
$\mathcal{R}$?
Notice that $\mathcal{R}$ is a closed subspace of $L^2$, so we are
entitled to consider the orthogonal projection to it: if $R$ is a
rotation and $f \in L^2$ we can define the rotation of $f$ by $Rf:\R^d
\to \R$, $Rf(x) = f(Rx)$. This $R:L^2 \to L^2$ is an isometry
(continuous in particular), and $\mathcal{R}$ is the set of all functions $g$
such that $Rg = g$ for all rotations $R$; that is, $\mathcal{R}$ is the
intersection of all sets of the form $(R-I)^{-1}(\{0\})$ for $R$ a
rotation, which is an intersection of closed sets and hence closed.
Take $f \in L^2$. Its projection to $\mathcal{R}$, which we call $\pi(f)$, is
defined as the element $g \in \mathcal{R}$ which is closest to $f$; that is,
such that
\begin{equation*}
\min_{g \in \mathcal{R}} \|f-g\| = \|f - \pi(f)\|.
\end{equation*}
One way to find it is the following: for all rotations $R$ and any $g
\in \mathcal{R}$ we can write
\begin{equation*}
\|f-g\|^2 =
\ird |f(x) - g(x)|^2 \d x
=
\ird |f(Rx) - g(Rx)|^2 \d x
=
\ird |f(Rx) - g(x)|^2 \d x.
\end{equation*}
Consider normalised Haar measure on $\mathcal{O}$.
Averaging by all rotations $R \in \mathcal{O}$ we have then
\begin{multline*}
\|f-g\|^2
=
\int_\mathcal{O}
\ird |f(Rx) - g(x)|^2 \d x
\d R
=
\ird
\int_\mathcal{O} |f(Rx) - g(x)|^2 \d R \d x
\\
\geq
\ird
\Big| \int_\mathcal{O} f(Rx) \d R - g(x) \Big|^2\d x.
\end{multline*}
This shows then that
\begin{equation*}
\pi(f) = \int_{\mathcal{O}} f(Rx) \d R,
\end{equation*}
since we have proved the function defined by this expression is the
closest one to $f$ among all functions in $\mathcal{R}$. Of course, it is in fact in $\mathcal{R}$:
for any rotation $T$,
\begin{equation*}
T \int_\mathcal{O} Rf \d R
= \int_\mathcal{O} TRf \d R
= \int_\mathcal{O} Rf \d R,
\end{equation*}
since the measure on $\mathcal{O}$ is Haar measure.
