In an oriented graph $\vec{G}$, the {\it inversion} of a subset $X$ of vertices consists in reversing the orientation of all arcs with both endvertices in $X$. The {\it $(\leq p)$-inversion graph} of a labelled graph $G$, denoted by ${\mathcal{I}}^{\leq p}(G)$, is the graph whose vertices are the labelled orientations of $G$ in which two labelled orientations $\vec{G}_1$ and $\vec{G}_2$ of $G$ are adjacent if and only if there is a set $X$ with $|X|\leq p$ whose inversion transforms $\vec{G}_1$ into $\vec{G}_2$. In this paper, we study the {\it $(\leq p)$-inversion diameter} of a graph, denoted by $\mathrm{id}^{\leq p}(G)$, which is the diameter of its $(\leq p)$-inversion graph. We show that there exists a smallest number $\Psi_p$ with $\frac{1}{4}p - \frac{3}{2} \leq \Psi_p \leq \frac{1}{2}p^2$ such that $\mathrm{id}^{\leq p}(G) \leq \left\lceil\frac{|E(G)|}{\lfloor p/2\rfloor}\right \rceil + \Psi_p$ for all graph $G$. We then establish better upper bounds for several families of graphs and in particular trees and planar graphs. Let us denote by $\mathrm{id}^{\leq p}_{\cal F}(n)$ (resp. $\mathrm{id}^{\leq p}_{\cal P}(n)$) the maximum $(\leq p)$-inversion diameter of a tree (resp. planar graph) of order $n$. For trees, we show $\mathrm{id}^{\leq 3}_{\cal F}(n) = \left\lceil \frac{n-1}{2}\right\rceil$, $\mathrm{id}^{\leq 4}_{\cal F}(n)=\frac{3}{8}n + \Theta(1)$, $\mathrm{id}^{\leq 5}_{\cal F}(n)= \frac{2}{7}n + \Theta(1)$, and $\mathrm{id}^{\leq p}_{\cal F}(n) \leq \frac{n-1}{p- c\sqrt{p}} + 2$ with $c = \sqrt{2 + \sqrt{2}}$ for all $p\geq 6$. For planar graphs, we prove $\mathrm{id}^{\leq 3}_{\cal P}(n) \leq \frac{11n}{6} - \frac{8}{3}$, $\mathrm{id}^{\leq 4}_{\cal P}(n) \leq \frac{4n}{3} + \frac{10}{3}$, and $\mathrm{id}^{\leq p}_{\cal P}(n) \leq \left\lceil\frac{3n-6}{\lfloor p/2\rfloor}\right \rceil + 8\lfloor p/2\rfloor - 8$ for all $p\geq 6$.