The conventional non-coherent differential detection of continuous phase modulation (CPM) is quite robust to channel impairments such as phase and Doppler shifts. Its implementation is on top of that simple. It consists in multiplying the received baseband signal by its conjugate and delayed version of one symbol period. However it suffers from a signal-to-noise ratio gap compared to the optimum coherent detection. In this paper, we improve the error rate performance of the conventional differential detection by using a delay higher than one symbol period. We derive the trellis description as well as the branch and cumulative metric that take into account a delay of $K$ symbol periods. We then derive an optimization criterion of $K$ based on the minimum Euclidean distance between two differential signals. We finally determine optimized delays for some popular CPM formats whose values are confirmed by error rate simulations.