We present a new higher-order accurate finite difference explicit jump Immersed Interface Method (EJIIM) for solving two-dimensional elliptic problems with singular source and discontinuous coefficients in the irregular region on a compact Cartesian mesh. We propose a new strategy for discretizing the solution at irregular points on a nine point compact stencil such that the high order compactness is maintained throughout the whole computational domain. The scheme is employed to solve four problems embedded with circular and star shaped interfaces in a rectangular region having analytical solutions and varied discontinuities across the interface in source and the coefficient terms. In the process, we show the superiority of the proposed strategy over the EJIIM and other existing IIM methods. We also simulate the steady-state flow past a circular cylinder governed by the Navier-Stokes equations. In all the cases our computed results extremely close to the available numerical and experimental results.