Caching popular files in small base stations (SBSs) has been proved to be an effective way to reduce bandwidth pressure on the backhaul links of dense small cell networks (DSCNs). Many existing studies on cache-enabled DSCNs attempt to improve user experience by optimizing end-to-end file delivery delay. However, under practical scenarios where files (e.g., video files) have diverse quality of service requirements, energy consumption at SBSs should also be concerned from the network perspective. In this paper,we attempt to optimize these two critical metrics in cache-enabled DSCNs. Firstly, we formulate the energy-delay optimization problem as a Mixed Integer Programming (MIP) problem, where file placement, user association and power control are jointly considered. To model the tradeoff relationship between energy consumption and end-to-end file delivery delay, a utility function linearly combining these two metrics is used as an objective function of the optimization problem. Then, we solve the problem in two stages, i.e. caching stage and delivery stage, based on the observation that caching is performed during off-peak time. At the caching stage, a local popular file placement policy is proposed by estimating user preference at each SBS. At the delivery stage, with given caching status at SBSs, the MIP problem is further decomposed by Benders' decomposition method. An efficient algorithm is proposed to approach the optimal association and power solution by iteratively shrinking the gap of the upper and lower bounds. Finally, extension simulations are performed to validate our analytical and algorithmic work. The results demonstrate that the proposed algorithms can achieve the optimal tradeoff between energy consumption and end-to-end file delivery delay.