To provide a better streaming experience, video clients today select their video rates by observing and estimating the available capacity. Recent work has shown that capacity estimation is fraught with difficulties because of complex interactions between the ABR control loop, HTTP server performance and TCP congestion control. Estimation-based rate selection algorithms can lead to unnecessary rebuffering events and suboptimal video quality. This paper argues that we should do away with estimating network capacity, and instead directly observe and control the playback buffer--which is the state variable we are most interested in controlling. We present a class of "buffer-based" rate selection algorithms that reduce the rebuffering rate while allowing us to control the delivered video quality. We implemented our algorithms inside the Netflix video client and ran a series of experiments spanning millions of Netflix users around the world. Our results show that by doing away with estimating network capacity and instead focusing on buffer occupancy, we can reduce rebuffer rates by 20% while holding video rate constant.