We focus on robot navigation in crowded environments. The challenge of predicting the motion of a crowd around a robot makes it hard to ensure human safety and comfort. Recent approaches often employ end-to-end techniques for robot control or deep architectures for high-fidelity human motion prediction. While these methods achieve important performance benchmarks in simulated domains, dataset limitations and high sample complexity tend to prevent them from transferring to real-world environments. Our key insight is that a low-dimensional representation that captures critical features of crowd-robot dynamics could be sufficient to enable a robot to wind through a crowd smoothly. To this end, we mathematically formalize the act of passing between two agents as a rotation, using a notion of topological invariance. Based on this formalism, we design a cost functional that favors robot trajectories contributing higher passing progress and penalizes switching between different sides of a human. We incorporate this functional into a model predictive controller that employs a simple constant-velocity model of human motion prediction. This results in robot motion that accomplishes statistically significantly higher clearances from the crowd compared to state-of-the-art baselines while maintaining competitive levels of efficiency, across extensive simulations and challenging real-world experiments on a self-balancing robot.