Popular social media users play a major role in the COVID-19 infodemic, as they can influence public opinion through their massive reach. We study the network of influential users discussing the pandemic on Twitter, where we consider users as nodes and following relationships as directed edges. The resulting structure is modeled by embedding the actors in a latent social space where users closer to one another have a higher probability of forming edges, thus producing a social map of the COVID-19 Twitter universe. The results suggest the existence of two different user communities, which can be interpreted as "generally pro" and "generally against" vaccine mandates, corroborating the presence of echo chamber effects. We further show that the two groups are not entirely homogeneous: the latent space accurately describes an entire spectrum of beliefs between the two extremes, demonstrating that polarization, while present, is not the only driver of the discussion.