In many real-world situations, there are strong constraints on the ways in which a physical system can be manipulated. We investigate the entropy production (EP) and extractable work involved in bringing a system from some initial distribution $p$ to some final distribution $p'$, given constraints on the set of master equations available to the driving protocol. Given some set of constraints, we first derive general bounds on EP and extractable work, as well as a decomposition of the nonequilibrium free energy into an "accessible free energy", which can be extracted as work, and "inaccessible free energy", which must be dissipated as EP. We then analyze the thermodynamics of feedback control in the presence of constraints, and decompose the information acquired in a measurement into "accessible information", which can be used to increase extracted work, and "inaccessible information", which cannot be used in this way. We use our framework to analyze EP and work for protocols subject to symmetry, modularity, and coarse-grained constraints. Our approach is demonstrated on numerous continuous and discrete-state systems, including different kinds of Szilard boxes.