We present a new perturbative approach to ‘constrained Ly α radiative transfer’ (RT) through the circum- and intergalactic medium (CGM and IGM). We constrain the H I content and kinematics of the CGM and IGM in a physically motivated model, using the galaxy-Ly α forest clustering data from spectroscopic galaxy surveys in quasar fields at z ∼ 2−3. This enables us to quantify the impact of the CGM/IGM on Ly α emission in an observationally constrained, realistic cosmological environment. Our model predicts that the CGM and IGM at these redshifts transmit ≈80 per cent of Ly α photons after having escaped from galaxies. This implies that while the interstellar medium primarily regulates Ly α escape, the CGM has a nonnegligible impact on the observed Ly α line properties and the inferred Ly α escape fraction, even at z ∼ 2−3. Ly α scattering in the CGM and IGM further introduces an environmental dependence in the (apparent) Ly α escape fraction, and the observed population of Lyα emitting galaxies: the CGM/IGM more strongly suppresses direct Ly α emission from galaxies in overdense regions in the Universe, and redistributes this emission into brighter Ly α haloes. The resulting mean surface brightness profile of the Ly α haloes is generally found to be a power law ∝r−2.4. Although our model still contains arbitrariness, our results demonstrate how (integral field) spectroscopic surveys of galaxies in QSO fields constrain circumgalactic Ly α RT, and we discuss the potential of these models for studying CGM physics and cosmology.