Dark matter in the cosmological concordance model is parametrized by a single number, describing the covariantly conserved energy density of a nonrelativistic fluid. Here we test this assumption in a model-independent and conservative way by considering the possibility that, at any point during the cosmological evolution, dark matter may be converted into a noninteracting form of radiation. This scenario encompasses, but is more general than, the cases where dark matter decays or annihilates into these states. We show that observations of the cosmic microwave background allow us to strongly constrain this scenario for any conversion time after big bang nucleosynthesis. We discuss in detail, both from a Bayesian and frequentist point of view, in which sense adding large-scale structure observations may even provide a certain preference for a conversion of dark matter to radiation at late times. Finally we apply our general results to a specific particle physics realization of such a scenario, featuring late kinetic decoupling and Sommerfeld-enhanced dark matter annihilation. We identify a small part of parameter space that both mitigates the tension between cosmic microwave and large-scale structure data and allows for velocity-dependent dark matter self-interactions strong enough to address the small-scale problems of structure formation.