Body size can have profound impacts on survival, movement, and reproductive schedules shaping individual fitness, making growth a central process in ecological and evolutionary dynamics. Realized growth is the result of a complex interplay between life history schedules, individual variation, and environmental influences. Integrating all of these aspects into growth models is methodologically difficult, depends on the availability of repeated measurements of identifiable individuals, and consequently represents a major challenge in particular for natural populations. Using a unique 30‐yr time series of individual length measurements inferred from scale year rings of wild brown trout, we develop a Bayesian hierarchical model to estimate individual growth trajectories in temporally and spatially varying environments. We reveal a gradual decrease in average juvenile growth, which has carried over to adult life and contributed to decreasing sizes observed at the population level. Commonly studied environmental drivers like temperature and water flow did not explain much of this trend and overall persistent and among‐year individual variation dwarfed temporal variation in growth patterns. Our model and results are relevant to a wide range of questions in ecology and evolution requiring a detailed understanding of growth patterns, including conservation and management of many size‐structured populations.