We report on a simple method allowing one to decompose the duration of arbitrary ultrashort light pulses, potentially distorted by space-time coupling, into four elementary durations. Such a decomposition shows that, in linear optics, a spatio-temporal pulse can be stretched with respect to its Fourier limit by only three independent phenomena: nonlinear frequency dependence of the spectral phase over the whole spatial extent of the pulse, spectral amplitude inhomogeneities in space, and spectral phase inhomogeneities in space. We illustrate such a decomposition using numerical simulations of complex spatio-temporal femtosecond and attosecond pulses. Finally we show that the contribution of two of these three effects to the pulse duration is measurable without any spectral phase characterization.