Maybe it has to do with their interaction with the time vortex or the TARDIS. As soon as the doctor stopped traveling and stayed in one place he started aging.