Wouldn’t time be going faster for him, slower for them? So whilst his aging and actions increased greatly, theirs slowed down… So they’ve only been away a short while, but he has rapidly gone through 40 or more years of time (quickly).