The curve is more apparent if you stick to realWho.

And individuals age at different rates. Troughton looked older than his years at the time; Davison looked younger than his.

Anyway, we can go round and round indefinitely on this. I think older actors are generally better for that part, for reasons I've already stated. If you don't, that's fine: I don't necessarily expect anyone to agree.