Then your password (your other, “first” factor) is the only thing preventing an intruder impersonates you.

You’ll still have to go through the hassle the now useless second factor puts you through, so you might as well update your second factor even if you trust your first to be very secure.