Leaving Lemmy for Threads was never really a concern I heard from anyone, they are completely different services. Like you say, Mastodon is more at risk.

What people on Lemmy were concerned with was rather whether undesirable users on Threads could filter onto Lemmy and negatively affect the user experience here. With the magnitudes larger userbase, Threads users could potentially dominate the Hot feed if they started posting to Lemmy communities.