Only if your Android connection is set to automatic DNS. Additionally, they are assuming it is an OS bug. However, they also acknowledge that they had to fix something on their app to mitigate. I tried myself with Wireguard instead, killed the network access to it, and nothing ever left my phone, as Android immediately killed all connections due to the VPN always on feature.

So, I’m going to take their claim with a grain of salt until AOSP says something about this and denies or confirms the alleged bug.