except it doesn’t work well for the rest of lemmy/the fediverse.
many other instances seem to be getting hit by this, but they don’t have as many activities generated locally for this to become much of a problem. additionally, this is mostly affecting instances with high latency to the instance that is being flooded by kbin, as lemmy currently has an issue where activity throughput between instances with high latency can’t keep up with too many activities being sent. the impact of this is can be a bit less on smaller instances with smaller communities often not having as many subscribers on remote instances, although we’ve seen problems reported by some other admins as well. this includes e.g. kbin.earth, which i suspect to have been hit by responses from a lemmy instance, while the lemmy instance was actually only answering the requests sent from that kbin instance.
during the last peak, when we decided to pull the plug for now, kbin.social was sending us more than 20 activities per second for 7 hours straight. lemmy.world can easily handle this amount of activities, but the problem arises when this impacts our federation towards other (lemmy) instances, as e.g. votes will get relayed by the community (magazine) instance, which means, depending on the type of activity being sent, we might have to be sending out the same 20 requests per second to up to 4,000+ other fediverse instances that are subscribed/following the community this is happening in. trying to send 20 requests per second, which lemmy does not do in parallel, requires us to use at most 50ms per activity total sending time to avoid creating lag. when the instance is in australia, with 200ms+ latency, this is simply not possible.
ps: if you’re wondering how i’m seeing this post, you can search for a post url and comment urls on lemmy to make lemmy fetch them, even if they haven’t been directly submitted through normal federation processes. this requires a logged in user on lemmy’s end.
except it doesn’t work well for the rest of lemmy/the fediverse.
many other instances seem to be getting hit by this, but they don’t have as many activities generated locally for this to become much of a problem. additionally, this is mostly affecting instances with high latency to the instance that is being flooded by kbin, as lemmy currently has an issue where activity throughput between instances with high latency can’t keep up with too many activities being sent. the impact of this is can be a bit less on smaller instances with smaller communities often not having as many subscribers on remote instances, although we’ve seen problems reported by some other admins as well. this includes e.g. kbin.earth, which i suspect to have been hit by responses from a lemmy instance, while the lemmy instance was actually only answering the requests sent from that kbin instance.
during the last peak, when we decided to pull the plug for now, kbin.social was sending us more than 20 activities per second for 7 hours straight. lemmy.world can easily handle this amount of activities, but the problem arises when this impacts our federation towards other (lemmy) instances, as e.g. votes will get relayed by the community (magazine) instance, which means, depending on the type of activity being sent, we might have to be sending out the same 20 requests per second to up to 4,000+ other fediverse instances that are subscribed/following the community this is happening in. trying to send 20 requests per second, which lemmy does not do in parallel, requires us to use at most 50ms per activity total sending time to avoid creating lag. when the instance is in australia, with 200ms+ latency, this is simply not possible.
looking at the activity generation rates of some popular lemmy instances, anything that is significantly above lemmy.world is likely not just sending legitimate activities.
ps: if you’re wondering how i’m seeing this post, you can search for a post url and comment urls on lemmy to make lemmy fetch them, even if they haven’t been directly submitted through normal federation processes. this requires a logged in user on lemmy’s end.