tekeousA to Lemmy Shitpost@lemmy.world · 1 year agorobots.txt is a suggestionimagemessage-square24fedilinkarrow-up1625arrow-down13
arrow-up1622arrow-down1imagerobots.txt is a suggestiontekeousA to Lemmy Shitpost@lemmy.world · 1 year agomessage-square24fedilink
minus-squaremarcos@lemmy.worldlinkfedilinkarrow-up13arrow-down41·1 year agoNothing in my post (or in robots.txt) has any relation to distributing your content.
minus-squareBlackPenguins@lemmy.worldlinkfedilinkarrow-up24arrow-down1·edit-21 year agoWhat else would they scrape your data for? Sure some could be for personal use but most of the time it will be to redistribute in a new medium. Like a recipe app importing recipes.
minus-squareAlbum@lemmy.calinkfedilinkarrow-up13·1 year agoSearch engine scrapers index. But that’s a subset of scrapers. There are data scrapers and content scrapers, and these are becoming more prolific as AI takes off and ppl need to feed it data. This post is specifically about AI scrapers.
minus-squareShiggles@sh.itjust.workslinkfedilinkarrow-up5arrow-down1·1 year agoI think by shitty they mean ineffective, not poor manners.
Nothing in my post (or in robots.txt) has any relation to distributing your content.
What else would they scrape your data for? Sure some could be for personal use but most of the time it will be to redistribute in a new medium. Like a recipe app importing recipes.
deleted by creator
Search engine scrapers index. But that’s a subset of scrapers.
There are data scrapers and content scrapers, and these are becoming more prolific as AI takes off and ppl need to feed it data.
This post is specifically about AI scrapers.
I think by shitty they mean ineffective, not poor manners.