tekeousA to Lemmy Shitpost@lemmy.world · 1 year agorobots.txt is a suggestionimagemessage-square25fedilinkarrow-up1625arrow-down13
arrow-up1622arrow-down1imagerobots.txt is a suggestiontekeousA to Lemmy Shitpost@lemmy.world · 1 year agomessage-square25fedilink
minus-squarelud@lemm.eelinkfedilinkarrow-up7arrow-down2·1 year agoIndexing is what “scrapers” mostly do. That’s how search engines work. If you don’t allow any scraping don’t be surprised if you get no visitors.
minus-squareAlbum@lemmy.calinkfedilinkarrow-up13·1 year agoSearch engine scrapers index. But that’s a subset of scrapers. There are data scrapers and content scrapers, and these are becoming more prolific as AI takes off and ppl need to feed it data. This post is specifically about AI scrapers.
Indexing is what “scrapers” mostly do.
That’s how search engines work. If you don’t allow any scraping don’t be surprised if you get no visitors.
Search engine scrapers index. But that’s a subset of scrapers.
There are data scrapers and content scrapers, and these are becoming more prolific as AI takes off and ppl need to feed it data.
This post is specifically about AI scrapers.