@tekeousA to Lemmy Shitpost@lemmy.world • 5 months agorobots.txt is a suggestionusenet.lolimagemessage-square25arrow-up1625arrow-down13
arrow-up1622arrow-down1imagerobots.txt is a suggestionusenet.lol@tekeousA to Lemmy Shitpost@lemmy.world • 5 months agomessage-square25
All this robot.txt stuff “perplex” me.