Listen man I’ve been working with web scraping for years though now I do the exact opposite (anti bot tech) and robots.txt is absolutely meaningless and there’s zero precedent in the US or elsewhere of it doing anything but providing web crawlers a map of your web site.
I can tell you the thing we tell to all of our clients - the only way to sue bots is to sue for direct damages not for automation. This has always been true and will continue to be true for foreseeable future in the US because you its impossible to set a precedent here as there are just too many players involved that benefit from web automation.
You can actually check out:
Meta v. Bright Data
hiq labs v. inkedIn
These cases are very recent and huge in web automation community and went all the way to the Ninth Circuit and settled at Supreme Court in favor of bots.
I’m telling you man copyright is so ruined that it’s really just a machine for feeding middle managers and lawyers. But hey it gives me a great job security and I can afford to work on actual free software which as you might know is invredibly hard to fund otherwise!
Listen man I’ve been working with web scraping for years though now I do the exact opposite (anti bot tech) and robots.txt is absolutely meaningless and there’s zero precedent in the US or elsewhere of it doing anything but providing web crawlers a map of your web site.
I can tell you the thing we tell to all of our clients - the only way to sue bots is to sue for direct damages not for automation. This has always been true and will continue to be true for foreseeable future in the US because you its impossible to set a precedent here as there are just too many players involved that benefit from web automation.
You can actually check out:
These cases are very recent and huge in web automation community and went all the way to the Ninth Circuit and settled at Supreme Court in favor of bots.
I’m telling you man copyright is so ruined that it’s really just a machine for feeding middle managers and lawyers. But hey it gives me a great job security and I can afford to work on actual free software which as you might know is invredibly hard to fund otherwise!