punrca@piefed.world to Linux@programming.devEnglish · 29 days agoI Gave Up on Windows 11. Linux Mint Is Simply Better in 7 Big Ways (PCMAG)plus-squarewww.pcmag.comexternal-linkmessage-square33linkfedilinkarrow-up1209arrow-down17
arrow-up1202arrow-down1external-linkI Gave Up on Windows 11. Linux Mint Is Simply Better in 7 Big Ways (PCMAG)plus-squarewww.pcmag.compunrca@piefed.world to Linux@programming.devEnglish · 29 days agomessage-square33linkfedilink
minus-squarepunrca@piefed.worldtoSelfhosted@lemmy.world•Based on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlerslinkfedilinkEnglisharrow-up30arrow-down3·3 months agoIt’s best to use either Cloudflare (best IMO) or Anubis. If you don’t want any AI bots, then you can setup Anubis (open source; requires JavaScript to be enabled by the end user): https://github.com/TecharoHQ/anubis Cloudflare automatically setups robots.txt file to block “AI crawlers” (but you can setup to allow “AI search” for better SEO). Eg: https://blog.cloudflare.com/control-content-use-for-ai-training/#putting-up-a-guardrail-with-cloudflares-managed-robots-txt Cloudflare also has an option of “AI labyrinth” to serve maze of fake data to AI bots who don’t respect robots.txt file. linkfedilink
It’s best to use either Cloudflare (best IMO) or Anubis.
If you don’t want any AI bots, then you can setup Anubis (open source; requires JavaScript to be enabled by the end user): https://github.com/TecharoHQ/anubis
Cloudflare automatically setups robots.txt file to block “AI crawlers” (but you can setup to allow “AI search” for better SEO). Eg: https://blog.cloudflare.com/control-content-use-for-ai-training/#putting-up-a-guardrail-with-cloudflares-managed-robots-txt
Cloudflare also has an option of “AI labyrinth” to serve maze of fake data to AI bots who don’t respect robots.txt file.