You might remember a company called Google. In 1998, they came up with an amazing product — a web search that didn’t suck! And it could find things! With the state of search in 1998, this was incre…
  • smeenz@lemmy.nzEnglish
    2·
    14 days ago

    Then the user agent string will just quietly become randomised so you can’t match it reliably because it turns out that honouring robots.txt was always little more than a “gentleman’s handshake”.

      • Evinceo@awful.systemsEnglish
        2·
        11 days ago

        Yeah an adversary like Google isn’t something you can easily block without really annoying legitimate users unfortunately. Nothing is stopping them from turning every chrome instance into a botnet node except for the angry article that would run in Ars Technica.