The New York Times blocks OpenAI’s web crawler::The New York Times has officially blocked GPTBot, OpenAI’s web crawler. The outlet’s robot.txt page specifically disallows GPTBot, preventing OpenAI from scraping content from its website to train AI models.

    • poke@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      3
      ·
      1 year ago

      They made a flag specifically for their crawler, so they can say that they do but in the most annoying way possible.