• BrianTheeBiscuiteer@lemmy.world
    link
    fedilink
    English
    arrow-up
    55
    ·
    9 months ago

    If it doesn’t get queried that’s the fault of the webscraper. You don’t need JS built into the robots.txt file either. Just add some line like:

    here-there-be-dragons.html
    

    Any client that hits that page (and maybe doesn’t pass a captcha check) gets banned. Or even better, they get a long stream of nonsense.