This is for Web Site Development -- Is this a good idea OR NOT for ALLOWING only Good Bots in robots.txt file?
https://www.ditig.com/publications/robots-txt-template
On the surface it seems better solution than a "Very Long List of Disallows!"
With all these Bad Bots running amok I also block their User Agent, IP or even IP Range.
WebDev: Robots.txt - ALLOWING only Good Bots?
Forum rules
Do not post support questions here. Before you post read the forum rules. Topics in this forum are automatically closed 30 days after creation.
Do not post support questions here. Before you post read the forum rules. Topics in this forum are automatically closed 30 days after creation.
WebDev: Robots.txt - ALLOWING only Good Bots?
Last edited by LockBot on Wed Apr 24, 2024 9:50 pm, edited 1 time in total.
Reason: Topic automatically closed 30 days after creation. New replies are no longer allowed.
Reason: Topic automatically closed 30 days after creation. New replies are no longer allowed.
Re: WebDev: Robots.txt - ALLOWING only Good Bots?
In my experience, it's a digital version of whac-a-mole. Also, there's no international law to say that web crawlers must obey the robots.txt contents. So in reality, you're only blocking those that abide by the "rules". So you can request that your least favorite big-tech stops indexing your site, but it won't do anything to prevent some nasty entity trying to break into your site in order to infect it with the malware of the day.
Individual IP addresses are much the same, as the nasty ones just move on to another address.
Address blocks might feel more effective. Though blocking out whole countries is tricky as the IP address ranges are not contiguous and nothing is stopping a hacker spinning up a cloud instance residing in your own country.
By all means, go for it, and you'll slow things down for a while. Eventually though, you'll probably grow weary of updating your allow/deny lists and admin defeat
Individual IP addresses are much the same, as the nasty ones just move on to another address.
Address blocks might feel more effective. Though blocking out whole countries is tricky as the IP address ranges are not contiguous and nothing is stopping a hacker spinning up a cloud instance residing in your own country.
By all means, go for it, and you'll slow things down for a while. Eventually though, you'll probably grow weary of updating your allow/deny lists and admin defeat
Re: WebDev: Robots.txt - ALLOWING only Good Bots?
Sorry for a long delay in the reply... I've been testing a modified ditig's robots.txt file, it's working great thus far...
Good bots can pass, and bad bots still does their thing; which causes me to block their hosts... You know who they are; go on you will never miss them!
Good bots can pass, and bad bots still does their thing; which causes me to block their hosts... You know who they are; go on you will never miss them!