install/apache2
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add 'DOC/README.bad_bot'.

Browse Source
This commit is contained in:
Christoph 2024-06-29 00:29:01 +02:00
parent ef4725adf2
commit 7edc6bce62
1 changed files with 18 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

18
DOC/README.bad_bot Normal file
View File

@ -0,0 +1,18 @@
# ----
# Deny User-Agent (i.e. Bbots not recognizing robots.txt
# ----
# Add to .htavvess
#
SetEnvIfNoCase User-Agent ".*MJ12bot.*" bad_bot
SetEnvIfNoCase User-Agent ".*Baiduspider.*" bad_bot
SetEnvIfNoCase User-Agent ".*Vagabondo.*" bad_bot
SetEnvIfNoCase User-Agent ".*lwp-trivial.*" bad_bot
SetEnvIfNoCase User-Agent ".*libwww.*" bad_bot
SetEnvIfNoCase User-Agent ".*Wget.*" bad_bot
SetEnvIfNoCase User-Agent ".*XoviBot.*" bad_bot
SetEnvIfNoCase User-Agent ".*xovibot.*" bad_bot
SetEnvIfNoCase User-Agent ".*AhrefsBot.*" bad_bot
SetEnvIfNoCase User-Agent ".*ClaudeBot.*" bad_bot
SetEnvIfNoCase User-Agent "SemrushBot" bad_bot
Deny from env=bad_bot