如何用IIS限制屏蔽恶意搜索引擎爬虫禁止抓取网站?
系统运维
51 人阅读
|
1 人回复
|
2024-05-05
|
iis环境下,如何限制屏蔽恶意搜索引擎爬虫禁止抓取网站?
|
|
|
|
|
|
|
大神
发表于 2024-5-5 18:32:04
|
显示全部楼层
IIS7/IIS8/IIS10及以上web服务请在网站根目录下创建web.config文件,并写入如下代码即可:
- <?xml version="1.0" encoding="UTF-8"?>
- <configuration>
- <system.webServer>
- <rewrite>
- <rules>
- <rule name="Block spider">
- <match url="(^robots.txt$)" ignoreCase="false" negate="true" />
- <conditions>
- <add input="{HTTP_USER_AGENT}" pattern="SemrushBot|Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|curl|perl|Python|Wget|Xenu|ZmEu" ignoreCase="true" />
- </conditions>
- <action type="AbortRequest"/>
- </rule>
- </rules>
- </rewrite>
- </system.webServer>
- </configuration>
复制代码
IIS6请在isapi重写组件中添加以下规则:
- #Block spider
- RewriteCond %{HTTP_USER_AGENT} (MegaIndex|MegaIndex.ru|BLEXBot|Qwantify|qwantify|semrush|Semrush|serpstatbot|hubspot|python|Bytespider|Go-http-client|Java|PhantomJS|SemrushBot|Scrapy|Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|perl|Python|Wget|Xenu|ZmEu|^$) [NC]
- RewriteRule !(^/robots.txt$) - [F]
复制代码
|
|
|
|
|
|