iq-mag.net
robots.txt

Robots Exclusion Standard data for iq-mag.net

Resource Scan

Scan Details

Site Domain iq-mag.net
Base Domain iq-mag.net
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-01T12:24:14+00:00
Next Scan 2024-10-31T12:24:14+00:00

Last Successful Scan

Scanned2024-07-04T11:58:41+00:00
URL https://iq-mag.net/robots.txt
Domain IPs 104.21.13.121, 172.67.167.243, 2606:4700:3032::ac43:a7f3, 2606:4700:3035::6815:d79
Response IP 104.21.13.121
Found Yes
Hash 7d040eb9dbc5223fe0892c8906e7a4aa8980b02edd6189e51db103b826c377fc
SimHash 1a580940e431

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Allow /wp-content/uploads/
Disallow /wp-content/plugins/
Disallow /publications/global-arena-guide/directory/
Disallow /directory-category
Disallow /wp-admin/
Disallow /readme.html
Disallow /refer/

ia_archiver-web.archive.org

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

solomonobot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

prodvigator

Rule Path
Disallow /

prodvigatorbot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

xenu

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

serpstat

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.iq-mag.net/sitemap_index.xml

Comments

  • (start) suspicious bot list
  • (end) suspicious bot list