Log In Sign Up

ien.com
robots.txt

Robots Exclusion Standard data for ien.com

Resource Scan

Scan Details

Site Domain ien.com
Base Domain ien.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-11-16T09:03:33+00:00
Next Scan 2024-12-16T09:03:33+00:00

Last Successful Scan

Scanned2024-10-18T05:49:15+00:00
URL https://ien.com/robots.txt
Redirect https://www.ien.com/robots.txt
Redirect Domain www.ien.com
Redirect Base ien.com
Domain IPs 89.106.200.1
Redirect IPs 104.22.66.237, 104.22.67.237, 172.67.29.122, 2606:4700:10::6816:42ed, 2606:4700:10::6816:43ed, 2606:4700:10::ac43:1d7a
Response IP 172.67.29.122
Found Yes
Hash 8f23d11921a1d21a3d5c2452285c4a2823175b84fc469dad0ecbdfcb6f8c0474
SimHash 500c88416ab1

Groups

*

Rule Path
Disallow /__
Disallow /ad-preview
Disallow /*/21427920/
Disallow /*/21451897/
Disallow /print/content

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

dataforseobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

imagesiftbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.ien.com/sitemap.xml
sitemap https://www.ien.com/sitemap-google-news.xml
Back to top

Comments

  • www.robotstxt.org/
  • www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
Back to top