Log In Sign Up

inthefrow.com
robots.txt

Robots Exclusion Standard data for inthefrow.com

Resource Scan

Scan Details

Site Domain inthefrow.com
Base Domain inthefrow.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-13T21:07:40+00:00
Next Scan 2024-08-11T21:07:40+00:00

Last Successful Scan

Scanned2023-01-18T09:47:34+00:00
URL https://inthefrow.com/robots.txt
Domain IPs 104.21.39.134, 172.67.146.22, 2606:4700:3031::ac43:9216, 2606:4700:3034::6815:2786
Response IP 172.67.146.22
Found Yes
Hash b89cf516de653f76d8fbc434fc8cda860f64283f377fd3d8068c28cb9f666a2c
SimHash 4a4cd742a233

Groups

*

Rule Path
Disallow /xmlrpc.php
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

seokicks-robot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

ezooms robot

Rule Path
Disallow /

perl lwp

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

netestate ne crawler (+http://www.website-datenbank.de/)

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://www.inthefrow.com/sitemap_index.xml
Back to top

Comments

  • Block SEOkicks
  • Block SemrushBot
  • Block BlexBot
  • Block SISTRIX
  • Block Ezooms Robot
  • Block Perl LWP
  • Block BlexBot
  • Block netEstate NE Crawler (+http://www.website-datenbank.de/)
  • Block Searchmetrics Bot
  • Block YandexBot
  • Block Baidu
  • Block SoGou
  • Block Youdao
  • Block MegaIndex.ru
  • Block Dotbot
Back to top