Log In Sign Up

cekresipaket.com
robots.txt

Robots Exclusion Standard data for cekresipaket.com

Resource Scan

Scan Details

Site Domain cekresipaket.com
Base Domain cekresipaket.com
Scan Status Ok
Last Scan2024-09-22T01:18:57+00:00
Next Scan 2024-09-29T01:18:57+00:00

Last Scan

Scanned2024-09-22T01:18:57+00:00
URL https://cekresipaket.com/robots.txt
Domain IPs 139.180.190.175
Response IP 139.180.190.175
Found Yes
Hash 8c467bceacbae1565d0bfed039576c4f7006b9504f4aa4e68bab653ccc7e8d7e
SimHash 325ccc10a130

Groups

newspaper/0.1.0.7

Rule Path
Disallow /

newspaper/0.2.5

Rule Path
Disallow /

newspaper/0.2.6

Rule Path
Disallow /

newspaper/0.2.7

Rule Path
Disallow /

newspaper/0.2.8

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

mozilla/5.0 (compatible; dataforseobot/1.0; +https://dataforseo.com/dataforseo-bot)

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

barkrowler/0.9

Rule Path
Disallow /

mozilla/5.0 (compatible; barkrowler/0.9; +https://babbar.tech/crawler)

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mozilla/5.0 (compatible; mj12bot/v1.4.8; http://mj12bot.com/)

Rule Path
Disallow /

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

mozilla/5.0 (compatible; ahrefssiteaudit/6.1; +http://ahrefs.com/robot/)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

dotbot/1.2

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

spbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

seokicks

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

seokicks-robot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

*

No rules defined. All paths allowed.

Back to top

Other Records

Field Value
sitemap https://cekresipaket.com/sitemap.xml
Back to top

Comments

  • http://webmeup-crawler.com
  • https://dataforseo.com
  • https://www.exensa.com/crawl/
  • http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
  • http://www.ahrefs.com
  • https://www.semrush.com/bot/
  • http://www.opensiteexplorer.org/dotbot
  • http://openlinkprofiler.org/bot
  • http://www.moz.com/dp/rogerbot
  • http://www.seokicks.de
Back to top