pe.top10place.com
robots.txt

Robots Exclusion Standard data for pe.top10place.com

Resource Scan

Scan Details

Site Domain pe.top10place.com
Base Domain top10place.com
Scan Status Ok
Last Scan2024-04-20T02:56:29+00:00
Next Scan 2024-05-20T02:56:29+00:00

Last Scan

Scanned2024-04-20T02:56:29+00:00
URL https://pe.top10place.com/robots.txt
Domain IPs 104.26.14.216, 104.26.15.216, 172.67.75.71, 2606:4700:20::681a:ed8, 2606:4700:20::681a:fd8, 2606:4700:20::ac43:4b47
Response IP 172.67.75.71
Found Yes
Hash 95ae5a93131fc7641771d3c9a82090e6b6b4808bd663535de809daddd19f4cb9
SimHash 6e1dd8c346b3

Groups

*

Rule Path
Disallow /cache/
Disallow /members/
Disallow /cron.php
Disallow /ajax.php
Disallow /maintenance.php
Disallow /error.html
Disallow /*/send-message.html
Disallow /*/send-message-friend.html
Disallow /*/add-review.html
Disallow /*/suggestion.html
Disallow /*/claim.html

mediapartners-google

Rule Path
Disallow

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 1

adsbot-google

Rule Path
Disallow

Other Records

Field Value
crawl-delay 1

googlebot-image

Rule Path
Disallow

Other Records

Field Value
crawl-delay 5

googlebot-mobile

Rule Path
Disallow

Other Records

Field Value
crawl-delay 1

googleother

Rule Path
Disallow

Other Records

Field Value
crawl-delay 5

msnbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 20

bingbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 5

slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 360

yahoo! slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 360

yandexbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 8

yandex

Rule Path
Disallow

Other Records

Field Value
crawl-delay 8

wasalive-bot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 180

kulturarw3

Rule Path
Disallow

Other Records

Field Value
crawl-delay 180

facebookexternalhit

Rule Path
Disallow

Other Records

Field Value
crawl-delay 8

ahrefsbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 360

mj12bot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 360

blexbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 360

proximic

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 360

seokicks-robot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 360

admantx

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 360

dotbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 360

wiederfreibot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 360

cliqzbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 360

grapeshotcrawler

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 360

seznambot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 360

coccocbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10