crowdappeal.com
robots.txt

Robots Exclusion Standard data for crowdappeal.com

Resource Scan

Scan Details

Site Domain crowdappeal.com
Base Domain crowdappeal.com
Scan Status Ok
Last Scan2025-05-08T08:36:35+00:00
Next Scan 2025-06-07T08:36:35+00:00

Last Scan

Scanned2025-05-08T08:36:35+00:00
URL https://crowdappeal.com/robots.txt
Domain IPs 62.122.191.132
Response IP 62.122.191.132
Found Yes
Hash dcd8ce7d97cb8b713cd882d5fb4440841c9db97a632e707303c806081f6fd38c
SimHash ed0ed8540513

Groups

*

Rule Path
Disallow /_components
Disallow /_css
Disallow /_js
Disallow /_pro
Disallow /ic
Disallow /sub

Other Records

Field Value
crawl-delay 10

*

Rule Path
Allow /

googlebot/2.1
infonavirobot(f107)
tv33_mercator_1-1.0
avsearch-3.0
scooter/2.0
slurp/2.0
searchenginelicencesheep_v1.0
shadow/2.0
multitext/0.1
fast-webcrawler/2.2.5
atomz/1.0
htdig/ (searchit@netmind.com)
spider00.logika.net.

Rule Path
Disallow /searchtools-rss.xml

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

Comments

  • don't let search engines see the RDF feed, it's just confusing.
  • updated 2002-11-11