Log In Sign Up

extralunchmoney.com
robots.txt

Robots Exclusion Standard data for extralunchmoney.com

Resource Scan

Scan Details

Site Domain extralunchmoney.com
Base Domain extralunchmoney.com
Scan Status Ok
Last Scan2024-09-19T19:34:40+00:00
Next Scan 2024-10-19T19:34:40+00:00

Last Scan

Scanned2024-09-19T19:34:40+00:00
URL https://extralunchmoney.com/robots.txt
Domain IPs 104.22.28.220, 104.22.29.220, 172.67.10.27, 2606:4700:10::6816:1cdc, 2606:4700:10::6816:1ddc, 2606:4700:10::ac43:a1b
Response IP 172.67.10.27
Found Yes
Hash d5066a02aed0c78ef1e004a1571c1674a805624cb993b3069a5d4b1b92807952
SimHash 2809db01c6bd

Groups

*

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 600

yandexbot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

twitterbot

Rule Path
Disallow

*

Rule Path
Disallow /

googlebot
googlebot-image
mediapartners-google
msnbot
msnbot-media
slurp
yahoo-blogs
yahoo-mmcrawler
bingbot/2.0
baiduspider-image+
msnbot-media/1.1
duckduckbot/1.0

Rule Path
Disallow /providers/gallery/registration-popup
Disallow /externals/*
Disallow /jobs/ethnicity%3A*
Disallow /jobs/amount%3A*
Disallow /jobs/profile_type%3A*
Disallow /jobs/gender%3A*
Disallow /jobs/orderby%3A*

Other Records

Field Value
crawl-delay 600
Back to top

Comments

  • disallow all
  • but allow only important bots
  • Directories
  • Sitemap: http://extralunchmoney.com/sitemap_files/sitemap-index.xml
Back to top