Log In Sign Up

getedfunding.com
robots.txt

Robots Exclusion Standard data for getedfunding.com

Resource Scan

Scan Details

Site Domain getedfunding.com
Base Domain getedfunding.com
Scan Status Ok
Last Scan2024-09-08T17:32:43+00:00
Next Scan 2024-10-08T17:32:43+00:00

Last Scan

Scanned2024-09-08T17:32:43+00:00
URL http://getedfunding.com/robots.txt
Domain IPs 64.71.133.21
Response IP 64.71.133.21
Found Yes
Hash 823200b514586de237f6393830b93c88a9a9d2606a043f75798a5c9b883bfb1e
SimHash 38705be3cb51

Groups

*

Rule Path
Disallow /w/

Other Records

Field Value
crawl-delay 30

ahrefsbot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-news

Rule Path
Disallow /

baiduspider-favo

Rule Path
Disallow /

baiduspider-cpro

Rule Path
Disallow /

baiduspider-ads

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

exabot

Rule Path
Disallow /

orangebot

Rule Path
Disallow /

scrapybot

Rule Path
Disallow /

contacts-crawler

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

mqqbrowser

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /
Back to top

Comments

  • robot exclusion for site
  • The file consists of one or more records separated by one or more blank lines
  • Disallow: /dev
  • Disallow: /c/*/dev
  • XoviBot
  • SemrushBot
  • Baiduspider
  • User-agent: Yahoo! Slurp
  • Disallow: /
  • User-agent:
  • Disallow: /
Back to top