Log In Sign Up

paperrater.com
robots.txt

Robots Exclusion Standard data for paperrater.com

Resource Scan

Scan Details

Site Domain paperrater.com
Base Domain paperrater.com
Scan Status Ok
Last Scan2024-09-26T07:59:13+00:00
Next Scan 2024-10-03T07:59:13+00:00

Last Scan

Scanned2024-09-26T07:59:13+00:00
URL https://paperrater.com/robots.txt
Redirect https://www.paperrater.com/robots.txt
Redirect Domain www.paperrater.com
Redirect Base paperrater.com
Domain IPs 18.161.97.107, 18.161.97.26, 18.161.97.51, 18.161.97.88
Redirect IPs 3.164.182.108, 3.164.182.128, 3.164.182.43, 3.164.182.47
Response IP 13.226.2.128
Found Yes
Hash fd3f2430bdfe1900bd8ca01f18fc8d387b5a9a4a8e327ca8b5f0604516329838
SimHash b2540e4d2cc0

Groups

mediapartners-google

Rule Path
Disallow

grapeshot
grapeshotcrawler
proximic
clickagy intelligence bot v2
maxpointcrawler
ias_crawler
a6-indexer
surdotlybot
dotbot
seznambot
istellabot
sogou web spider
bubing
femtosearchbot
archive.org_bot
getintent crawler
ltx71
mj12bot
mbcrawler
kerrigan
ia_archiver

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.paperrater.com/sitemap.xml
Back to top

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • User-Agent: *
  • Disallow: /
  • User-Agent: *
  • Disallow: /site/show_sub/
Back to top