wp-data.com
robots.txt

Robots Exclusion Standard data for wp-data.com

Resource Scan

Scan Details

Site Domain wp-data.com
Base Domain wp-data.com
Scan Status Ok
Last Scan2025-10-06T05:39:54+00:00
Next Scan 2025-10-13T05:39:54+00:00

Last Scan

Scanned2025-10-06T05:39:54+00:00
URL https://wp-data.com/robots.txt
Redirect https://www.wp-data.com/robots.txt
Redirect Domain www.wp-data.com
Redirect Base wp-data.com
Domain IPs 15.165.95.209
Redirect IPs 15.165.95.209
Response IP 15.165.95.209
Found Yes
Hash 144a5851e25dfa9f5fe6a3476154a7d3cd3dc166a059d82a9f1be7f7f57dc96f
SimHash f0bc73456fb9

Groups

*

Rule Path
Disallow /assets/
Disallow /assets-frontpage/

test-bot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

netseer

Rule Path
Disallow /

pinterestbot

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

bidswitchbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

owler

Rule Path
Disallow /

tracemyfile

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

hybridbot

Rule Path
Disallow /

feedly

Rule Path
Disallow /

feedburner

Rule Path
Disallow /

boardreader

Rule Path
Disallow /

theoldreader.com

Rule Path
Disallow /

semantic-visions.com

Rule Path
Disallow /

proximic

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

ias_crawler

Rule Path
Disallow /

Comments

  • http://www.bing.com/bingbot.htm
  • http://www.trendiction.com/en/publisher/bot
  • http://www.apple.com/go/applebot
  • http://www.grapeshot.co.uk/crawler.php
  • http://www.semrush.com/bot.html
  • http://www.semrush.com/bot.html
  • http://mj12bot.com
  • http://www.opensiteexplorer.org/dotbot
  • http://www.netseer.com/crawler.html
  • http://www.pinterest.com/bot.html
  • http://getintent.com/bot.html