Log In Sign Up

smartpaperhelps.com
robots.txt

Robots Exclusion Standard data for smartpaperhelps.com

Resource Scan

Scan Details

Site Domain smartpaperhelps.com
Base Domain smartpaperhelps.com
Scan Status Ok
Last Scan2024-11-16T04:13:24+00:00
Next Scan 2024-12-16T04:13:24+00:00

Last Scan

Scanned2024-11-16T04:13:24+00:00
URL https://smartpaperhelps.com/robots.txt
Domain IPs 147.182.132.102
Response IP 147.182.132.102
Found Yes
Hash fd6ddaf45a2f649106084d5d07755c7cf903e8905e4342a5434a53346bb78cf5
SimHash 6769d3578e00

Groups

googlebot

Rule Path
Disallow /nogooglebot/

*

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /tag/
Disallow /author/
Disallow /category/
Disallow /attachments/
Disallow /archives/
Disallow /*?*

pinterest

Rule Path
Disallow

pinterestbot

Rule Path
Disallow

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

neevabot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

adsbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

petalbot

Rule Path
Disallow /
Disallow /?utm_source=

ccbot

Rule Path
Disallow /

ccbot/2.0

Rule Path
Disallow /

ccbot/2.0 (http://commoncrawl.org/faq/)

Rule Path
Disallow /

wikido

Rule Path
Disallow /

fr_crawler

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-favo

Rule Path
Disallow /

baiduspider-news

Rule Path
Disallow /

baiduspider-cpro

Rule Path
Disallow /

baiduspider-ads

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

bitvorebot

Rule Path
Disallow /

blp_bbot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

kraken

Rule Path
Disallow /

moatbot

Rule Path
Disallow /

bhcbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

synthesio

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

brandonbot

Rule Path
Disallow /

germcrawler

Rule Path
Disallow /

sogou

Rule Path
Disallow /

exabot

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

admantx

Rule Path
Disallow /

rogerbot
exabot
mj12bot
dotbot
gigabot
ahrefsbot
blackwidow
chinaclaw
custo
disco
download\ demon
ecatch
eirgrabber
emailsiphon
emailwolf
express\ webpictures
extractorpro
eyenetie
flashget
getright
getweb!
go!zilla
go-ahead-got-it
grabnet
grafula
hmview
httrack
image\ stripper
image\ sucker
indy\ library
interget
internet\ ninja
jetcar
joc\ web\ spider
larbin
leechftp
mass\ downloader
midown\ tool
mister\ pix
navroad
nearsite
netants
netspider
net\ vampire
netzip
octopus
offline\ explorer
offline\ navigator
pagegrabber
papa\ foto
pavuk
pcbrowser
realdownload
reget
sitesnagger
smartdownload
superbot
superhttp
surfbot
takeout
teleport\ pro
voideye
web\ image\ collector
web\ sucker
webauto
webcopier
webfetch
webgo\ is
webleacher
webreaper
websauger
website\ extractor
website\ quester
webstripper
webwhacker
webzip
wget
widow
wwwoffle
xaldon\ webspider
zeus
anthropic-ai
awariorssbot
awariosmartbot
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
dataforseobot
diffbot
facebookbot
gptbot
google-extended
magpie-crawler
newsnow
news-please
omgili
omgilibot
perplexitybot
scrapy
turnitinbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /wp-content/uploads/wpo/wpo-plugins-tables-list.json
Back to top

Other Records

Field Value
sitemap https://smartpaperhelps.com/sitemap_index.xml
Back to top

Comments

  • New crawlers to block 2016
Back to top

Warnings

  • `host` is not a known field.
Back to top