bandedessinee.info
robots.txt

Robots Exclusion Standard data for bandedessinee.info

Resource Scan

Scan Details

Site Domain bandedessinee.info
Base Domain bandedessinee.info
Scan Status Ok
Last Scan2024-09-19T13:24:23+00:00
Next Scan 2024-09-26T13:24:23+00:00

Last Scan

Scanned2024-09-19T13:24:23+00:00
URL https://bandedessinee.info/robots.txt
Redirect https://www.bandedessinee.info/robots.txt
Redirect Domain www.bandedessinee.info
Redirect Base bandedessinee.info
Domain IPs 147.135.136.100
Redirect IPs 147.135.136.100
Response IP 147.135.136.100
Found Yes
Hash 4e36fd3a4f619fef69b608db744c75922001e71b397d89633e170696efbfc50c
SimHash 620cd852deb0

Groups

googlebot
googlebot-image
mediapartners-google
msnbot
msnbot-media

Rule Path
Disallow /ecrire/

ahrefsbot
amazonbot
barkrowler
blexbot
buck
ccbot
dataforseobot
dotbot
gptbot
grapeshot
mj12bot
mojeekbot
petalbot
rogerbot
seekportbot
semrushbot
semrushbot-desktop
semrushbot-sa
sentibot
seznambot
slurp
trendictionbot

Rule Path
Disallow /

Comments

  • disallow all
  • User-agent: *
  • Disallow: /
  • Crawl-delay: 600
  • but allow only important bots
  • User-agent: Slurp
  • User-agent: Yahoo-Blogs
  • User-agent: Yahoo-MMCrawler
  • Directories
  • Disallow: /extensions/
  • Disallow: /lib/
  • Disallow: /pro/
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Disallow: /spip.php?page=recherche&recherche=*
  • User-agent: Google-Extended