Log In Sign Up

diversey.com.pt
robots.txt

Robots Exclusion Standard data for diversey.com.pt

Resource Scan

Scan Details

Site Domain diversey.com.pt
Base Domain diversey.com.pt
Scan Status Ok
Last Scan2024-10-30T02:09:32+00:00
Next Scan 2024-11-29T02:09:32+00:00

Last Scan

Scanned2024-10-30T02:09:32+00:00
URL https://diversey.com.pt/robots.txt
Domain IPs 13.225.4.107, 13.225.4.49, 13.225.4.68, 13.225.4.91, 2600:9000:21b4:1a00:5:adff:b40:93a1, 2600:9000:21b4:6000:5:adff:b40:93a1, 2600:9000:21b4:7400:5:adff:b40:93a1, 2600:9000:21b4:a800:5:adff:b40:93a1, 2600:9000:21b4:bc00:5:adff:b40:93a1, 2600:9000:21b4:ee00:5:adff:b40:93a1, 2600:9000:21b4:f200:5:adff:b40:93a1, 2600:9000:21b4:fc00:5:adff:b40:93a1
Response IP 13.225.4.107
Found Yes
Hash 922d12d3e9dfa74041245db80b89578912f7c9fac074fb4bfc9534ee05910ffc
SimHash 38169f538760

Groups

grapeshotcrawler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bo

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

googlebot-image

Rule Path
Disallow /

slurp

Rule Path
Disallow /

*

Rule Path
Disallow /en/*
Disallow /wp-login/*
Disallow /wp_login/*
Disallow /wp-admin/*
Disallow /administrator/*
Disallow /pmy/*
Disallow /phpmyadmin/*
Disallow /xmlrpc.php
Disallow /*/*.php
Disallow /*/*/*.php
Disallow /sites/default/files/*.jpg
Disallow /sites/default/files/*.png
Disallow /core/*.css$
Disallow /core/*.css?
Disallow /core/*.js$
Disallow /core/*.js?
Disallow /core/*.gif
Disallow /core/*.jpg
Disallow /core/*.jpeg
Disallow /core/*.png
Disallow /core/*.svg
Disallow /profiles/*.css$
Disallow /profiles/*.css?
Disallow /profiles/*.js$
Disallow /profiles/*.js?
Disallow /profiles/*.gif
Allow /profiles/*.jpg
Disallow /profiles/*.jpeg
Disallow /profiles/*.png
Disallow /profiles/*.svg
Disallow /core/
Disallow /profiles/
Disallow /README.txt
Disallow /web.config
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips
Disallow /node/add/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /user/logout/
Disallow /index.php/admin/
Disallow /index.php/comment/reply/
Disallow /index.php/filter/tips
Disallow /index.php/node/add/
Disallow /index.php/search/
Disallow /index.php/user/password/
Disallow /index.php/user/register/
Disallow /index.php/user/login/
Disallow /index.php/user/logout/
Disallow /nodes/*
Disallow /node/*
Disallow /taxonomy/*
Disallow /product/*
Disallow /s3/files/*
Disallow /sites/*
Disallow /Media/*
Disallow /media/*
Disallow /blog?*
Disallow /media-center/news?*
Disallow /search?*
Disallow /success-stories?*
Disallow /video-hub?*
Disallow /product-catalogue?*
Disallow /*?
Disallow /*/node/*
Disallow /*/nodes/*
Disallow /*/taxonomy/*
Disallow /*/s3/files/*
Disallow /*/sites/*
Disallow /*/Media/*
Disallow /*/media/*
Disallow /*/blog?*
Disallow /*/media-center/news?*
Disallow /*/search?*
Disallow /*/success-stories?*
Disallow /*/video-hub?*
Disallow /*/product-catalogue?*
Disallow /*/*?
Disallow /*/product/*
Disallow /*/safety-data/*
Allow /profiles/*.jpg
Allow /blog/*
Allow /media-center/news/*
Allow /success-stories/*
Allow /video-hub/*
Allow /product-catalogue/*
Allow /*/blog/*
Allow /*/media-center/news/*
Allow /*/success-stories/*
Allow /*/video-hub/*
Allow /*/product-catalogue/*
Back to top

Comments

  • robots.txt - PT
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
  • SEO Optimizations
  • Block wp paths
  • Disallow old images path
  • CSS, JS, Images
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • No index
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Allowed URLs
Back to top

Warnings

  • `noindex` is not a known field.
Back to top