Log In Sign Up

pgb-europe.com
robots.txt

Robots Exclusion Standard data for pgb-europe.com

Resource Scan

Scan Details

Site Domain pgb-europe.com
Base Domain pgb-europe.com
Scan Status Ok
Last Scan2025-12-09T19:32:44+00:00
Next Scan 2026-01-08T19:32:44+00:00

Last Scan

Scanned2025-12-09T19:32:44+00:00
URL https://pgb-europe.com/robots.txt
Redirect https://www.pgb-europe.com/robots.txt
Redirect Domain www.pgb-europe.com
Redirect Base pgb-europe.com
Domain IPs 104.26.2.115, 104.26.3.115, 172.67.71.199, 2606:4700:20::681a:273, 2606:4700:20::681a:373, 2606:4700:20::ac43:47c7
Redirect IPs 104.26.2.115, 104.26.3.115, 172.67.71.199, 2606:4700:20::681a:273, 2606:4700:20::681a:373, 2606:4700:20::ac43:47c7
Response IP 172.67.71.199
Found Yes
Hash 1bc7bbf5c15010dd4d1f9586fef7f0fa24c0e5386db5990266af7554fb7f63f0
SimHash 6670cbdc8fe8

Groups

*

Rule Path
Disallow /account
Disallow /backorders
Disallow /checkout
Disallow /download
Disallow /error
Disallow /impersonation
Disallow /orders
Disallow /template
Disallow /terminal
Disallow /upload

dotbot
ia_archiver
rogerbot
sapphirewebcrawler
amazonbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://pgb-europe.com/sitemap.xml
Back to top

Comments

  • robots.txt from wwwroot
  • Set request rate, crawl delay, and visit time for all user agents
  • For all robots - Blocks access to specific groups of pages for all user agents
  • Blocks specific user agents from accessing the entire website
Back to top

Warnings

  • 1 invalid line.
  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top