Log In Sign Up

pgdp.net
robots.txt

Robots Exclusion Standard data for pgdp.net

Resource Scan

Scan Details

Site Domain pgdp.net
Base Domain pgdp.net
Scan Status Ok
Last Scan2025-12-06T08:34:44+00:00
Next Scan 2026-01-05T08:34:44+00:00

Last Scan

Scanned2025-12-06T08:34:44+00:00
URL https://pgdp.net/robots.txt
Redirect https://www.pgdp.net//robots.txt
Redirect Domain www.pgdp.net
Redirect Base pgdp.net
Domain IPs 206.210.91.62
Redirect IPs 206.210.91.62
Response IP 206.210.91.62
Found Yes
Hash fd1d25a768eb4b84257e3de20436e850df79ae560c63e21e5e5b3002b7d7f352
SimHash 6274b170c4b7

Groups

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

lssbot

Rule Path
Disallow /

librabot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

yandex

Rule Path
Disallow /

exabot

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

vortex

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

*

Rule Path
Disallow /feed/
Allow /dp_branding/
Disallow /c/accounts/
Disallow /c/crontab/
Disallow /c/graphics/
Disallow /c/locale/
Disallow /c.new/
Disallow /c.bak/
Disallow /c/pinc/
Disallow /c/SETUP/
Disallow /c/stats/
Disallow /c/tools/
Disallow /c/users/
Disallow /dpmail/
Disallow /d/
Disallow /out/
Disallow /jpgraph/
Disallow /jpgraph-1.14
Disallow /archive
Disallow /projects/
Disallow /mailman/
Disallow /mailman3/
Disallow /noncvs/
Disallow /phpBB2/
Disallow /phpBB3/
Disallow /sawiki/
Disallow /squirrels/
Disallow /stats/
Disallow /tools/
Disallow /w/
Disallow /wikiheiro/

Other Records

Field Value
crawl-delay 30
Back to top

Comments

  • wfarrell 2022-03-25 Disallowed BLEXBot crawler as it uses too many phpBB sessions.
  • Bing also checks for msnbot - cpeel 2020-03-08
  • https://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question/
  • User-agent: msnbot
  • Disallow: /
  • SEMrushBot is taking up a lot of bandwidth - cpeel 2020-03-08
  • ---------------------------------------------
  • Disable AI scraping - cpeel 2023-08-23
  • ---------------------------------------------
  • Allow dp_branding to be crawled as our social media previews use it
Back to top

Warnings

  • 2 invalid lines.
Back to top