Log In Sign Up

en.dailypakistan.com.pk
robots.txt

Robots Exclusion Standard data for en.dailypakistan.com.pk

Resource Scan

Scan Details

Site Domain en.dailypakistan.com.pk
Base Domain dailypakistan.com.pk
Scan Status Ok
Last Scan2024-11-12T10:07:52+00:00
Next Scan 2024-11-19T10:07:52+00:00

Last Scan

Scanned2024-11-12T10:07:52+00:00
URL https://en.dailypakistan.com.pk/robots.txt
Domain IPs 212.95.63.9
Response IP 212.95.63.9
Found Yes
Hash d6cfc58ca5c8ecf8204c05b8a912996a904d0f8a7d5a610f5eea719d7144bea4
SimHash bf44d921e980

Groups

*

Rule Path
Disallow /wp-json/
Disallow /?s=*
Disallow /search/*
Disallow */trackback
Disallow */comments

nuclei
wikido
riddler
petalbot
zoominfobot
node/simplecrawler
cazoodlebot
dotbot/1.0
barkrowler
blexbot
magpie-crawler
ias_crawler
semrushbot
dataforseobot
siteauditbot
semrushbot-ba
semrushbot-si
semrushbot-swa
splitsignalbot
semrushbot-ocob
gptbot
imagesiftbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://en.dailypakistan.com.pk/sitemap_index.xml
Back to top

Comments

  • Global rules
  • -----------------
  • Disallow: /*.pdf$
  • Disallow: /tag
  • Sitemap
  • -----------------
  • Ban bots that don't benefit us.
  • --------------------------------
Back to top