Log In Sign Up

netizenme.com
robots.txt

Robots Exclusion Standard data for netizenme.com

Resource Scan

Scan Details

Site Domain netizenme.com
Base Domain netizenme.com
Scan Status Ok
Last Scan2025-05-24T20:48:36+00:00
Next Scan 2025-05-31T20:48:36+00:00

Last Scan

Scanned2025-05-24T20:48:36+00:00
URL https://netizenme.com/robots.txt
Domain IPs 104.21.73.160, 172.67.163.218, 2606:4700:3034::6815:49a0, 2606:4700:3035::ac43:a3da
Response IP 172.67.163.218
Found Yes
Hash 3779b4823578cbe76314e86fbe2f5c2f68890119ee024359e29de36ed05ca8bf
SimHash 6c18588167cd

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /cgi-bin/
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /?s=
Disallow /comments/
Disallow /feed/

Other Records

Field Value
crawl-delay 20

googlebot

Rule Path
Disallow /wp-admin/
Disallow /wp-login.php

Other Records

Field Value
crawl-delay 20

bingbot

Rule Path
Disallow /wp-admin/
Disallow /wp-login.php

Other Records

Field Value
crawl-delay 20

yandex

Rule Path
Disallow /

slurp

Rule Path
Disallow /

duckduckbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

sogou

Rule Path
Disallow /

slurp

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

uptimerobot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

curl

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

wget

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

httrack

Rule Path
Disallow /

webcopier

Rule Path
Disallow /
Back to top

Comments

  • =====================
  • General Bot Controls
  • =====================
  • Optional: Block comment and feed URLs (can be uncommented if needed)
  • Disallow: /category/
  • Disallow: /tag/
  • =====================
  • Major Search Engine Bots - Allow with Crawl Delay
  • =====================
  • =====================
  • Sitemap (Replace Per Site or Remove)
  • =====================
  • Sitemap: https://yourdomain.com/sitemap.xml
Back to top