bollywoodhungama.com
robots.txt

Robots Exclusion Standard data for bollywoodhungama.com

Resource Scan

Scan Details

Site Domain bollywoodhungama.com
Base Domain bollywoodhungama.com
Scan Status Ok
Last Scan2025-03-09T07:08:34+00:00
Next Scan 2025-03-16T07:08:34+00:00

Last Scan

Scanned2025-03-09T07:08:34+00:00
URL https://www.bollywoodhungama.com/robots.txt
Domain IPs 23.46.230.143, 23.46.230.153, 2600:1413:b000:1e::17d1:2e4b, 2600:1413:b000:1e::17d1:2e57
Response IP 23.54.118.74
Found Yes
Hash 3d1e972a383850fec5c70bc57190888844e6062e72e898de04be1d3529272aae
SimHash 6e6c985aa43b

Groups

*
ahrefsbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

yandex

Rule Path
Allow /

googlebot

Rule Path
Allow /
Disallow /private/

petalbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

majestic

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

alphabot

Rule Path
Disallow /

nutch

Rule Path
Disallow /
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/
Disallow /wp-content/mu-plugins/
Disallow /readme.html
Disallow /license.txt
Disallow /wp-config.php
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /xmlrpc.php
Disallow /?p=*
Disallow /*?*
Disallow /*?s=*
Disallow /*?search=*
Allow /wp-content/uploads/
Allow /wp-admin/admin-ajax.php
Allow /wp-includes/*.css
Allow /wp-includes/*.js
Allow /wp-content/themes/*.css
Allow /wp-content/themes/*.js
Allow /wp-content/plugins/*.css
Allow /wp-content/plugins/*.js

Other Records

Field Value
sitemap https://www.bollywoodhungama.com/sitemap_index.xml
sitemap https://www.bollywoodhungama.com/critic_sitemap/critic_review_sitemap.xml
sitemap https://www.bollywoodhungama.com/post-sitemap2.xml
sitemap https://www.bollywoodhungama.com/post-sitemap101.xml

Comments

  • Global rules for all user agents
  • Block specific malicious or unwanted bots
  • Block access to sensitive WordPress directories
  • Block access to sensitive files
  • Block query parameters that may be associated with XSS attacks or content that shouldn't be indexed
  • Allow indexing of media uploads (e.g., images)
  • Allow crawling of CSS and JavaScript files for proper page rendering