Log In Sign Up

gubbicivilengineers.com
robots.txt

Robots Exclusion Standard data for gubbicivilengineers.com

Resource Scan

Scan Details

Site Domain gubbicivilengineers.com
Base Domain gubbicivilengineers.com
Scan Status Ok
Last Scan2025-06-06T20:15:20+00:00
Next Scan 2025-07-06T20:15:20+00:00

Last Scan

Scanned2025-06-06T20:15:20+00:00
URL https://gubbicivilengineers.com/robots.txt
Domain IPs 2a02:4780:38:8d73:1af4:fd10:efc2:8bf3, 93.127.196.177
Response IP 93.127.201.1
Found Yes
Hash 82edda93fd6d2733bd8026e2bd7960eb44e2af2700f55ce357389dbe20c2a81b
SimHash 7476de20623b

Groups

*

Rule Path
Disallow
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /wp-includes/
Disallow /cgi-bin/
Disallow /tmp/
Disallow /private/
Disallow /cart/
Disallow /checkout/
Disallow /login/
Disallow /register/
Disallow /search/
Disallow /tag/
Disallow /*?s=
Disallow /*?orderby=
Disallow /*?add-to-cart=
Disallow /*?replytocom
Disallow /*?utm_source=
Disallow /trackback/
Disallow /*.php$
Disallow /blogs/author/krunal
Disallow /privacy-policy
Disallow /disclaimer
Disallow /blogs/elementor-hf/footer_gubbi/
Disallow /blogs/elementor-hf/gubbi_header/
Allow /wp-admin/admin-ajax.php
Allow /wp-content/uploads/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/
Disallow /wp-content/cache/
Disallow /wp-json/
Disallow /readme.html
Disallow /license.txt
Disallow /?author=*
Disallow /*/feed/

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claude

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

adsbot-google

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

googlebot-image

Rule Path
Allow /wp-content/uploads/

googlebot-mobile

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

yandex

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://gubbicivilengineers.com/sitemap.xml
sitemap https://gubbicivilengineers.com/page-sitemap.xml
sitemap https://gubbicivilengineers.com/post-sitemap.xml
Back to top

Comments

  • ========================
  • Robots.txt for SEO & Security
  • Website: https://gubbicivilengineers.com
  • Created on: 15/03/2025
  • ========================
  • Allow All Search Engines to Crawl the Site
  • Block Unwanted Pages from Indexing (Prevents Duplicate & Low-Value Pages)
  • Block Specific Pages from Indexing
  • Allow Important Sections for Crawling
  • Sitemaps for Fast Indexing & Better Crawling
  • Prevent Crawling of Plugins, Themes & Cache
  • Blocking Sensitive & Unnecessary Files
  • Block AI Bots, Scrapers & Aggressive Crawlers
  • Google Ads & AdSense Bot Allowance (Ensures Ads are Properly Crawled)
  • Allow Image Crawlers for Image SEO
  • Allow Google Mobile Crawlers (Mobile SEO Optimization)
  • Crawl Delay for Non-Google Bots (Prevents Server Overload)
  • Custom User-Agent Rules (For Bing & Other Crawlers)
  • ========================
  • End of robots.txt file
  • ========================
Back to top