cambridgeinfotech.io
robots.txt

Robots Exclusion Standard data for cambridgeinfotech.io

Resource Scan

Scan Details

Site Domain cambridgeinfotech.io
Base Domain cambridgeinfotech.io
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2026-01-30T22:45:45+00:00
Next Scan 2026-03-01T22:45:45+00:00

Last Successful Scan

Scanned2025-12-09T22:20:53+00:00
URL https://cambridgeinfotech.io/robots.txt
Domain IPs 2a02:4780:16:dcd5:f278:bfff:d21c:d142, 2a02:4780:38:5708:d249:9ef8:c029:5b2f, 84.32.84.205, 84.32.84.236
Response IP 91.108.100.149
Found Yes
Hash d1c5591c1d0e2c6e598245026e19ab1511612053defd21bef8a8802af7f84ae3
SimHash 721c5c00c29b

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-includes/
Disallow /readme.html
Disallow /license.txt
Disallow /xmlrpc.php
Disallow /cgi-bin/
Disallow /trackback/
Disallow /wp-content/plugins/
Disallow /wp-content/cache/
Disallow /wp-content/themes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /search/
Disallow /cart/
Disallow /checkout/
Disallow /my-account/
Disallow /wishlist/
Disallow /thank-you/
Disallow /order-received/
Disallow /*?orderby=
Disallow /*?filter=
Disallow /*?add-to-cart=
Disallow /*?s=
Disallow /*?utm_source=
Disallow /*?utm_medium=
Disallow /*?utm_campaign=
Disallow /*?gclid=
Disallow /*?fbclid=
Allow /*.webp$
Allow /*.jpg$
Allow /*.png$
Allow /*.gif$
Allow /*.pdf$
Allow /*.docx$
Allow /*.html$
Allow /*.php$

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

yandex

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

pinterest

Rule Path
Allow /

whatsapp

Rule Path
Allow /

telegrambot

Rule Path
Allow /

discordbot

Rule Path
Allow /

dotbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Allow /
Allow /course-level/all-levels/

Other Records

Field Value
sitemap https://www.cambridgeinfotech.io/sitemap_index.xml

Comments

  • Allow Search Engine Crawlers
  • Allow Social Media Bots
  • Block Scraper Bots
  • Allow AhrefsBot