Log In Sign Up

mobilityengineeringtech.com
robots.txt

Robots Exclusion Standard data for mobilityengineeringtech.com

Resource Scan

Scan Details

Site Domain mobilityengineeringtech.com
Base Domain mobilityengineeringtech.com
Scan Status Ok
Last Scan2025-11-25T20:03:58+00:00
Next Scan 2025-12-25T20:03:58+00:00

Last Scan

Scanned2025-11-25T20:03:58+00:00
URL https://mobilityengineeringtech.com/robots.txt
Redirect https://www.mobilityengineeringtech.com/robots.txt
Redirect Domain www.mobilityengineeringtech.com
Redirect Base mobilityengineeringtech.com
Domain IPs 52.5.61.5
Redirect IPs 52.5.61.5
Response IP 52.5.61.5
Found Yes
Hash 5c56df5b62339f1310c7e37195f9449fcf8ff59dc3b42d5b2e808a024d2426bf
SimHash 601e9c58c858

Groups

*

Rule Path
Disallow /administrator/
Disallow /api/
Disallow /bin/
Disallow /cache/
Disallow /cli/
Disallow /components/
Disallow /includes/
Disallow /installation/
Disallow /language/
Disallow /layouts/
Disallow /libraries/
Disallow /logs/
Disallow /modules/
Disallow /plugins/
Disallow /tmp/
Disallow /tb/search
Disallow /mdb/search
Disallow /search

turnitinbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

chrome-lighthouse

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot-sa

Rule Path
Disallow /

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

barkrowler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 8

serpstatbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

megaindex.ru/2.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

magpie-crawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

dataforseobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

barkrowler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

neevabot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

seekportbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

friendlycrawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

imagesiftbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bytespider

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

coccocbot-web

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

shareaholic

Rule Path
Allow

awariobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

awariorssbot
awariosmartbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

perplexitybot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

perplexity‑user

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • If the Joomla site is installed within a folder
  • eg www.example.com/joomla/ then the robots.txt file
  • MUST be moved to the site root
  • eg www.example.com/robots.txt
  • AND the joomla folder name MUST be prefixed to all of the
  • paths.
  • eg the Disallow rule for the /administrator/ folder MUST
  • be changed to read
  • Disallow: /joomla/administrator/
  • For more information about the robots.txt standard, see:
  • https://www.robotstxt.org/orig.html
  • try to keep bots away from search
  • the below needs to be added to svn
  • https://moz.com/help/moz-procedures/crawlers/rogerbot
  • Disallow: /
  • https://babbar.tech/crawler
  • https://serpstatbot.com/
  • https://megaindex.com/crawler
  • https://www.brandwatch.com/legal/magpie-crawler/
  • https://www.linkdex.com/en-us/about/bots/
  • User-agent: linkdexbot
  • Crawl-Delay: 10
  • https://dataforseo.com/dataforseo-bot
  • https://www.babbar.tech/crawler
  • see https://www.abuseipdb.com/check/154.54.249.204
  • https://neeva.com/neevabot
  • 100.26.127.17
  • https://bot.seekport.com
  • http://yandex.com/bots
  • https://imagesift.com/about
  • https://platform.openai.com/docs/gptbot
  • https://developers.facebook.com/docs/sharing/webmasters/web-crawlers
  • https://help.coccoc.com/en/search-engine
  • https://commoncrawl.org/faq
  • http://www.shareaholic.com/steve (added by OPR -- 2024-07-10)
  • based on https://support.shareaholic.com/hc/en-us/articles/20468744584980-How-do-I-whitelist-IP-addresses-and-user-agents
  • https://awario.com/bots.html
  • https://moz.com/help/moz-procedures/crawlers/dotbot#slow-dotbot-down
  • Perplexity Crawlers
  • https://docs.perplexity.ai/guides/bots
  • https://www.perplexity.com/perplexity-user.json
  • https://help.seranking.com/en/blex-crawler
  • https://help.seranking.com/hc/en-us/articles/17126130916636-BLEXBot-Crawler
Back to top