Log In Sign Up

hostgator.pe
robots.txt

Robots Exclusion Standard data for hostgator.pe

Resource Scan

Scan Details

Site Domain hostgator.pe
Base Domain hostgator.pe
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-09-07T03:00:19+00:00
Next Scan 2025-10-07T03:00:19+00:00

Last Successful Scan

Scanned2025-07-17T01:46:42+00:00
URL https://hostgator.pe/robots.txt
Redirect https://www.hostgator.pe/robots.txt
Redirect Domain www.hostgator.pe
Redirect Base hostgator.pe
Domain IPs 104.21.45.83, 172.67.212.67, 2606:4700:3031::ac43:d443, 2606:4700:3037::6815:2d53
Redirect IPs 104.21.45.83, 172.67.212.67, 2606:4700:3031::ac43:d443, 2606:4700:3037::6815:2d53
Response IP 104.21.45.83
Found Yes
Hash 2e399d7b4296227f4ba136fe3ab05b208ec8eaac1ee8faab4faf5a12ffaca27e
SimHash 49549946c711

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

siteauditbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

gptbot

Rule Path
Allow /

applebot

Rule Path
Allow /

bytespider

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

ccbot

Rule Path
Allow /

amazonbot

Rule Path
Allow /

claudebot

Rule Path
Allow /

archive.org_bot

Rule Path
Allow /

nicecrawler

Rule Path
Allow /

*

Rule Path
Disallow /sac-registro

*

Rule Path
Disallow /*?itm*

*

Rule Path
Disallow /*?*

*

Rule Path
Disallow /busca?*
Back to top

Other Records

Field Value
sitemap https://www.hostgator.pe/sitemap.xml
Back to top

Comments

  • *
  • Googlebot
  • Googlebot-Image
  • Mediapartners-Google
  • Adsbot-Google
  • SiteAuditBot
  • ChatGPT-User
  • GPTBot
  • Applebot
  • Bytespider
  • PerplexityBot
  • OAI-SearchBot
  • CCBot
  • Amazonbot
  • ClaudeBot
  • archive.org_bot
  • Nicecrawler
  • *
  • *
  • *
  • *
  • Host
  • Sitemaps
Back to top

Warnings

  • `host` is not a known field.
Back to top