Log In Sign Up

steema.com
robots.txt

Robots Exclusion Standard data for steema.com

Resource Scan

Scan Details

Site Domain steema.com
Base Domain steema.com
Scan Status Ok
Last Scan2024-10-28T10:36:34+00:00
Next Scan 2024-11-27T10:36:34+00:00

Last Scan

Scanned2024-10-28T10:36:34+00:00
URL https://steema.com/robots.txt
Domain IPs 216.92.101.67
Response IP 216.92.101.67
Found Yes
Hash 1b0ffd93085b028eec6013c7b8fabb90037ed540ebddd4e44809449f3a095db3
SimHash 15185dc0c645

Groups

*

Rule Path
Disallow /cgi-bin

amazonbot

Product Comment
amazonbot Amazon's user agent
Rule Path
Disallow /

seokicks

Rule Path
Disallow /

mj12bot
ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

adsbot/3.1

Rule Path
Disallow /

adsbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

googlebot/2.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20
Back to top

Comments

  • Created with RoboGen SE Trial
  • http://www.rietta.com/robogen/
  • --------------------------------
  • Steema Software -- www.steema.com
  • Robot Exclusion File -- robots.txt
  • Author: David Berneda
  • Last Updated: 24/Jan/2024
  • agrego SEOkicks por alto consumo de peticiones por indexación.
Back to top