Log In Sign Up

offerzen.com
robots.txt

Robots Exclusion Standard data for offerzen.com

Resource Scan

Scan Details

Site Domain offerzen.com
Base Domain offerzen.com
Scan Status Ok
Last Scan2024-10-21T21:24:38+00:00
Next Scan 2024-11-20T21:24:38+00:00

Last Scan

Scanned2024-10-21T21:24:38+00:00
URL https://offerzen.com/robots.txt
Redirect https://www.offerzen.com:443/robots.txt
Redirect Domain www.offerzen.com
Redirect Base offerzen.com
Domain IPs 13.227.254.113, 13.227.254.24, 13.227.254.63, 13.227.254.99, 2600:9000:200a:200:15:e905:7f00:93a1, 2600:9000:200a:4400:15:e905:7f00:93a1, 2600:9000:200a:7600:15:e905:7f00:93a1, 2600:9000:200a:a000:15:e905:7f00:93a1, 2600:9000:200a:b800:15:e905:7f00:93a1, 2600:9000:200a:e000:15:e905:7f00:93a1, 2600:9000:200a:e400:15:e905:7f00:93a1, 2600:9000:200a:ee00:15:e905:7f00:93a1
Redirect IPs 13.227.254.113, 13.227.254.24, 13.227.254.63, 13.227.254.99, 2600:9000:200a:1600:15:e905:7f00:93a1, 2600:9000:200a:4000:15:e905:7f00:93a1, 2600:9000:200a:4600:15:e905:7f00:93a1, 2600:9000:200a:5800:15:e905:7f00:93a1, 2600:9000:200a:8400:15:e905:7f00:93a1, 2600:9000:200a:8c00:15:e905:7f00:93a1, 2600:9000:200a:9200:15:e905:7f00:93a1, 2600:9000:200a:ec00:15:e905:7f00:93a1
Response IP 13.227.254.113
Found Yes
Hash 61be81f799db31ef1e3ccad6b5944e508e347b3f4db15b341414589736e76012
SimHash 1885cd1fea40

Groups

*

Rule Path
Disallow /blog/www.educonnect.co.za$
Disallow /blog/www.linkedin.com/in/duke-coulbanis$
Disallow /blog/www.pandora.com$
Disallow /blog/www.nicharalambous.com$
Disallow /blog/www.codewars.com$
Disallow /blog/www.spin.com$
Disallow /blog/www.invisionapp.com$
Disallow /blog/www.devobsession.com$
Disallow /blog/www.linkedin.com/in/*$
Disallow /blog/www.4dicapital.com$
Disallow /blog/how-i-get-the-most-out-of-hackathons/trackback$
Disallow /blog/github.com/HypothesisWorks/hypothesis/tree/master/hypothesis-python$
Disallow /blog/automating-my-development...$
Disallow /blog/dries%40deeplearning-cafe.com$
Disallow /blog/ben%40offerzen.com$
Disallow /community/svelte-origins-documentary-premiere/registered
Disallow /marketing/tools/2020_OfferZen_Remote_Work_Poll_Report_SA_Newsletter.pdf$
Disallow /marketing/tools/private/*$

brightedge crawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

python-urllib/2.7

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

python-urllib/2.6

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://www.offerzen.com/sitemap.xml
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • User-agent: *
  • Disallow: /
  • Sitemap URL
  • Crawl delay for bot crawlers
Back to top