Log In Sign Up

femalemag.com.sg
robots.txt

Robots Exclusion Standard data for femalemag.com.sg

Resource Scan

Scan Details

Site Domain femalemag.com.sg
Base Domain femalemag.com.sg
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-28T17:16:33+00:00
Next Scan 2025-01-26T17:16:33+00:00

Last Successful Scan

Scanned2024-07-01T17:11:20+00:00
URL https://femalemag.com.sg/robots.txt
Redirect https://www.femalemag.com.sg/robots.txt
Redirect Domain www.femalemag.com.sg
Redirect Base femalemag.com.sg
Domain IPs 108.156.133.115, 108.156.133.66, 108.156.133.8, 108.156.133.90
Redirect IPs 13.225.4.28, 13.225.4.30, 13.225.4.6, 13.225.4.78
Response IP 13.225.4.6
Found Yes
Hash 52dac02193f9194a01d10d0e237fef643c6161fa55b6c7ec1cf919f97ea01cba
SimHash ba10990a8575

Groups

*

Rule Path
Disallow /*/feed/$
Disallow /advanced-galleries/*
Disallow /search/*
Disallow /wp-admin/
Disallow /wp-content/plugins/
Disallow /author/
Disallow /featured-video-section/
Disallow /featured-video-section/brand/
Disallow /featured-video-section/contenthub/
Disallow /featured-video-section/homepage/
Disallow /featured-video-section/videos/
Disallow /featured-video-section/beauty/
Disallow /featured-video-section/fashion/
Disallow /featured-video-section/life/
Disallow /featured-video-section/videos/
Disallow /featured-video-section/women/
Disallow /embedded-gallery/
Disallow /anchor_block/
Disallow /poll/

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.femalemag.com.sg/sitemap_index.xml
Back to top

Comments

  • $Id: robots.txt,v 1.9.2.2 2010/09/06 10:37:16 goba Exp $
  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources......
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/wc/robots.html
  • For syntax checking, see:
  • http://www.sxw.org.uk/computing/robots/check.html
  • For new training only
  • Not for training, only for user requests
  • Marker for disabling Bard and Vertex AI
  • Multi-purpose, commercial uses; including LLMs
Back to top