Log In Sign Up

femalemag.com.sg
robots.txt

Robots Exclusion Standard data for femalemag.com.sg

Resource Scan

Scan Details

Site Domain femalemag.com.sg
Base Domain femalemag.com.sg
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-18T17:10:47+00:00
Next Scan 2024-06-01T17:10:47+00:00

Last Successful Scan

Scanned2024-05-03T16:52:23+00:00
URL https://femalemag.com.sg/robots.txt
Redirect https://www.femalemag.com.sg/robots.txt
Redirect Domain www.femalemag.com.sg
Redirect Base femalemag.com.sg
Domain IPs 108.156.133.115, 108.156.133.66, 108.156.133.8, 108.156.133.90
Redirect IPs 192.229.189.122
Response IP 192.229.189.122
Found Yes
Hash 000d9b3513145517433b0b5bb83dd2b67a344397503e745957f83cc7e798648c
SimHash ba10990a0575

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-content/plugins/
Disallow /author/
Disallow /featured-video-section/
Disallow /featured-video-section/brand/
Disallow /featured-video-section/contenthub/
Disallow /featured-video-section/homepage/
Disallow /featured-video-section/videos/
Disallow /featured-video-section/beauty/
Disallow /featured-video-section/fashion/
Disallow /featured-video-section/life/
Disallow /featured-video-section/videos/
Disallow /featured-video-section/women/
Disallow /embedded-gallery/
Disallow /anchor_block/
Disallow /poll/

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.femalemag.com.sg/sitemap_index.xml
Back to top

Comments

  • $Id: robots.txt,v 1.9.2.2 2010/09/06 10:37:16 goba Exp $
  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources......
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/wc/robots.html
  • For syntax checking, see:
  • http://www.sxw.org.uk/computing/robots/check.html
  • For new training only
  • Not for training, only for user requests
  • Marker for disabling Bard and Vertex AI
  • Multi-purpose, commercial uses; including LLMs
Back to top