Log In Sign Up

newtonim.com
robots.txt

Robots Exclusion Standard data for newtonim.com

Resource Scan

Scan Details

Site Domain newtonim.com
Base Domain newtonim.com
Scan Status Ok
Last Scan2024-09-09T15:47:53+00:00
Next Scan 2024-10-09T15:47:53+00:00

Last Scan

Scanned2024-09-09T15:47:53+00:00
URL https://newtonim.com/robots.txt
Redirect https://www.newtonim.com/robots.txt
Redirect Domain www.newtonim.com
Redirect Base newtonim.com
Domain IPs 160.254.113.6, 167.222.113.6
Redirect IPs 108.157.254.125, 108.157.254.19, 108.157.254.41, 108.157.254.76
Response IP 108.157.254.41
Found Yes
Hash 00ff1b9285df7254a9a532e0a60d54f367c7d9f212ee3ef9d936f2ba343fa75f
SimHash 007c1ed2e46a

Groups

*

Rule Path
Disallow

*

Rule Path
Disallow /allowable-websites/
Disallow /*__trashed
Disallow /*/post_tag/
Disallow /%jurisdiction%/

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

blp_bbot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

img2dataset

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.newtonim.com/sitemap_index.xml
Back to top

Comments

  • Robots exclusion file for newtonim.com. Updated 2023-06-22
  • START YOAST BLOCK
  • --------------------------
  • ------------------------------------
  • END YOAST BLOCK
  • Directives to present crawling of undesirable URLs
  • Undesirable crawlers that obey robots exclusion rules
  • Ahrefs.com - SEO tool
  • Majestic.com - SEO tool
  • SEMRush.com - SEO tool
  • Moz.com - SEO tool
  • Open AI
  • Misc bots
Back to top