Log In Sign Up

bentasker.co.uk
robots.txt

Robots Exclusion Standard data for bentasker.co.uk

Resource Scan

Scan Details

Site Domain bentasker.co.uk
Base Domain bentasker.co.uk
Scan Status Ok
Last Scan2025-06-25T06:50:50+00:00
Next Scan 2025-07-25T06:50:50+00:00

Last Scan

Scanned2025-06-25T06:50:50+00:00
URL https://bentasker.co.uk/robots.txt
Redirect https://www.bentasker.co.uk/robots.txt
Redirect Domain www.bentasker.co.uk
Redirect Base bentasker.co.uk
Domain IPs 2001:41d0:2:a192::2, 94.237.56.152
Redirect IPs 138.199.46.65, 2400:52e0:1500::1274:1
Response IP 138.199.46.68
Found Yes
Hash 960352492537b662cee3bc4b2bb4ed2f0a4df40f5de2243dd2a3a02c3365b53a
SimHash 6e06ad10a4f3

Groups

amazonbot
applebot-extended
anthropic-ai
bytespider
google-extended
gptbot
ccbot
perplexitybot
chatgpt-user
imagesiftbot
img2dataset
claudebot

Rule Path
Disallow /

*

Rule Path
Disallow /paid/

wellknownbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.bentasker.co.uk/sitemap.xml
Back to top

Comments

  • Excluded specifically because of it's self serving interpretation
  • of robots.txt
  • "Because it is not a crawler, WellKnownBot does not follow generic User-Agent: * crawling rules in robots.txt files."
  • https://well-known.dev/about/
Back to top