Log In Sign Up

searchbysite.com
robots.txt

Robots Exclusion Standard data for searchbysite.com

Resource Scan

Scan Details

Site Domain searchbysite.com
Base Domain searchbysite.com
Scan Status Ok
Last Scan2025-12-09T06:41:28+00:00
Next Scan 2025-12-16T06:41:28+00:00

Last Scan

Scanned2025-12-09T06:41:28+00:00
URL https://searchbysite.com/robots.txt
Domain IPs 104.21.44.151, 172.67.200.230, 2606:4700:3034::ac43:c8e6, 2606:4700:3036::6815:2c97
Response IP 104.21.44.151
Found Yes
Hash c31a7789478ec11bc208fba38e816dd1831ae6b140f14133a571cb4cf77772db
SimHash c4001253e707

Groups

*

Rule Path
Allow /

bingbot

Rule Path
Allow /
Disallow /search?*
Disallow /x?*
Disallow /reddit?*
Disallow /instagram?*
Disallow /snapchat?*
Disallow /all?*

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://searchbysite.com/sitemap.xml
Back to top

Comments

  • robots.txt for SearchBySite
  • Specific directives for BingBot
  • Disallow search results pages with query parameters
  • Sitemap location
Back to top