Log In Sign Up

mappinandwebb.com
robots.txt

Robots Exclusion Standard data for mappinandwebb.com

Resource Scan

Scan Details

Site Domain mappinandwebb.com
Base Domain mappinandwebb.com
Scan Status Ok
Last Scan2024-10-29T04:21:36+00:00
Next Scan 2024-11-28T04:21:36+00:00

Last Scan

Scanned2024-10-29T04:21:36+00:00
URL https://mappinandwebb.com/robots.txt
Redirect https://www.mappinandwebb.com/robots.txt
Redirect Domain www.mappinandwebb.com
Redirect Base mappinandwebb.com
Domain IPs 23.215.7.6, 23.215.7.9, 2600:1413:b000:1b::17d7:706, 2600:1413:b000:1b::17d7:709
Redirect IPs 23.215.7.6, 23.215.7.9, 2600:1413:b000:1b::17d7:706, 2600:1413:b000:1b::17d7:709
Response IP 23.215.7.6
Found Yes
Hash 5031a4080ab9c0448890169eabd44f8f25a011b0c9c32adeecde7cadc244efe0
SimHash 2c711f16ceeb

Groups

*

Rule Path
Disallow /cart
Disallow /checkout
Disallow /my-account
Disallow /login
Disallow /search
Disallow /*?=
Disallow /v12finance*

cazoodlebot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.mappinandwebb.com/sitemap.xml
Back to top

Comments

  • For all robots
  • Block access to specific groups of pages
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block dotbot as it cannot parse base urls properly
Back to top