Log In Sign Up

mawoopets.com
robots.txt

Robots Exclusion Standard data for mawoopets.com

Resource Scan

Scan Details

Site Domain mawoopets.com
Base Domain mawoopets.com
Scan Status Ok
Last Scan2024-09-22T21:41:44+00:00
Next Scan 2024-10-22T21:41:44+00:00

Last Scan

Scanned2024-09-22T21:41:44+00:00
URL https://mawoopets.com/robots.txt
Redirect https://www.mawoopets.com/robots.txt
Redirect Domain www.mawoopets.com
Redirect Base mawoopets.com
Domain IPs 13.33.88.29, 13.33.88.73, 13.33.88.76, 13.33.88.89
Redirect IPs 76.76.21.142, 76.76.21.98
Response IP 76.76.21.241
Found Yes
Hash 29f8711a90abb876514be71efc8ddb34c4a3cc6493a2745f8a6274239841b301
SimHash 82108d3002d2

Groups

*

Rule Path
Disallow /api
Disallow /admin
Disallow /admin/*
Disallow /customer
Disallow /customer/*
Disallow /breeder
Disallow /breeder/applications/*
Disallow /breeder/chat/*
Disallow /breeder/edit/*
Disallow /breeder/notifications/*
Disallow /breeder/pets/*
Disallow /breeder/verify/*
Disallow /breeder/buyers
Disallow /breeder/calendar
Disallow /breeder/dashboard
Disallow /breeder/earnings
Disallow /breeder/faq
Disallow /breeder/pending
Disallow /breeder/profile
Disallow /login/admin
Disallow /.well-known/*
Disallow /apple-app-site-association
Allow /
Allow /puppies-for-sale/*
Disallow /*.json$
Disallow /*_buildManifest.js$
Disallow /*_middlewareManifest.js$
Disallow /*_ssgManifest.js$
Disallow /*.js$
Back to top

Other Records

Field Value
sitemap https://www.mawoopets.com/pages-sitemap.xml
sitemap https://www.mawoopets.com/breeds-states-sitemap.xml
sitemap https://www.mawoopets.com/vet-sitemap.xml
Back to top

Comments

  • *
  • Next.JS Crawl Budget Performance Updates
  • Block files ending in .json, _buildManifest.js, _middlewareManifest.js, _ssgManifest.js, and any other JS files
  • The asterisks allows any file name
  • The dollar sign ensures it only matches the end of an URL and not a oddly formatted url (e.g. /locations.json.html)
  • Sitemaps
Back to top