mawsoninc.com
robots.txt
Robots Exclusion Standard data for mawsoninc.com
Resource Scan
Scan Details
| Site Domain | mawsoninc.com |
| Base Domain | mawsoninc.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2025-12-23T01:57:10+00:00 |
| Next Scan | 2026-02-21T01:57:10+00:00 |
Last Successful Scan
| Scanned | 2025-09-30T13:47:20+00:00 |
| URL | https://mawsoninc.com/robots.txt |
| Domain IPs | 104.21.15.110, 172.67.162.88, 2606:4700:3034::ac43:a258, 2606:4700:3036::6815:f6e |
| Response IP | 172.67.162.88 |
| Found | Yes |
| Hash | a753e729ed59798bcad2c1c0165cfbbbf8c043ea619e7160f763d0dcf99a4b28 |
| SimHash | 6da1fd937f2c |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /wp-admin/ |
| Disallow | /author/ |
| Disallow | /category/ |
| Disallow | /2021/ |
| Disallow | /2020/ |
| Disallow | /2019/ |
| Disallow | /2018/ |
| Disallow | /2017/ |
| Disallow | /cs-category/rules |
| Disallow | /e-landing-page-sitemap.xml |
| Disallow | /ot_portfolio-sitemap.xml |
| Disallow | /cs-category-sitemap.xml |
| Allow | /wp-admin/admin-ajax.php |
| Allow | /post-sitemap.xml |
| Allow | /page-sitemap.xml |
| Allow | /category-sitemap.xml |
| Allow | /post_tag-sitemap.xml |
| Allow | /clickfunnels-sitemap.xml |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 90 |