northcarolina.global-free-classified-ads.com
robots.txt
Robots Exclusion Standard data for northcarolina.global-free-classified-ads.com
Resource Scan
Scan Details
| Site Domain | northcarolina.global-free-classified-ads.com |
| Base Domain | global-free-classified-ads.com |
| Scan Status | Ok |
| Last Scan | 2024-05-25T20:11:41+00:00 |
| Next Scan | 2024-06-24T20:11:41+00:00 |
Last Scan
| Scanned | 2024-05-25T20:11:41+00:00 |
| URL | https://northcarolina.global-free-classified-ads.com/robots.txt |
| Domain IPs | 104.26.10.146, 104.26.11.146, 172.67.69.25, 2606:4700:20::681a:a92, 2606:4700:20::681a:b92, 2606:4700:20::ac43:4519 |
| Response IP | 104.26.11.146 |
| Found | Yes |
| Hash | db38cddb26937fe5861edba63287b706e2c9f227a9d251678b4985497052198a |
| SimHash | 756e97e8c600 |
Groups
emailcollector
emailsiphon
emailwolf
python-urllib
webzip
website quester
webster pro
wget
tocrawl/urldispatcher
amazonbot
semrushbot
semrushbot-sa
megaindex.ru
gptbot
dotbot
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Allow | /*?*a=19 |
| Allow | /*?*a=11 |
| Disallow | /show_help.php |
| Disallow | /*?*a=tag |
| Disallow | /*?a=ap |
| Disallow | /*?*a=13 |
| Disallow | /*?*a=18 |
| Disallow | /*?*a=20 |
| Disallow | /*?*a=4 |
| Disallow | /*?*a=1 |
| Disallow | /index.php?a=cart |
| Disallow | /*?*addon=sharing |
| Disallow | /other/ |
| Disallow | /cdn-cgi/ |
| Disallow | /help/print_sec_img.php |
| Disallow | /AJAX.php |
| Disallow | /votes/ |
| Disallow | /*?*hash= |
| Disallow | /get_image.php |
| Disallow | /seller- |
| Disallow | /.well-known/assetlinks.json |
| Disallow | /.well-known/apple-app-site-association |
| Disallow | /rss*.xml |
| Disallow | /register.php?b= |
| Disallow | /ads.txt |
| Disallow | /site_off.htm |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |