arkansas.global-free-classified-ads.com
robots.txt
Robots Exclusion Standard data for arkansas.global-free-classified-ads.com
Resource Scan
Scan Details
| Site Domain | arkansas.global-free-classified-ads.com |
| Base Domain | global-free-classified-ads.com |
| Scan Status | Ok |
| Last Scan | 2024-05-11T05:42:51+00:00 |
| Next Scan | 2024-06-10T05:42:51+00:00 |
Last Scan
| Scanned | 2024-05-11T05:42:51+00:00 |
| URL | https://arkansas.global-free-classified-ads.com/robots.txt |
| Domain IPs | 104.26.10.146, 104.26.11.146, 172.67.69.25, 2606:4700:20::681a:a92, 2606:4700:20::681a:b92, 2606:4700:20::ac43:4519 |
| Response IP | 104.26.11.146 |
| Found | Yes |
| Hash | 0fb95a428a4fdbb4002af6975aa913a8f57c6785da281b0ecebe4f9d5f005a85 |
| SimHash | 456e9068c704 |
Groups
emailcollector
emailsiphon
emailwolf
python-urllib
webzip
website quester
webster pro
wget
tocrawl/urldispatcher
amazonbot
semrushbot
semrushbot-sa
megaindex.ru
gptbot
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Allow | /*?*a=19 |
| Disallow | /show_help.php |
| Disallow | /*?*a=tag |
| Disallow | /*?a=ap |
| Disallow | /*?*a=13 |
| Disallow | /*?*a=18 |
| Disallow | /*?*a=20 |
| Disallow | /*?*a=4 |
| Disallow | /index.php?a=cart |
| Disallow | /*?*addon=sharing |
| Disallow | /other/ |
| Disallow | /cdn-cgi/ |
| Disallow | /help/print_sec_img.php |
| Disallow | /AJAX.php |
| Disallow | /votes/ |
| Disallow | /*?*hash= |
| Disallow | /get_image.php |
| Disallow | /seller- |
| Disallow | /.well-known/assetlinks.json |
| Disallow | /.well-known/apple-app-site-association |
| Disallow | /rss*.xml |
| Disallow | /register.php?b= |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |