americantownspolitics.com
robots.txt

Robots Exclusion Standard data for americantownspolitics.com

Resource Scan

Scan Details

Site Domain americantownspolitics.com
Base Domain americantownspolitics.com
Scan Status Ok
Last Scan2024-11-12T11:30:21+00:00
Next Scan 2024-11-19T11:30:21+00:00

Last Scan

Scanned2024-11-12T11:30:21+00:00
URL https://americantownspolitics.com/robots.txt
Domain IPs 104.21.31.38, 172.67.174.239, 2606:4700:3032::ac43:aeef, 2606:4700:3033::6815:1f26
Response IP 104.21.31.38
Found Yes
Hash 1e33f4371dd73918e320cdc5f45d692ea6015565792ea5a6182aee40171459b1
SimHash 6b07ca83df94

Groups

*

Rule Path
Disallow /ajax/
Disallow /internal/
Disallow /forms/

googlebot-image

Rule Path
Disallow /

msnbot-media

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

garlikcrawler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /