trendsmap.com
robots.txt

Robots Exclusion Standard data for trendsmap.com

Resource Scan

Scan Details

Site Domain trendsmap.com
Base Domain trendsmap.com
Scan Status Ok
Last Scan2024-06-26T07:48:16+00:00
Next Scan 2024-07-03T07:48:16+00:00

Last Scan

Scanned2024-06-26T07:48:16+00:00
URL https://trendsmap.com/robots.txt
Redirect https://www.trendsmap.com/robots.txt
Redirect Domain www.trendsmap.com
Redirect Base trendsmap.com
Domain IPs 136.243.46.236
Redirect IPs 136.243.46.236
Response IP 136.243.46.236
Found Yes
Hash 83fb5077cebfb1da09e2a469008f45f38e06f876d4cc9f13ccbcfc09675fe753
SimHash 1004d43a4bd7

Groups

twitterbot

Rule Path
Allow /

ahrefsbot

Rule Path
Disallow /

vegi

Rule Path
Disallow /

vegi bot

Rule Path
Disallow /

newspaper

Rule Path
Disallow /

google-http-java-client

Rule Path
Disallow /

something

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

python

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

ruby

Rule Path
Disallow /

ruby

Rule Path
Disallow /

curl

Rule Path
Disallow /

wget

Rule Path
Disallow /

wget

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

gozaikbot

Rule Path
Disallow /

java

Rule Path
Disallow /

jakarta

Rule Path
Disallow /

postano

Rule Path
Disallow /

jack

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

apache-httpclient

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

googlebot-image

Rule Path
Disallow /tdata
Disallow /ipu
Disallow /ipx

*

Rule Path
Disallow /account
Disallow /analytics
Disallow /alerts
Disallow /vis
Disallow /auth
Disallow /admin
Disallow /localx
Disallow /al
Disallow /paypal
Disallow /api
Disallow /misc
Disallow /iad
Disallow /sentiment/melbourne
Disallow /megamaker

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

gptbot

Rule Path
Disallow /