bastavid.com
robots.txt

Robots Exclusion Standard data for bastavid.com

Resource Scan

Scan Details

Site Domain bastavid.com
Base Domain bastavid.com
Scan Status Ok
Last Scan2025-10-18T15:03:25+00:00
Next Scan 2025-11-17T15:03:25+00:00

Last Scan

Scanned2025-10-18T15:03:25+00:00
URL https://bastavid.com/robots.txt
Domain IPs 104.21.65.177, 172.67.147.225, 2606:4700:3030::6815:41b1, 2606:4700:3033::ac43:93e1
Response IP 172.67.147.225
Found Yes
Hash 3ee6ff125af2f8e115fb5b8e9cb1c4f1cbf2e14c4e7b7e9a338951680eae878d
SimHash 635c9820a780

Groups

*

Rule Path
Disallow /cgi-bin
Disallow /wp-admin

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

gptbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

dataforseobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

claudebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

amazonbot

Rule Path
Disallow /