Log In Sign Up

boobam.com.br
robots.txt

Robots Exclusion Standard data for boobam.com.br

Resource Scan

Scan Details

Site Domain boobam.com.br
Base Domain boobam.com.br
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-07-24T01:08:11+00:00
Next Scan 2024-10-22T01:08:11+00:00

Last Successful Scan

Scanned2023-03-09T23:29:28+00:00
URL https://boobam.com.br/robots.txt
Domain IPs 34.206.244.223, 52.1.9.22
Response IP 52.1.9.22
Found Yes
Hash cbb7ce3180d48676aceda0f6c5801454ccf7c5fd1e4970c8fd298f3bd4558063
SimHash cb147e4361e2

Groups

bubing

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

lcc

Rule Path
Disallow /

downloaderchrome

Rule Path
Disallow /

firefox/55.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

mtrobot

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

turnitin (https://bit.ly/2uvnfoq)

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ucbrowser

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

*

No rules defined. All paths allowed.

Back to top

Comments

  • Request-rate: 10/5 # maximum rate is 10 pages every 5 seconds
  • Crawl-delay: 0.1 # 100ms entre uma pagina e outra
Back to top