arab-dream.net
robots.txt
Robots Exclusion Standard data for arab-dream.net
Resource Scan
Scan Details
| Site Domain | arab-dream.net |
| Base Domain | arab-dream.net |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 3/21/2025, 3:44:35 AM |
| Next Scan | 4/20/2025, 3:44:35 AM |
Last Successful Scan
| Scanned | 1/1/2025, 3:40:51 AM |
| URL | https://arab-dream.net/robots.txt |
| Redirect | https://arab-dream.news/robots.txt |
| Redirect Domain | arab-dream.news |
| Redirect Base | arab-dream.news |
| Domain IPs | 104.21.59.31, 172.67.211.238, 2606:4700:3030::6815:3b1f, 2606:4700:3036::ac43:d3ee |
| Redirect IPs | 104.21.89.247, 172.67.166.46, 2606:4700:3031::ac43:a62e, 2606:4700:3034::6815:59f7 |
| Response IP | 104.21.89.247 |
| Found | Yes |
| Hash | 5b94805fa6b629abf48deef4008c3ef3eaa791c515a7206b0c8d29d08501cfbd |
| SimHash | 456c57d496f7 |
Groups
*
| Rule | Path |
|---|---|
| Allow | / |
| Allow | /uploads/* |
| Disallow | /admin/ |
| Disallow | /templates/ |
| Disallow | /include/ |
| Disallow | /lang/ |
| Disallow | /Smarty/ |
| Disallow | /404.php |
| Disallow | /config.php |
| Disallow | /cdn-cgi/ |
| Disallow | *sortby%3D* |
| Disallow | *order%3D* |
| Disallow | *page%3D* |
| Disallow | *%26* |
| Disallow | /*.js$ |
| Disallow | /*.inc$ |
| Disallow | /*.gz$ |
| Disallow | /*.wmv$ |
| Disallow | /*.cgi$ |
| Disallow | /*.xhtml$ |
| Disallow | /?* |
| Disallow | /*.php$ |
| Disallow | /new/?* |
ia_archiver-web.archive.org
autogpt
agent gpt
anthropic-ai
aria browser ai
aria browse aria ai
aisearchbot
applebot-extended
amazon-kendra
amazon silk
amazon sagemaker
aws trainium
amazon bedrock
alexa
alexatm
amazon textract
amazon comprehend
anypicker
alphaai
ai2 bot
ai2bot-dolma
brave leo ai
bingbot-chat/2.0
bing ai
cohere-ai
claudebot
crewai
claude 3.5 sonnet
claude 3.5 haiku
ccbot
ccbot
ccbot/2.0
cc-crawler/2.0
chatgpt
chatgpt-user
dataprovider
diffbot
dialogpt
depolarizinggpt
doubao ai
duckassistbot
dalvik/2
dalvik/2.1.0
gptbot
gptbot/0.1
gptbot/1.0
gptbot/1.2
gpt-1
gpt-2
gpt-3
gpt-3.5
gpt-4
gpt-4-turbo
gpt-4o
gpt-4v
gpt-4o mini
gpt-3.5 turbo
gpt 4 omni
gpt 4 omni mini
gpt-sw3
gptzero
zerogpt
zerochat
searchgpt
google-extended
google gemini
googleother
google-cloudvertexbot
facebookbot
facebookbot/1.0
facebookexternalhit
facebookexternalhit/1.1
meta-externalfetcher/1.1
meta-externalagent
meta-externalfetcher
meta ai
llama 3.2
friendlycrawler
friendlycrawler/1.0
friendlycrawler/nutch-1.20-snapshot
icc-crawler
intelliseek
intelliseek.ai
imagesiftbot
img2dataset
iaskspider/2.0
kangaroo bot
leftwinggpt
nicecrawler
magpie-crawler
omgilibot
omgili
openai
openbot
openai gpt
oai searchbot
oai-searchbot/1.0
openai o1
openai o1-mini
owler
searchgpt
scrapy
stability ai
scrapergpt
shadowgpt
thehive.ai
rightwinggpt
timpibot
timpibot/0.8
timpibot/0.9
webchatgpt
wormgpt v3.0
wpbot
wpbot/1.1
webzio-extended
velenpublicwebcrawler
paperlibot
paperlibot/2.1
perplexitybot
proximic
peer39_crawler/1.0
piplbot
youbot
yargpt
yarchatgpt
duggmirror
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| sitemap | https://arab-dream.news/sitemap-index.xml |