Log In Sign Up

irshadpc.com
robots.txt

Robots Exclusion Standard data for irshadpc.com

Resource Scan

Scan Details

Site Domain irshadpc.com
Base Domain irshadpc.com
Scan Status Ok
Last Scan2025-12-18T03:55:29+00:00
Next Scan 2025-12-25T03:55:29+00:00

Last Scan

Scanned2025-12-18T03:55:29+00:00
URL https://irshadpc.com/robots.txt
Redirect https://www.irshadpc.com/robots.txt
Redirect Domain www.irshadpc.com
Redirect Base irshadpc.com
Domain IPs 31.43.160.6, 31.43.161.6
Redirect IPs 35.71.142.77, 52.223.52.2
Response IP 52.223.52.2
Found Yes
Hash 15248f48c1494bcb4312e4eae2d9e999de63b0b0a504c8d770136fcdba893916
SimHash 72965350c076

Groups

gptbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

chatgpt-editor

Rule Path
Allow /

google-extended

Rule Path
Allow /

googleother

Rule Path
Allow /

googleother-image

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

perplexitybot/1.0

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

facebookbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

meta-externalagent

Rule Path
Allow /

ccbot

Rule Path
Allow /

amazonbot

Rule Path
Allow /

applebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

bingpreview

Rule Path
Allow /

adsbot-bing

Rule Path
Allow /

msnbot

Rule Path
Allow /

huggingface

Rule Path
Allow /

youbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

bravebot

Rule Path
Allow /

neevabot

Rule Path
Allow /

yandexbot

Rule Path
Allow /

bytespider

Rule Path
Allow /

baiduspider

Rule Path
Allow /

sogou spider

Rule Path
Allow /

ia_archiver

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

pinterestbot

Rule Path
Allow /

*

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.irshadpc.com/sitemap.xml
Back to top

Comments

  • OpenAI
  • OpenAI (ChatGPT Plugins & new crawlers)
  • Google AI training and Gemini
  • Perplexity AI
  • Anthropic Claude
  • Meta AI (Facebook / Instagram AI training)
  • Common Crawl (massive AI dataset source)
  • Amazon (AWS AI / Alexa)
  • AppleBot (Siri / Apple AI)
  • Bing / Microsoft (Copilot indexing)
  • HuggingFace (HF Dataset crawler)
  • You.com AI search
  • DuckDuckGo
  • Brave Search AI
  • NeevaAI (acquired but crawler still seen)
  • Yandex AI search
  • ByteDance (TikTok / CapCut AI training crawlers)
  • Baidu AI search
  • Sogou AI
  • Internet Archive
  • LinkedIn (SEO + AI embeddings)
  • Pinterest scraping
  • General
Back to top