bigoven.com
robots.txt

Robots Exclusion Standard data for bigoven.com

Resource Scan

Scan Details

Site Domain bigoven.com
Base Domain bigoven.com
Scan Status Ok
Last Scan2024-06-08T02:45:01+00:00
Next Scan 2024-06-15T02:45:01+00:00

Last Scan

Scanned2024-06-08T02:45:01+00:00
URL https://bigoven.com/robots.txt
Redirect https://www.bigoven.com/robots.txt
Redirect Domain www.bigoven.com
Redirect Base bigoven.com
Domain IPs 20.184.245.19
Redirect IPs 20.184.245.19
Response IP 20.184.245.19
Found Yes
Hash 6ece01f748647ee01897b9e450ffe0e1ae794a4ee32b449594193388881eb2c3
SimHash 7449a3148792

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /media/editimgecaption/*
Disallow /community/compose/*
Disallow /recipe/link/*
Disallow /recipe/zapcache/*
Disallow /recipe/*/bitflag
Disallow /recipe/*/feedback
Disallow /recipe/*/*/nutrition
Disallow /recipe/*/*/print
Disallow /admin/*
Disallow /proxy/*
Disallow /utils/*
Disallow /media/*
Disallow /medal/*
Disallow /recipe/email*
Disallow /*verify$
Disallow /*aspx$
Disallow /*customize$

amazonbot
awariorssbot
anthropic-ai
bytespider
chatgpt-user
claude-web
claudebot
ccbot
diffbot
dataforseobot
facebookbot
google-extended
gptbot
google-extended
magpie-crawler
omgilibot
piplbot
peer39_crawler
peer39_crawler/1.0
perplexitybot
scrapy
turnitinbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.bigoven.com/sitemaps/sitemap.xml