wams.de
robots.txt

Robots Exclusion Standard data for wams.de

Resource Scan

Scan Details

Site Domain wams.de
Base Domain wams.de
Scan Status Ok
Last Scan2024-09-22T07:41:27+00:00
Next Scan 2024-09-29T07:41:27+00:00

Last Scan

Scanned2024-09-22T07:41:27+00:00
URL https://wams.de/robots.txt
Redirect https://www.welt.de/robots.txt
Redirect Domain www.welt.de
Redirect Base welt.de
Domain IPs 13.248.163.210, 76.223.57.215
Redirect IPs 23.32.29.90, 23.32.29.91, 2600:1413:b000:1b::17d7:710, 2600:1413:b000:1b::17d7:71a
Response IP 23.52.40.66
Found Yes
Hash ddd783fe7bf2ccc9379f338f9d817ce73524b28f99272e0b7c6a12952da7b834
SimHash 1424f304fc43

Groups

facebot

Rule Path
Allow /

sogou spider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

*

Rule Path
Disallow /channels-extern/
Disallow /sportdaten/
Disallow /testgpr/
Disallow /boerse/data/
Disallow /partner/
Disallow /reisetipps/
Disallow /z/
Disallow /appl/
Disallow /woa/
Disallow /am-sonntag/vorproduktion/
Disallow /audiofiles/
Disallow /out-of-home/
Disallow /immobilien/expose
Disallow /suche
Disallow /onward/
Disallow /api/
Disallow /*?config
Disallow /*?config=newsmli_bloomberg2
Disallow /*.xmli
Disallow /*?service=ajax
Disallow /*?service=Ajax
Disallow /*?ajax
Disallow /*?ajax&wid
Disallow /*?config=print
Disallow /*?config=articleidfromurl
Disallow /*?config=endscreen
Disallow /*?config=iframewelt
Disallow /*?config=langeslestueck
Disallow /*?config=latest_videos
Disallow /*?config=menu_home
Disallow /*?config=mostviewed_videos
Disallow /*?config=recommended_videos
Disallow /*?config=regioarticlemarginal
Disallow /*?config=zoom
Disallow /*?config=zoomopener
Disallow /*?noredirect=true&config=standalone
Disallow /*?config=standalone
Disallow /*?wtmc=XING
Disallow /*?config=articleidfromurl
Disallow /*?print=yes
Disallow /*?tabPane
Disallow /video/embeded/
Disallow /img/*-wWIDTH*.jpg

Other Records

Field Value
sitemap https://www.welt.de/sitemaps/newssitemap/newssitemap.xml
sitemap https://www.welt.de/sitemaps/sitemap/sitemap.xml
sitemap https://www.welt.de/sitemaps/videositemap/videositemap.xml

Comments

  • The Facebook Crawler