Log In Sign Up

memegenerator.com
robots.txt

Robots Exclusion Standard data for memegenerator.com

Resource Scan

Scan Details

Site Domain memegenerator.com
Base Domain memegenerator.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-04-28T17:33:03+00:00
Next Scan 2024-06-27T17:33:03+00:00

Last Successful Scan

Scanned2021-10-05T20:09:16+00:00
URL http://memegenerator.com/robots.txt
Redirect https://knowyourmeme.com/robots.txt
Redirect Domain knowyourmeme.com
Redirect Base knowyourmeme.com
Found Yes
Hash f61edcbfb77aa5eb2e501bd7c807938359afb59a0eed16d3eb7005a66448b18d
SimHash 73166e4085b9

Groups

mediapartners-google

Rule Path
Disallow /search

mediapartners

Rule Path
Disallow /search

slurp

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

yahoo! slurp

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

yandex

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

yandexbot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

netseer

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

maxpointcrawler

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

blexbot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

bingbot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 1

msnbot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

webindex

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

scrapy

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

baiduspider

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

applebot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

steeler

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

archive.org_bot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

ia_archiver

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

sputnikbot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

nutch

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

wesee_bot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

scoutjet

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

spbot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

getintent crawler

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

ntentbot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

leikibot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

mozilla/5.0 (compatible;contxbot/1.0)

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

contxbot

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

yeti

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

qwantify/vr

Rule Path
Disallow /search

Other Records

Field Value
crawl-delay 30

ccbot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

sogou

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

*

Rule Path
Disallow /search
Disallow /edits
Disallow /*/edits$
Disallow /*/edits?*$
Disallow /*/edits/*
Disallow /users/*/forum_posts
Back to top

Other Records

Field Value
sitemap https://knowyourmeme.com/sitemap.xml
sitemap https://knowyourmeme.com/memes/confirmed/sitemap.xml
sitemap https://knowyourmeme.com/memes/researching/sitemap.xml
sitemap https://knowyourmeme.com/memes/most-viewed/sitemap.xml
sitemap https://knowyourmeme.com/photos/trending/sitemap.xml
sitemap https://knowyourmeme.com/photos/most-liked/sitemap.xml
sitemap https://knowyourmeme.com/photos/most-viewed/sitemap.xml
sitemap https://knowyourmeme.com/photos/most-commented/sitemap.xml
sitemap https://knowyourmeme.com/photos/most-favorited/sitemap.xml
sitemap https://knowyourmeme.com/videos/trending/sitemap.xml
sitemap https://knowyourmeme.com/videos/most-liked/sitemap.xml
sitemap https://knowyourmeme.com/videos/most-viewed/sitemap.xml
sitemap https://knowyourmeme.com/videos/most-commented/sitemap.xml
sitemap https://knowyourmeme.com/videos/most-favorited/sitemap.xml
sitemap https://knowyourmeme.com/episodes/sitemap.xml
Back to top