sptechnology.info
robots.txt

Robots Exclusion Standard data for sptechnology.info

Archived Snapshots

Resource Scan

Scan Details

Site Domain	sptechnology.info
Base Domain	sptechnology.info
Scan Status	Ok
Last Scan	2025-07-12T14:01:55+00:00
Next Scan	2025-08-11T14:01:55+00:00

Last Scan

Scanned	2025-07-12T14:01:55+00:00
URL	https://sptechnology.info/robots.txt
Domain IPs	45.77.190.253
Response IP	45.77.190.253
Found	Yes
Hash	0ffdedb75632526622d05973846f23b2952a55420c04ca9ba288bc319d9fabe3
SimHash	520a1571c385

Groups

*

Rule	Path
Allow	/

Rule

Path

Allow

amazonbot

Rule	Path
Disallow	/

Rule

Path

Disallow

aihitbot

Rule	Path
Disallow	/

Rule

Path

Disallow

applebot

Rule	Path
Disallow	/

Rule

Path

Disallow

aspiegelbot

Rule	Path
Disallow	/

Rule

Path

Disallow

barkrowler

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bytespider

Rule	Path
Disallow	/

Rule

Path

Disallow

ccbot

Rule	Path
Disallow	/

Rule

Path

Disallow

claudebot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

criteobot

Rule	Path
Disallow	/

Rule

Path

Disallow

dataforseobot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

detectify

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

flipboardproxy

Rule	Path
Disallow	/

Rule

Path

Disallow

go-http-client

Rule	Path
Disallow	/

Rule

Path

Disallow

grapeshot

Rule	Path
Disallow	/

Rule

Path

Disallow

gptbot

Rule	Path
Disallow	/

Rule

Path

Disallow

httpclient

Rule	Path
Disallow	/

Rule

Path

Disallow

imagesiftbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ias-ie/3.3

Rule	Path
Disallow	/

Rule

Path

Disallow

ias-va/3.3

Rule	Path
Disallow	/

Rule

Path

Disallow

intelx.io_bot

Rule	Path
Disallow	/

Rule

Path

Disallow

java

Rule	Path
Disallow	/

Rule

Path

Disallow

linkpadbot

Rule	Path
Disallow	/

Rule

Path

Disallow

megaindex

Rule	Path
Disallow	/

Rule

Path

Disallow

meta-externalagent

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot

Rule	Path
Disallow	/

Rule

Path

Disallow

netcraftsurveyagent

Rule	Path
Disallow	/

Rule

Path

Disallow

owler

Rule	Path
Disallow	/

Rule

Path

Disallow

petalbot

Rule	Path
Disallow	/

Rule

Path

Disallow

python-urllib

Rule	Path
Disallow	/

Rule

Path

Disallow

riddler

Rule	Path
Disallow	/

Rule

Path

Disallow

scrapy

Rule	Path
Disallow	/

Rule

Path

Disallow

screaming frog

Rule	Path
Disallow	/

Rule

Path

Disallow

semrushbot

Rule	Path
Disallow	/

Rule

Path

Disallow

serpstatbot

Rule	Path
Disallow	/

Rule

Path

Disallow

timpibot

Rule	Path
Disallow	/

Rule

Path

Disallow

trovitbot

Rule	Path
Disallow	/

Rule

Path

Disallow

velenpublicwebcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

zgrab

Rule	Path
Disallow	/

Rule

Path

Disallow

zoominfobot

Rule	Path
Disallow	/

Rule

Path

Disallow

Other Records

Field	Value
sitemap	https://sptechnology.info/sitemaps/sitemap.xml
sitemap	https://sptechnology.info/sitemaps/sitemap_list.xml
sitemap	https://sptechnology.info/feed.rss

Field

Value

sitemap

https://sptechnology.info/sitemaps/sitemap.xml

sitemap

https://sptechnology.info/sitemaps/sitemap_list.xml

sitemap

https://sptechnology.info/feed.rss

sptechnology.inforobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

amazonbot

aihitbot

applebot

aspiegelbot

barkrowler

blexbot

bytespider

ccbot

claudebot/1.0

criteobot

dataforseobot/1.0

detectify

dotbot

flipboardproxy

go-http-client

grapeshot

gptbot

httpclient

imagesiftbot

ias-ie/3.3

ias-va/3.3

intelx.io_bot

java

linkpadbot

megaindex

meta-externalagent

mj12bot

netcraftsurveyagent

owler

petalbot

python-urllib

riddler

scrapy

screaming frog

semrushbot

serpstatbot

timpibot

trovitbot

velenpublicwebcrawler

zgrab

zoominfobot

Other Records

sptechnology.info
robots.txt