origin.pegipegi.com
robots.txt

Robots Exclusion Standard data for origin.pegipegi.com

Resource Scan

Scan Details

Site Domain origin.pegipegi.com
Base Domain pegipegi.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2024-10-04T18:10:10+00:00
Next Scan 2025-01-02T18:10:10+00:00

Last Successful Scan

Scanned2023-12-03T18:03:30+00:00
URL https://origin.pegipegi.com/robots.txt
Redirect https://www.pegipegi.com/robots.txt
Redirect Domain www.pegipegi.com
Redirect Base pegipegi.com
Domain IPs 13.33.88.101, 13.33.88.6, 13.33.88.75, 13.33.88.82
Redirect IPs 18.138.8.30, 3.1.104.174
Response IP 3.1.104.174
Found Yes
Hash cec4fd07cb23be29064739c35ee6e2ffcf58bdb531933e14fc83f96d08cef83e
SimHash e4014385c32a

Groups

*

Rule Path
Allow /
Disallow /cgi-bin/
Disallow /pembayaran/kredivo/finish.html
Disallow /mobile/pembayaran/kredivo/finish.html
Disallow /rsc/common/js/base/TripadvisorShow.js
Disallow /rsc/common/mobile/js/TripadvisorShowSp.js
Disallow /HotelUniqueReview/v1/hotel/tripadvisor/review/list

yandex

Rule Path
Disallow /

moget
ichiro

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

python-requests/2.13.0

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

bingbot
tineye-bot

Rule Path
Disallow /

facebookexternalhit
facebookexternalhit/1.1

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.pegipegi.com/siteindex-hotel.xml
sitemap https://www.pegipegi.com/flight/xml/airlines-sitemap.xml
sitemap https://www.pegipegi.com/flight/xml/airlinesmonth-sitemap.xml
sitemap https://www.pegipegi.com/flight/xml/area-sitemap.xml
sitemap https://www.pegipegi.com/flight/xml/areadate-sitemap.xml
sitemap https://www.pegipegi.com/flight/xml/areatoarea-sitemap.xml
sitemap https://www.pegipegi.com/flight/xml/areatoareamonth-sitemap-one.xml
sitemap https://www.pegipegi.com/flight/xml/areatoareamonth-sitemap-two.xml
sitemap https://www.pegipegi.com/flight/xml/areatoareamonth-sitemap-three.xml
sitemap https://www.pegipegi.com/flight/xml/promo-scheduledate-sitemap.xml
sitemap https://www.pegipegi.com/kereta-api/xml/sitemap_train_area.xml
sitemap https://www.pegipegi.com/kereta-api/xml/sitemap_train_areatoarea.xml
sitemap https://www.pegipegi.com/travel/sitemap.xml
sitemap https://www.pegipegi.com/siteindex-bus.xml