Log In Sign Up

travers.com
robots.txt

Robots Exclusion Standard data for travers.com

Resource Scan

Scan Details

Site Domain travers.com
Base Domain travers.com
Scan Status Ok
Last Scan2024-10-22T14:26:45+00:00
Next Scan 2024-11-21T14:26:45+00:00

Last Scan

Scanned2024-10-22T14:26:45+00:00
URL https://travers.com/robots.txt
Redirect https://www.travers.com/robots.txt
Redirect Domain www.travers.com
Redirect Base travers.com
Domain IPs 151.101.1.124, 151.101.129.124, 151.101.193.124, 151.101.65.124
Redirect IPs 151.101.1.124, 151.101.129.124, 151.101.193.124, 151.101.65.124
Response IP 199.232.45.124
Found Yes
Hash f67da8dacad9fe113e555b540d56e73dc1669a18d8c23b146f2232f9bf7a7219
SimHash 38153b4a5130

Groups

adbeat_bot
alexibot
appengine
aqua_products
archive.org_bot
archive
asterias
b2w/0.1
backdoorbot/1.0
barkrowler
becomebot
blekkobot
blexbot
blowfish/1.0
bookmark search tool
botalot
builtbottough
bullseye/1.0
bunnyslippers
cazoodlebot
ccbot
chatgpt-user
cheesebot
cherrypicker
cherrypickerelite/1.0
cherrypickerse/1.0
chroot
copernic
copyrightcheck
cosmos
crescent
crescent internet toolpak http ole control v.1.0
dittospyder
dotbot
dotbot/1.0
dumbot
emailcollector
emailsiphon
emailwolf
enterprise_search
enterprise_search/1.0
erocrawler
es
etaospider
exabot
extractorpro
fairad client
flaming attackbot
foobot
gaisbot
getright/4.2
gigabot
gptbot
grub
grub-client
go-http-client
harvest/1.5
hatena antenna
hloader
http://www.searchengineworld.com bot
http://www.webmasterworld.com bot
httplib
humanlinks
ia_archiver
ia_archiver/1.6
infonavirobot
iron33/1.0.2
jamesbot
jennybot
jetbot
jetbot/1.0
jorgee
kenjin spider
keyword density/0.9
larbin
lexibot
libweb/clshttp
linkextractorpro
linkpadbot
linkscan/8.1a unix
linkwalker
lnspiderguy
looksmart
lwp-trivial
lwp-trivial/1.34
mata hari
megalodon
microsoft url control
microsoft url control - 5.01.4511
microsoft url control - 6.00.8169
miixpc
miixpc/4.2
mister pix
mj12bot
moget
moget/2.1
mozilla
mozilla
mozilla/3
mozilla/4
mozilla/4.0 (compatible; bullseye; windows 95)
mozilla/4.0 (compatible; msie 4.0; windows 2000)
mozilla/4.0 (compatible; msie 4.0; windows 95)
mozilla/4.0 (compatible; msie 4.0; windows 98)
mozilla/4.0 (compatible; msie 4.0; windows nt)
mozilla/4.0 (compatible; msie 4.0; windows xp)
mozilla/5
msiecrawler
naver
nerdybot
netants
netmechanic
nicerspro
nutch
offline explorer
openbot
openfind
openfind data gathere
oracle ultra search
perman
propowerbot/2.14
prowebwalker
psbot
python-urllib
queryn metasearch
radiation retriever 1.1
repomonkey
repomonkey bait & tackle/v1.01
rma
rogerbot
scooter
searchpreview
semrushbot
semrushbot
semrushbot-sa
seokicks-robot
sitesnagger
sootle
spankbot
spanner
spbot
stanford
stanford comp sci
stanford compclub
stanford compsciclub
stanford spiderboys
surveybot
surveybot_ignoreip
suzuran
szukacz/1.4
szukacz/1.4
teleport
teleportpro
telesoft
teoma
the intraformant
thenomad
tocrawl/urldispatcher
true_robot
true_robot/1.0
turingos
typhoeus
url control
url_spider_pro
urly warning
vci
vci webviewer vci webviewer win32
voltron
web image collector
webauto
webbandit
webbandit/3.50
webcopier
webenhancer
webmasterworld extractor
webmasterworldforumbot
websauger
website quester
webster pro
webstripper
webvac
webzip
webzip/4.0
wget
wget/1.5.3
wget/1.6
www-collector-e
xenu's
xenu's link sleuth 1.1c
zeus
zeus 32297 webster pro v2.9 win32
zeus link scout
zoominfobot

Rule Path
Disallow /

*

Rule Path Comment
Disallow /app/ -
Disallow /bin/ -
Disallow /dev/ -
Disallow /lib/ -
Disallow /phpserver/ -
Disallow /pkginfo/ -
Disallow /report/ -
Disallow /setup/ -
Disallow /update/ -
Disallow /var/ -
Disallow /vendor/ -
Disallow /index.php/ -
Disallow /catalog/product_compare/ -
Disallow /control/ -
Disallow /customize/ -
Disallow /newsletter/ -
Disallow /sendfriend/ -
Disallow /wishlist/ -
Disallow /onepagecheckout/ -
Disallow /catalog/ -
Disallow /review/ -
Disallow /customer/ -
Disallow /*SID%3D -
Disallow /composer.json -
Disallow /composer.lock -
Disallow /CONTRIBUTING.md -
Disallow /CONTRIBUTOR_LICENSE_AGREEMENT.html -
Disallow /COPYING.txt -
Disallow /Gruntfile.js -
Disallow /LICENSE.txt -
Disallow /LICENSE_AFL.txt -
Disallow /nginx.conf.sample -
Disallow /package.json -
Disallow /php.ini.sample -
Disallow /RELEASE_NOTES.txt -
Disallow /*.php$ -
Disallow /*.CVS -
Disallow /*.Zip$ -
Disallow /*.Svn$ -
Disallow /*.Idea$ -
Disallow /*.Sql$ -
Disallow /*.Tgz$ -
Disallow /catalogsearch/result/? -
Disallow /catalogsearch/result/?brand*brand* -
Allow /catalogsearch/result/?brand* -
Disallow /category/*?wheel_diameter* -
Disallow /category/*?for_use_with* -
Disallow /category/*?shopby_instock* -
Disallow /category/*?wheel_thickness* -
Disallow /category/*?abrasive_material* -
Disallow /category/*?hole_thread_size* -
Disallow /category/*?outside_diameter* -
Disallow /category/*?type* -
Disallow /category/*?brush_diameter* -
Disallow /category/*?filament_diameter* -
Disallow /category/*?arbor_hole_size* -
Disallow /category/*?fill_material* -
Disallow /category/*?face_width* -
Disallow /category/*?trim_length* -
Disallow /category*?filam_wire* -
Disallow /category/*?wire_type* -
Disallow /category*?wire_diameter* -
Disallow /category/*?brush_length* -
Disallow /category/*?brand* -
Disallow /category/*?brush_type* -
Disallow /category/*?brush_width* -
Disallow /category/*?overall_length* -
Disallow /category/*?overall_width* -
Disallow /category/*?overall_thickness* -
Disallow /category/*?grit* -
Disallow /category/*?size* -
Disallow /category/*?buffing_composition* -
Disallow /category/*?compound_type* -
Disallow /category/*?color* -
Disallow /category/*?compound_grade* -
Disallow /category/*?mandr_shank_diame* -
Disallow /category/*?maximum_diameter* -
Disallow /category/*?bob_material* -
Disallow /category/*?bob_shape* -
Disallow /category/*?stone_material* -
Disallow /category/*?width* -
Disallow /category/*?length* -
Disallow /category/*?thickness* -
Disallow /category/*?shape* -
Disallow /category/*?grade* -
Disallow /category/*?oil_filled* -
Disallow /category/*?buff_type* -
Disallow /category/*?density* -
Disallow /category/*?no_of_ply* -
Disallow /category/*?thickness* -
Disallow /category/*?band_width* -
Disallow /category/*?shank_diameter* -
Disallow /category/*?maximum_roll_diameter* -
Disallow /category/*?pilot_hole_diameter* -
Disallow /category/*?disc_diameter* -
Disallow /category/*?dry* -
Disallow /category/*?tape_width* -
Disallow /category/*?backing_pad_type* -
Disallow /category/*?pad_diameter* -
Disallow /category/*?for_use_with* -
Disallow /category/*?flap_disk_type* -
Disallow /category/*?mount_type* -
Disallow /category/*?blade_type* -
Disallow /category/*?tool_material* -
Disallow /category/*?hole_chamfer* -
Disallow /category/*?blade_orentation* -
Disallow /category/*?blade_holder_adapter_types* -
Disallow /category/*?blade_material* -
Disallow /category/*?blade_number_compatibility* -
Disallow /category/*?brand_name_compatibility* -
Disallow /category/*?swivel_or_scraper* -
Disallow /category/*?holder_compatibility* -
Disallow /category/*?point_shape_code* -
Disallow /category/*?point_shape* -
Disallow /category/*?head_diameter* -
Disallow /category/*?heas_thickness* -
Disallow /category/*?arbor_hole_diameter* -
Disallow /category/*?wheel_type* -
Disallow /category/*?thickness_of_wheels* -
Disallow /category/*?diamond_carat_size* -
Disallow /category/*?holds_shank_diameter* -
Disallow /category/*?adjustable* -
Disallow /category/*?point_type* -
Disallow /category/*?dresser_type* -
Disallow /category/*?no_of_stones* -
Disallow /category/*?shank_diameter* -
Disallow /category/*?stick_shape* -
Disallow /category/*?grit_compatability* -
Disallow /category/*?file_type* -
Disallow /category/*?coarseness_cut* -
Disallow /category/*?cut_type* -
Disallow /category/*?tang_or_handle* -
Disallow /category/*?file_card_type* -
Disallow /category/*?for_files* -
Disallow /category/*?wheel_width* -
Disallow /category/*?voltage* -
Disallow /category/*?house_power* -
Disallow /category/*?phase* -
Disallow /category/*?dust_collection_system* -
Disallow /category/*?arbor* -
Disallow /category/*?file_set_type* -
Disallow /category/*?file_types_included* -
Disallow /category/*?no._of_pieces* -
Disallow /category/*?type_of_cut* -
Disallow /category/*?industry_number* -
Disallow /category/*?level_of_precision* -
Disallow /category/*?cut* -
Disallow /category/*?length_of_cut* -
Disallow /category/*?repair_range* -
Disallow /category/*?for_inter_or_exter* -
Disallow /category/*?minim_bore_diame* -
Disallow /category/*?maxim_bore_diame* -
Disallow /category/*?abrasive_length* -
Disallow /category/*?lab_type* -
Disallow /category/*?lab_diameter* -
Disallow /category/*?lab_length* -
Disallow /category/*?abrasive_type* -
Disallow /category/*?sheet_size* -
Disallow /category/*?pad_color* -
Disallow /category/*?attaching_system* -
Disallow /category/*?coarseness* -
Disallow /category/*?file_style* -
Disallow /category/*?filament_diameter* -
Disallow /category/*?flared* -
Disallow /category/*?filam_wire_diame_range* -
Disallow /category/*?container_size* -
Disallow /category/*?container_type* -
Disallow /category/*?form* -
Disallow /category/*?series* -
Disallow /category/*?color_family* -
Disallow /category/*?finishing_coating* -
Disallow /category/*?watts* -
Disallow /category/*?maximum_tempatures* -
Disallow /category/*?melting_point* -
Disallow /category/*?chemical_type* -
Disallow /category/*?paintable* -
Disallow /category/*?function* -
Disallow /category/*?frequency_band* -
Disallow /category/*?range* -
Disallow /category/*?channels* -
Disallow /category/*?battery_type* -
Disallow /category/*?frequency* -
Disallow /category/*?material_application* -
Disallow /category/*?disinfectant* -
Disallow /category/*?iso_grade* -
Disallow /category/*?form_or_style* -
Disallow /category/*?sae_grade* -
Disallow /category/*?features* -
Disallow /category/*?tool_type* -
Disallow /category/*?npt_size* -
Disallow /category/*?closing_type* -
Disallow /category/*?grease_coupler_type* -
Disallow /category/*?adapter_type* -
Disallow /category/*?swivel_type* -
Disallow /category/*?capacity* -
Disallow /category/*?lubrication_type* -
Disallow /category/*?rigid_or_flexible_hose* -
Disallow /category/*?hose_length* -
Disallow /category/*?pump_type* -
Disallow /category/*?reach* -
Disallow /category/*?oil_removale_capacity* -
Disallow /category/*?belt_width* -
Disallow /category/*?belt_length* -
Disallow /category/*?belt_material* -
Disallow /category/*?tool_type* -
Disallow /category/*?thread_mount* -
Disallow /category/*?tool_nr* -
Disallow /category/*?cutting_direction* -
Disallow /category/*?minimum_thread_diameter* -
Disallow /category/*?shank_width* -
Disallow /category/*?shank_height* -
Disallow /category/*?included_angle* -
Disallow /category/*?single_or_double_end* -
Disallow /category/*?height* -
Disallow /category/*?recommended_bars* -
Disallow /category/*?square_size* -
Disallow /category/*?radius* -
Disallow /category/*?minimum_bore* -
Disallow /category/*?maximum_bore_depth* -
Disallow /category/*?minim_hole_diame* -
Disallow /category/*?threads_per_inch* -
Disallow /category/*?projection* -
Disallow /category/*?for_tool_bit_size* -
Disallow /category/*?hole_size* -
Disallow /category/*?broach_style* -
Disallow /category/*?bushing_type* -
Disallow /category/*?minim_lengt_of_cut* -
Disallow /category/*?maxim_lengt_of_cut* -
Disallow /category/*?no_of_broaches* -
Disallow /category/*?no_of_bushings* -
Disallow /category/*?no_of_combinations* -
Disallow /category/*?case_type_material* -
Disallow /category/*?minimum_capacity* -
Disallow /category/*?maximum_capacity* -
Disallow /category/*?geometric_equivalent* -
Disallow /category/*?thread_type* -
Disallow /category/*?to_fit_die_head_size* -
Disallow /category/*?thread_size* -
Disallow /category/*?range_of_pilot_sizes* -
Disallow /category/*?pilot_hole_diamter* -
Disallow /category/*?pilot_diamter* -
Disallow /category/*?port_size* -
Disallow /category/*?port_standard* -
Disallow /category/*?cutting_diameter* -
Disallow /category/*?screw_size* -
Disallow /category/*?tap_size* -
Disallow /category/*?step_length* -
Disallow /category/*?no_of_flutes* -
Disallow /category/*?no_of_pieces_in_set* -
Disallow /category/*?bla_hol_ada_typ* -
Disallow /category/*?body_diameter* -
Disallow /category/*?round_die_outsi_diame* -
Disallow /category/*?flute_length* -
Disallow /category/*?drill_point_angle* -
Disallow /category/*?industry_standard* -
Disallow /category/*?industry_standard_type* -
Disallow /category/*?drill_diameter* -
Disallow /category/*?morse_taper* -
Disallow /category/*?no_of_holes* -
Disallow /category/*?size_range* -
Disallow /category/*?size_of_holes* -
Disallow /category/*?point_diameter* -
Disallow /category/*?first_choice_for* -
Disallow /category/*?point_sharp_type* -
Disallow /category/*?single_angle* -
Disallow /category/*?no_of_teeth* -
Disallow /category/*?style* -
Disallow /category/*?helix_angle* -
Disallow /category/*?chamfer_angle* -
Disallow /category/*?radius_size* -
Disallow /category/*?radius_r* -
Disallow /category/*?largest_diameter* -
Disallow /category/*?corner_radius* -
Disallow /category/*?cutter__diameter* -
Disallow /category/*?cutter_thickness* -
Disallow /category/*?pitch* -
Disallow /category/*?coolant_through* -
Disallow /category/*?american_standard_* -
Disallow /category/*?tooth_pattern* -
Disallow /category/*?removes_sizes* -
Disallow /category/*?use_for_tap_sizes* -
Disallow /category/*?knurl_size* -
Disallow /category/*?knurling_capacity* -
Disallow /category/*?shank_length* -
Disallow /category/*?knurl_hole* -
Disallow /category/*?knurl_series* -
Disallow /category/*?circle_diameter* -
Disallow /category/*?pressure_angle* -
Disallow /category/*?diametrical_pitch* -
Disallow /category/*?module* -
Disallow /category/*?cutter_number* -
Disallow /category/*?shank_style* -
Disallow /category/*?reaming_range* -
Disallow /category/*?no._of_blades* -
Disallow /category/*?flute_shape* -
Disallow /category/*?small_end_diameter* -
Disallow /category/*?large_end_diameter* -
Disallow /category/*?diameter_small_end* -
Disallow /category/*?diameter_large_end* -
Disallow /category/*?taper_angle* -
Disallow /category/*?gripping_end* -
Disallow /category/*?brown_&_sharp_taper* -
Disallow /category/*?use_arbor_* -
Disallow /category/*?size_type* -
Disallow /category/*?packing* -
Disallow /category/*?teeth_per_inch* -
Disallow /category/*?flexible* -
Disallow /category/*?fits_hole_saw_diameter* -
Disallow /category/*?fits_arbor_no* -
Disallow /category/*?chuck_size* -
Disallow /category/*?saw_diameter* -
Disallow /category/*?no_of_hole_saws* -
Disallow /category/*?maxim_cutti_depth* -
Disallow /category/*?saw_style* -
Disallow /category/*?tooth_type* -
Disallow /category/*?hole_diameter* -
Disallow /category/*?kerf* -
Disallow /category/*?thread_length* -
Disallow /category/*?limit* -
Disallow /category/*?drill_size* -
Disallow /category/*?drill_length* -
Disallow /category/*?chamfer_style* -
Disallow /category/*?thread_limit* -
Disallow /category/*?inch_or_metric* -
Disallow /category/*?thread_size_range* -
Disallow /category/*?tap_size_range* -
Disallow /category/*?drill_size_range* -
Disallow /category/*?tap_capacity_range* -
Disallow /category/*?pipe_tap_capacity* -
Disallow /category/*?wrench_style* -
Disallow /category/*?ratcheting* -
Disallow /category/*?lathe_swing* -
Disallow /category/*?holder_dimensions* -
Disallow /category/*?blade_size* -
Disallow /category/*?bar_diameter* -
Disallow /category/*?screw_model_* -
Disallow /category/*?clamp_number* -
Disallow /category/*?tool_bit_square_size* -
Disallow /category/*?chipbreaker* -
Disallow /category/*?edge_width* -
Disallow /category/*?insert_ic* -
Disallow /category/*?torx_size* -
Disallow /category/*?ic_* -
Disallow /category/*?use_insert_ *
Disallow /category/*?coolant_use* -
Disallow /category/*?lead_angle* -
Disallow /category/*?hand* -
Disallow /category/*?holder* -
Disallow /category/*?insert_num* -
Disallow /category/*?maxim_cutti_diame* -
Disallow /category/*?minim_cutti_diame* -
Disallow /category/*?no_of_inserts* -
Disallow /category/*?drill_diame_x_lengt_ratio* -
Disallow /category/*?blade_series* -
Disallow /category/*?blade_diameter_cutting_range* -
Disallow /category/*?maximum_drill_depth* -
Disallow /category/*?cutter_diameter* -
Disallow /category/*?maximum_depth_of_cut* -
Disallow /category/*?neck_diameter* -
Disallow /category/*?neck_length* -

Other Records

Field Value Comment
crawl-delay 10 10 seconds between page requests
Back to top

Other Records

Field Value
sitemap https://www.travers.com/media/traversUs/sitemap.xml
Back to top

Comments

  • Directories
  • Paths (clean URLs)
  • Disallow: */referer/*
  • Files
  • CVS, SVN directory and dump files
  • All Brands as Indexable
  • Facets
  • Declaring Sitemap file
Back to top

Warnings

  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top