|
[an article] [vB 2 - 3] Getting better results from search engines
by
06 Jul 2003
Search engines, how they index your forum; and getting better results from them. This article is a brief overview of search engines, how they work and what you can do as an admin to improve your results on sites like google.com This thread is open to replies so fell free to if you have any questions.  What are search engines and 'bots'? Search engines are your friend, without search engines there would be no way to do research online. Search engines also help you (the webmaster) by bringing in hits. Search engines use things called bots to 'crawl' your site, these bots visit your website and index them so the info from your site can later be searched via their engines. Below are sections that explain how to get better results from bots and how to control what they crawl on your web-site. Tip 1: Link exchange/submit your site When bots spider sites they get around like any human, they click the links on the current page they are viewing. So, a good way to ensure more hits from a search engine bot is having your site linked from anther. This is why starting up a link exchange with sites similar to yours is always a good idea. You may also submit your URL directly to search engines so they may index them, here are a few links: https://www.google.com/addurl.html http://www.addurl.altavista.com/ http://www.hotbot.lycos.co.uk/submit.html http://www.alltheweb.com/add_url.php Tip 2: Meta tags Meta tags are used to allow search engines to index your pages more relevantly. We will only be dealing with two types of meta tags in this article, 'keywords' and 'descriptions'. Keywords are just that, common words that someone might input into a search engine that relate to your site. A description is also just that, a description of the content hosted on your web-site. Say for example I was building a site based on soda, my meta tags might read something like this: Code:
<meta name="keywords" content="soda,pepsi,coke" /> <meta name="description" content="This is a site all about soda ." /> see: http://www.vbulletintemplates.com/mods/showthread.php?threadid=1689 --editing the meta tags in vBulletin v3 v3 includes a way to do this from the admincp (without editing the templates), the options for controlling the meta tags are under the general settings in the vBulletin options area of the admin cp. Tip 3: Spider Friendly URL's this section is mostly for vB v2 users Most search engines can not spider urls with query stings in them (? and & for example), vBulletin makes use of these in nearly every url it generates. However in vbulletin the session hash causes most of our troubles, and because spiders (bots) are guest at our forums they will get the session hash in there urls (vB 2 only, v3 strips them stock). To solve these problems two hacks have been made, one converts all urls into friendly ones, the other strips the session hash but nothing more. Also, both these hacks will not make the pages optimized for search engines, check out the next tip for more on that. http://www.vbulletin.org/forum/showthread.php?s=&threadid=18035&highlight=spider+friendly+links http://www.vbulletin.org/forum/showthread.php?s=&threadid=50805&highlight=hash - google bot only Tip: 4: Archives Archives are wonderful tools that make a spider bots life a lot easier! They are search engine optimized to allow the bots to index you more quickly. For an example of an archive check out vBulletintemplates here: http://www.vbulletintemplates.com/mods/archive/index.php/ Notice how that page has a simple layout, dose not use images, and uses friendly url's. All these things make this page 'optimized' for spider bots. You can have an archive of your own by turning the option on in the control panel (vB 3 only), or installing an archive hack if you are running vB v2. Tip 5: using robots.txt to save bandwidth Like I have mentioned before, spiders (bots) are just like any guest. They will see everything a normal user would see including images. But we can control what a bot can 'see' with a file called robots.txt. robots.txt is a file any spider checks for before it begins to spider the site, the spider is only allowed to do what it is told to in robots.txt note: robot.txt is always uploaded to the ROOT public_html folder So, with robots.txt we can dis-allow a bot from reading a file, for example the code below would dis-allow any bot from loading images on my site: Code:
User-agent: * Disallow: /forum/images/ Code:
User-agent: * Disallow: /forum/usercp.php Code:
User-agent: * Disallow: /forum/attachment.php Disallow: /forum/avatar.php Disallow: /forum/editpost.php Disallow: /forum/member.php Disallow: /forum/member2.php Disallow: /forum/misc.php Disallow: /forum/moderator.php Disallow: /forum/newreply.php Disallow: /forum/newthread.php Disallow: /forum/online.php Disallow: /forum/poll.php Disallow: /forum/postings.php Disallow: /forum/printthread.php Disallow: /forum/private.php Disallow: /forum/private2.php Disallow: /forum/report.php Disallow: /forum/search.php Disallow: /forum/sendtofriend.php Disallow: /forum/threadrate.php Disallow: /forum/usercp.php Disallow: /forum/admin/ Disallow: /forum/images/ Disallow: /forum/mod/ Tip 6: Dont cheat! The last thing you every want to do is cheat spiders, this can get you banned from many index's. Here are a few things you should never do. - Hidden link to your archives - Image link to archive without alt tag - Do not use automatic search engine submitters |