|
|
|
10 Tips to become an Affiliate Champion
Affiliates programs are one of the easiest ways to make money in the internet. Small fortunes have been made for smart people who have taken advantage of this powerful marketing technique. This tips intend to help you on your way to become a...
Affiliate Surprise
I can’t believe it! I open up my mail today, and what do I receive? A statement from one of my affiliates saying that I’ve earned a pretty good amount of money for the last quarter. Wow! How’d that happen? Since my web site is only a couple...
Increase Web Site Traffic by Writing Your Own Ezine Article
In my previous article (Part 1 - www.isitebuild.com/writearticles) we discussed the 7 ways to benefit from writing an ezine article. Part 2 will cover the formula and format you can use to write your own article as well as provide valuable...
Reciprocal Linking: A Disturbing Trend
In the world of Internet Marketing, and Web site promotion, nothing changes faster than the parameters that rule good Search Engine rankings and placement. This has led to a developed new science in the last few years, SEM (Search Engine...
Why Google Blog Search Matters to Your Business
According to Google, Google's Blog Search is "Google search technology focused on blogs". It includes search engine results specific to blogs not just in the Blogger.com community, but across the blogosphere at large. You can access it at ...
|
|
| |
|
|
|
|
|
|
How to make sure your site gets properly crawled and indexed by robots
Search engines have robots that come to your site and grab
everything there is to grab. But because competition is so
fierce, there is no way to get in the search engines, unless you
pay for ads or hire a SEO (Search Engine Optimization)
consultant, right? Wrong!
Even if you pay big money, if your site is not properly seen by
the robots used by search engines for indexing, chances are many
of your pages will never make it.
In this article I will discuss the importance of having your
website structured properly, the importance of using the old
fashioned hyperlinks versus the modern Flash menus, scripts and
extensions and provide you with a very simple and free tool that
will allow you to see your site in a similar fashion most
indexing robots do. But first, let's define some of the concepts.
What is a www robot?
A robot is a computer program that automatically reads web pages
and goes through every link that it finds.
The first robot was developed by MIT and launched in 1993. It
was named the World Wide Web Wander and its initial purpose was
of a purely scientific nature, its mission was to measure the
growth of the web. The index generated from the experiment's
results proved to be an awesome tool and effectively became the
first search engine. Most of the online stuff we can't live
without today was born as a side effect of some scientific
experiment.
What is a search engine?
Generically, a search engine is a program that searches through
a database. In the popular sense, as referred to the web, a
search engine is considered to be a system that has a user
search form, which can search through a repository of web pages
gathered by a robot.
What is a bot? What is a spider? What is a crawler?
Bot is just a shorter, cooler (for some) version of the word
robot. Spiders and crawlers are robots, only the names sound
more interesting in the press and within metro-geek circles. For
reasons of consistency, I will use the term robot throughout
this article, when referring to spiders, crawlers and bots.
Are there other... things that crawl out there?
Oh yeah, but these things are way beyond the scope of this
article. Well, for the conspiracy theory aficionados, let's
see... we have worms - self-replicating programs, webants (or
ants) - distributed cooperating robots, autonomous agents,
intelligent agents and many other bots and beasties.
How do robots work?
As with all other things technical, I believe that the only way
you will utilize a technology to its full potential and to your
best advantage is if and when you understand how that technology
works. When I say how it works, I don't mean intricate technical
details, but fundamental processes, big picture stuff.
Generally, robots are nothing but stripped down versions of web
browsers, programmed to automatically browse and record
information about web pages. There are some very specialized
robots out there, some that look only for blogs, some that index
nothing but images. Many (such as Google's GoogleBot) are based
on one of the first popular browsers, called Lynx. Lynx was
initially a pure text browser, therefore, in today's internet
Lynx would be extremely robust and fast. Basically, if you can
program, you can take Lynx, modify it and make a robot.
So how do these things actually work? They get a list of
websites, and literally start "browsing" them. They come to your
site and then start reading the pages and following every link,
while storing different information, such as page titles, the
actual text of the page, etc.
Based on the above, what would
happen if instead of your beloved
Internet Explorer, Firefox, Opera or whatever browser you are
attached to, you go dig on the internet and download a version
of the venerable Lynx browser?
I'll tell you what would happen, and some will probably accuse
me of giving away one of the secrets the SEO corporate community
does not want you to know:
You will be able to see your site very close to the way a robot
sees it. You will be able to look for errors in your pages and
track down navigation errors that might block a robot from
seeing portions of your site.
In plain English, let's say you built a great looking site.
There is an index page, the first page one sees when entering
your site. On that page you have the most incredible Flash
navigation system, with a huge button pointing to your products
and services and the rest of the site. If Lynx goes to your
index page and will not see a standard link, it will not be able
to see the rest of your site. There are extremely high chances
that a lot of indexing robots will not see your site either.
You will then understand why your very large site, that has one
of the most intricate and functional Flash based navigation
systems on the planet never makes it high into the search
engines, even after all your efforts of manually submitting it
everywhere. It's simply because you forgot to add basic
hyperlinks. It's because when you submit a site - even manually
- all that really happens is you telling the search engine "hey,
Mr. Search Engine, whenever you think you can find some time,
please send your trusty robot to my site".
Folks, robots can't usually use a navigation menu made in Flash,
Java script, PHP, etc. and will not be able to get to your
pages, it's as simple as that.
How do I get Lynx?
Lynx first started life as a UNIX application, written by the
University of Kansas as part of their campus-wide information
system. It then became a gopher application (a pre-web search
tool), then a web browser. The official page for Lynx is
http://lynx.isc.org, however, if you are not a Linux geek, used
to play with binary distribution files and used to compiling
your own apps (don't worry about what I just said), you might
want to find a version that someone else already made usable for
your computer. For example, if you are a PC user running
Windows, you might want to check links to "Win32 compiled
versions". At the time of writing, one such site is
http://csant.info/lynx.htm (called a distribution site) where
you can download a version that will install onto Windows
machines in a fashion that will be familiar to non-geeks. After
you install the browser, you might want to read the
documentation. To get you going and to alleviate your beginner
frustrations, I'll tell you that you must press the G key (as in
"go"), then type the complete URL of the site you want to browse
(starting with "http://"), then hit Enter. Use the arrows to
navigate.
Bottom line, use Lynx to verify that every page of site is
accessible and let the robots do all the work for you. You'll
save yourself a lot of aggravation and maybe some money that you
would waste on advertising your otherwise non-indexable site.
About the author:
Andrei co-owns bsleek ( http://www.bsleek.com ) - a site that
specializes in web hosting, design, promotional items, printing,
CD Presentations and more. Andrei is on the Board of Consultants
for Daterade.com and has amassed an extensive technical
knowledge and experience through his career as the CIO for a
major travel management company and through his past careers in
military research, data acquisition and aerospace engineering.
|
|
|
|
|
|