| How Deep into Your Site Are the Spiders Crawling? |
By Robin Nobles
Have you ever wondered how far down into your site the search
engine spiders are going? Are the pay inclusion spiders coming at
regular intervals like they're supposed to?
Wouldn't it be great to see which spiders are visiting your site
and which pages they're crawling . . . at a glance?
Robot Manager Pro
(http://www.websitemanagementtools.com/robot.html) is undoubtedly
one of the most helpful software programs I've seen in a long
time. It actually serves two purposes:
1) It will create robots.txt files for you, and
2) It will monitor spider activity to your Web sites.
What is a Robots.txt File?
A robots.txt file is simply a text file that "disallows" (or
keeps spiders out) of certain pages or areas of your Web site.
For example, if you have a section of your site that's devoted to
your personnel, you can keep the spiders from finding and
crawling that entire section by including it in a robots.txt
file.
If you've ever tried to create a robots.txt file by hand, you
know that it's nothing short of tedious, and it's so easy to make
a mistake. So having a program create the file for you
automatically is ideal.
However, what's really exciting to me about Robot Manager Pro is
how it monitors spider traffic. But first, let's look at the
software as a whole.
At Look at Robot Manager Pro
The program is simple to use. On the left-hand side of the
screen, you'll see numbered areas. On the right-hand side of the
screen, you'll see a Quick Help section that explains how to use
that portion of the program.
What you're doing in steps #1-3 is creating your robots.txt file.
You'll tell the program which areas or pages that you want to
keep the spiders from crawling, and the software will create the
robots.txt file for you and upload it to the root directory of
your Web site.
Keep in mind that you don't have to create a robots.txt file if
you don't want or need to. You can simply use the spider
monitoring portion by itself.
Now Let's Get to the "Fun" Stuff
As I stated previously, the most exciting part of the program to
me is the way it monitors spider traffic.
The first time you download your log files to analyze your
traffic, it won't be quite as interesting as it gets each
additional time you use the program.
Every time you use the program, you'll open the initial file you
created, hit "refresh" to download the latest log files, and let
the program update your spider visits.
If we were to click on one of the spiders, such as Googlebot,
we'd see how many pages from this site that Google has
spidered.
Google is a deep crawling engine, and it really can find most of
your Web pages if you provide text links to them.
If we were to click on any of the actual Web pages, we'd see a
screen that tells us how often the spider has visited the page.
Can you see the value in having this type of information?
* If an engine hasn't found one of your important pages, maybe
you need to add more links to that page from pages that are
getting spidered more frequently.
* When you create new pages, you'll be able to see at a glance
when the spiders find and crawl those pages.
* If you use pay inclusion, you'll be able to monitor whether the
spiders are visiting like they're supposed to.
* You'll know exactly which pages of your Web site are being
spidered by which search engines, and when.
In Conclusion
If you're interested in tracking spider visits to your site,
Robot Manager Pro is a "must have" software program for you. It's
fast and easy to use, and it provides key information about how
the spiders are crawling your Web site.
Visit the Robot Manager Pro Web site at
http://www.websitemanagementtools.com/robot.html to learn more or
to download a free trial version. They also offer a regular
version of Robot Manager, but it doesn't monitor spider traffic.
Robin Nobles with Search Engine Workshops teaches SEO strategies
the "stress free" way through hands-on, search engine marketing
workshops (http://www.searchengineworkshops.com) in locations
across the globe and online search engine marketing courses
(http://www.onlinewebtraining.com). Visit the World Resource
Center, a new networking community for search engine marketers.
(http://www.sew-wrc.com)
Copyright 2003 Robin Nobles. All rights reserved.
|
Back to Articles Index
|
|
 |
|
