Spiderability test

This test was removed in Sitebeam 5.1. 

What does it do?

Test whether or not a website can be spidered, i.e. explored automatically by search engines and other automated programs.

Example results

Example of spiderability test results

Why is it important?

Spidering is the process by which a computer program (such as Sitebeam, Google or Yahoo) can discover the pages in a website – it reads one webpage, looks for pages that page links to, before reading those pages, and so on. If for any reason a website cannot be spidered, it will suffer enormously:

  • It can’t be explored by search engines, and will usually not appear in them (or appear lower than they should).
  • It will usually be totally inaccessible, as most software to help the visually impaired or disabled is unable to penetrate the website. In many countries, this could make the website illegal.

Sites that are non-spiderable should fix this as their first (if only) SEO priority.

How is it measured?

Sitebeam has a more advanced spider than most, with the unusual capability of exploring many traditionally ‘non-spiderable’ sites. In addition, Sitebeam knows about any manual pages which you have instructed it to explore manually.

This means that (paradoxically) Sitebeam can usually identify non-spiderable content via use of its own spider. Whenever a page is accessed via a means that a traditional spider would not manage, that route is marked as non-spiderable. Pages that can only be accessed via a non-spiderable route are used to calculate the volume of the site which is non-spiderable.

The score is calculated based on the proportion of spiderable pages, with a perfect 10 being awarded to sites where every page is spiderable.

Potential problems

My site doesn’t need to be spiderable. If you’re testing a site that is intentionally non-spiderable, e.g. behind a login area, this test is redundant. Exclude this test to circumvent this issue.

How to improve this score

Ensure the website can be spidered. Depending on the technology you use this could be a trivial problem, or an extremely complex and involved one.

How to use this test effectively

This test should return a perfect score, or very close to it. If this score is low, it should be immediately addressed as your first priority.

Was this article helpful? Contact our support team if you have a question.