During a meeting with a Client last week we were doing some competitive analysis and came across a particular page in the SERPs where only the URL was proposed during our queries – it was a situation they had never came across before and it is a rather rare occasion when Google proposes only the URL of a page instead of the typical title and description.
What’s different about these pages ?
They have been identified via external linking but have not been crawled by Google because a robots.txt file is in place requesting that page not be spidered. An unusual situation that can (and does) happen every now and then.
This table is a brief summary of the different possibilities you have to manage your website presence using the robots.txt file or the NOINED and NOFOLLOW tags, considering also another 2 important aspects i.e. if they allow Page Rank to pass along and if the pages are indexed.
I’m not a Page Rank Sculptor (=one who believes that you should try to manipulate the flow of Page Rank within your site by using the NOFOLLOW attribute) but you might be interested in knowing how you can use these attributes and who is doing what on your site …
| Passes PR ? | Crawled ? | Indexed ? | |
| robots.txt | YES | NO | YES |
| NOINDEX | YES | YES | NO |
| NOFOLLOW | NO | YES | YES |
{ 1 comment }
Interesting
Comments on this entry are closed.