Googlebot is getting on my nerves

[AtW]

I am refactoring in REST URL conventions into Plan B, I just wanted to know the priority.

Make it high - optimising your pages titles, urls etc is one of the easiest long term wins that you can get, the sooner you do (especially the urls) it the better.

[NickFitz]

I have read the google SEO doc, what I was asking was the magnitude of the difference, as I previously pointed out I understood there was a difference. I am refactoring in REST URL conventions into Plan B, I just wanted to know the priority.

It's just another thing to add to the Plan B list, i just wanted to know where to put it on the list.

The Webmaster Central piece should give you a better idea of how to prioritise it. However, it's probably worthwhile going for meaningful URLs right from the start, as if you switch later you'll have to put a whole bunch of redirects in place.

I'd suggest going for the ".html" extension (e.g. http://example.com/foo.html), whatever the backend technology might be; the URL should reflect the content type of the resource's representation, rather than revealing details of the backend implementation. Alternatively, don't use an extension at all (http://example.com/foo).

Also consider whether you'll want to offer different representations of the same resource; if you want to present a resource like a list of products as, say HTML, Atom, CSV and JSON (so you can provide an API) then using the appropriate extension can make content negotiation much easier:

http://example.com/foo
http://example.com/foo.html
http://example.com/foo.xml
http://example.com/foo.csv
http://example.com/foo.json

would all be references to the same resource, just served in different formats. (The first without an extension would allow for content negotiation using the "Accept" header, and probably default to HTML.)

If you go the road of serving different representations of the same resource at different URLs, you should look into using <link rel="canonical"> as well.

[suityou01]

Virtually all these questions are addressed in Google's own guide on the subject, which I linked to the last time this question was asked in January.

The Google Webmaster Central blog also posted a pretty detailed examination of the matter.

As an aside, I always find it quite amusing that people ask questions about these matters on various forums where they have no real chance of judging the quality of the advice they receive (there's a lot of mumbo-jumbo, voodoo-chicken-waving and cargo-cult-nonsense surrounding SEO) when the answer to their question has usually been published by Google itself.

Google may be understandably secretive about their core algorithms, but they publish an enormous amount of information about SEO best practices. After all, the easier people make it for Google to find stuff, and the more they are taught to avoid gaming the system, the better Google's index becomes, and the more money it makes. Google actually needs people to be good at this stuff, so it tries its best to help them.

In addition to the Webmaster Central blog and the SEO Guide, I highly recommend the blog of Matt Cutts from their anti-spam team. (He's currently carrying out an experiment by moving his blog wholesale to a new domain to see how it affects his position in search results, so don't be surprised if that link redirects you to dullest.com instead.)

Well, firstly the site it hosted in Drupal, which has it's own SEO module check off list, which I have followed to the letter. I have also read ALL of googles advisory. I have created a sitemap, and notified google to come and crawl the site which it has done, twice.

It still lists the keywords from the old site, even though the new site has been live for two days and been crawled twice. There are no crawler errors. Hence the thread, googlebot is p*ssing me off, I have done everything by the book and yet I am not any further forward.

[AtW]

Appearing in searches for particular keywords is ranking and it has not much to do with googlebot (unless your site fails or can't be parsed etc).

You are probably in a competitive category and in this case in order to go on top you need quality relevant backlinks with good anchor text.

[TheRefactornator]

It still lists the keywords from the old site, even though the new site has been live for two days and been crawled twice.

Two days is not very long and the Google indexing algorithm takes a while for changes to be applied and actually appear in the SERPs. Rest assured that Googlebot has seen your new content and posted it to the indexer for update..it just doesn't quite "believe" the changes are permanent yet so they are effectively sand boxed until it believes your content is stable again. Even when this time has passed you might still need additional effort to get to page 1 of the natural SERPs.

Important sites are serviced much more quickly because of their "importance" within the Google PR system which is why new items on news sites and blogs with high PR are indexed very quickly into the SERPs.