Reply to: Microsoft Damned, AtW Praised

[NickFitz]

Majestic 12(www.majestic12.co.uk/bot.php) appears in the stats for my website as one of the 'referrers'. Surprisingly my website gets more referals from thickmom.com though (DO NOT VISIT THAT SITE AT WORK, MILFS AND LARGE RACKS DISPLAYED), and have no idea why. I wonder if sites ping other sites for advertising reasons? I've looked on that site (well someone had to do it ) to see if a link to my website exists but it doesn't and see no reason why it would - just large racks are shown

Google Webmaster Tools will allow you to find inbound links. If you have a Google account for GMail or some such you can log in with that, associate the domain with your account, put a file with the provided name in the root of the site to verify your ownership, and then you'll have access to all kinds of useful information about how Google sees your site. (The file isn't a script or anything - there just has to be a file with the name provided to prove you have control of the content on that site.)

Also, check to see if there's a specific file or files that the referring requests are hitting: referrer headers (strictly speaking "referer" because TBL can't spell) are sent for any request originating from a page, not just when somebody follows a link. It could be that somebody has deep-linked to an image file or a script file (even some people who should know better are too lazy to simply put a copy of a script on their own server). If so, you can prevent them wasting your bandwidth by renaming the relevant file and leaving it to 404 for the leecher.

A friend of mine who runs a photography site often gets people deep-linking to images on his site from eBay listings. His usual practice is to change his site to use a renamed copy of the original, then replace the original with an image saying "Item withdrawn from sale" in big red letters

[AtW]

if there are no external links to the inner pages, they'll never find the rest of the site.

WTCS

[NickFitz]

Nick,

What determines whether a site can be searched via the "site:sitename" construct? I ask because some sites don't give any results.

I'm not the chimp, and he may be along soon to correct me, but as far as I know any site can be searched using site:sitename. If you're not getting any results then it's probably because Google haven't indexed the pages for some reason. Maybe they're hidden behind a form, maybe Google is blocked in the robots file (not sure why they would do that). Do the pages appear in normal search results without specifying the site?

WHS

At one place I worked they had a site they'd done for a client which simply didn't show up in Google. The site had a Flash intro (I hate those things, but the client loved it ). Upon investigation, I found that my illustrious predecessor hadn't included a "Skip this" link in the HTML - well, he had, but for some reason it was commented out. I enabled the link and, within a couple of days, the client started appearing as expected in relevant searches.

So, if the site has a splash page, make sure there's a link to the actual content in the HTML. Otherwise the bots have nowhere to go when they hit that domain, and if there are no external links to the inner pages, they'll never find the rest of the site.

[SuperZ]

Majestic 12(www.majestic12.co.uk/bot.php) appears in the stats for my website as one of the 'referrers'. Surprisingly my website gets more referals from thickmom.com though (DO NOT VISIT THAT SITE AT WORK, MILFS AND LARGE RACKS DISPLAYED), and have no idea why. I wonder if sites ping other sites for advertising reasons? I've looked on that site (well someone had to do it ) to see if a link to my website exists but it doesn't and see no reason why it would - just large racks are shown

[Drewster]

I think Google likes sites with lots of new posts, especially when they contain plenty of URLs pointing elsewhere.

All hail "One-a-day"

[Bunk]

Nick,

What determines whether a site can be searched via the "site:sitename" construct? I ask because some sites don't give any results.

I'm not the chimp, and he may be along soon to correct me, but as far as I know any site can be searched using site:sitename. If you're not getting any results then it's probably because Google haven't indexed the pages for some reason. Maybe they're hidden behind a form, maybe Google is blocked in the robots file (not sure why they would do that). Do the pages appear in normal search results without specifying the site?

[Sysman]

It appears that we are on something like a twenty minute refresh cycle at the moment, as my ten-minute search returned no results. EDIT: No, looks like twenty, not fifteen.

However the Daily Fail seems to be getting indexed as frequently as every five minutes

Nick,

What determines whether a site can be searched via the "site:sitename" construct? I ask because some sites don't give any results.

[Fred Bloggs]

I have been amazed at how quickly Google indexes threads from here within minutes of them first appearing. CUK must be doing something right versus Google.

[AtW]

Which bit?

All the bytes mate!

P.S. This post was made under the influence of Veuve Clicquot that was used in pre-celebration of upcoming imminent SKA news...

[NickFitz]

That's a load of drivel...

Which bit?

[AtW]

That's a load of drivel...

[Bunk]

Just because it says .HTML doesn't mean it's not something like php or asp generating it.

Yep, it's still a PHP page. The URL is being rewritten so that http://forums.contractoruk.com/gener...praised-3.html actually hits something like (and I'm guessing here) http://forums.contractoruk.com/view.php?topic=51259. The "microsoft-damned-atw-praised-3" gets thrown away, the 51259 gets grabbed and used for the real URL. Have a look at Apache's mod_rewrite or something similar.

Edit: Yep, it's definitely not that URL but you get the idea hopefully.

[NickFitz]

And with no php? or asp? but looks like a link to a static HTML page:

http://forums.contractoruk.com/gener...praised-2.html

I guess a static HTML page that's often updated must rate higher than any kind of script generated content.

I think Google's algorithms used to draw some distinction between dynamic and static content on the basis of the URL some years ago, but it doesn't have anything like as much relevance now, as they've tuned things to use other heuristics for determining what's likely to change and what isn't.

For example, it used to be worthwhile avoiding URLs with a query string, but that advice fell by the wayside quite a long time ago.

FWIW, the vBulletin software powering CUK is written in PHP.

[oversteer]

Just because it says .HTML doesn't mean it's not something like php or asp generating it.

[VectraMan]

I think there's an SEO friendly URLs addon of some description, so that you get the thread title in the URL, instead something like ?topic=51259.

And with no php? or asp? but looks like a link to a static HTML page:

http://forums.contractoruk.com/gener...praised-2.html

I guess a static HTML page that's often updated must rate higher than any kind of script generated content.