Tool to inspect a website structure?

[NickFitz]

That is the question being asked. When a new site goes up Google finds it and crawls the home-page... how does it find the home-page in the first place?

I always thought if I put up a page mysite.com/some_random_page.html, Google would find it and index it even if my homepage doesn't link to it. Not the case?

Assuming the site is at a newly-registered domain, Google's nameservers will find out about the new domain and tell the spider to go and have a look.

Other than that, as PAH and NLUK have said, it's just a question of following links.

However don't thereby start to believe that putting a page/file on a server and not linking to it is a good way of keeping it secure from prying eyes. There are a number of ways things can end up being accidentally linked to. For example, it's not unknown for server logs to accidentally be made available at an unsecured URL...

[TheFaQQer]

However don't thereby start to believe that putting a page/file on a server and not linking to it is a good way of keeping it secure from prying eyes. There are a number of ways things can end up being accidentally linked to. For example, it's not unknown for server logs to accidentally be made available at an unsecured URL...

There was a great article in the most recent 2600 magazine about how people edit files on the server, and the text editor automatically creates a backup including a ~ at the end of the extension.

So, if you do a search by filetype on Google, you can easily find (for example) sites which have *.php~ files. Which won't get executed as php, and will expose the contents to anyone that looks.

If you do a search for "wp-config.php~" I reckon you could quite easily find the database connection and password for quite a few Wordpress blogs out there.....

[northernladuk]

Not completely related but anyone read the investigation of how they caught the Facebook worm Koobface creators? Much of that was from info left on servers..

Very interesting... if you like that type of thing...

The Koobface malware gang – exposed! | Naked Security

[Sysman]

For example, it's not unknown for server logs to accidentally be made available at an unsecured URL...

I've come across a few folks who willingly publish their web stats without massaging them first. It doesn't take much imagination to realise that a supposedly hidden URL could pop up in those stats.