Visitors can check out the Forum FAQ by clicking this link. You have to register before you can post: click the REGISTER link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. View our Forum Privacy Policy.
Want to receive the latest contracting news and advice straight to your inbox? Sign up to the ContractorUK newsletter here. Every sign up will also be entered into a draw to WIN £100 Amazon vouchers!
Webby I demand you end this madness, its wasting bandwith.
Are you addressing me?
I wondered how long it would be before somebody would pop up and demonstrate their complete lack of understanding of how the web works
To start with: the spider isn't running at the moment, so it isn't using any bandwidth anyway.
When it was running "intensively" (which it wasn't really) it was over a Bank Holiday weekend (deliberately) when few people were viewing the site. To be honest, I could run it at the busiest time and it would make no difference.
When it's running it only requests the HTML, which is around 16K for an average page - all the additional bandwidth used by web browsers, downloading images, CSS and scripts, isn't necessary. This saves around 49 HTTP request per page over the amount your browser makes (this varies depending on the number of smilies on the page, and whether you have avatars turned on).
This amounts to about 190K a page that it doesn't request, which probably drops to around 12K if your browser has effective caching - although each of the 49 request is still made, only to get a "not-modified" response, and it's HTTP requests that slow down a server, not serving data.
When you made your post, around 100K of extra scripts and CSS were loaded by your browser, and 138K of traffic for the smiley gallery. Therefore, your post was the equivalent of around 16 pages worth of indexing. And another four or five for reading the page you were replying to. And for every page you read. And every post you make.
If there was the slightest possibility that my spider would have a harmful effect on the site I have no doubt the administrators would have been on to me to ask me to stop. (If there were the possibility, I wouldn't have written it in the first place.) I daresay they also realise that 16K here and there, in a single HTTP request/response cycle, is so unutterably trivial to a web server that it's not worth bothering about.
Google probably uses more of this site's bandwidth each day than I would consume in a week, as it looks at everything, and revisits everything, not just looking for the last page on one thread.
Now go back to your database, and leave us "webbys" to build the future for you
Comment