• Want to receive the latest contracting news and advice straight to your inbox? Sign up to the ContractorUK newsletter here. Every sign up will also be entered into a draw to WIN £100 Amazon vouchers!

Website crawling

RasputinDude

Fingers like lightning
Joined
Feb 19, 2009
Messages
603
Likes received
54
Thanks received
9
I need to get some data from a web site that has a report with a fixed maximum of 20 lines per page. There are about 20k pages.

I need a tool that will download the data into some format - CSV, MySQL, I don't care - and then autmatically follow the link to the next page and repeat.

I could write one, but has anyone used a tool that does this?

Going straight to the database is not an option, I'm afraid. This is the only way to the data.
 
Download the Microsoft SEO Toolkit... its free and will do what you need and more.
 

Advertisers

Back
Top