Website crawling

[RasputinDude]

I need to get some data from a web site that has a report with a fixed maximum of 20 lines per page. There are about 20k pages.

I need a tool that will download the data into some format - CSV, MySQL, I don't care - and then autmatically follow the link to the next page and repeat.

I could write one, but has anyone used a tool that does this?

Going straight to the database is not an option, I'm afraid. This is the only way to the data.

 

[stek]

wget or curl

 

[northernladuk]

'A friend' used a product from Lencom Software to do this with great success. I can't remember the exact name of the tool but looking at their offerings they have one called Visual Web Task which looks like it does the job. If it was the same one I used it worked very well pulling data and pictures from a page.

Lencom Software Inc Software Informer: Latest Lencom Software Inc software updates and reviews: Fast Email Extractor,...

 

[Bunk]

YQL?

 

[bluelogic]

Download the Microsoft SEO Toolkit... its free and will do what you need and more.

 

[zerointeractive]

Dapper: The Data Mapper
free and does what you need