Website crawling

Visitors can check out the Forum FAQ by clicking this link. You have to register before you can post: click the REGISTER link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. View our Forum Privacy Policy.
Want to receive the latest contracting news and advice straight to your inbox? Sign up to the ContractorUK newsletter here. Every sign up will also be entered into a draw to WIN £100 Amazon vouchers!

You are not logged in or you do not have permission to access this page. This could be due to one of several reasons:

You are not logged in. If you are already registered, fill in the form below to log in, or follow the "Sign Up" link to register a new account.
You may not have sufficient privileges to access this page. Are you trying to edit someone else's post, access administrative features or some other privileged system?
If you are trying to post, the administrator may have disabled your account, or it may be awaiting activation.

zerointeractive replied

3 October 2013, 18:14
Dapper: The Data Mapper
free and does what you need
Leave a comment:
bluelogic replied

20 September 2013, 12:21
Download the Microsoft SEO Toolkit... its free and will do what you need and more.
Leave a comment:
Bunk replied

5 September 2013, 13:20
YQL?
Leave a comment:
northernladuk replied

5 September 2013, 13:19
'A friend' used a product from Lencom Software to do this with great success. I can't remember the exact name of the tool but looking at their offerings they have one called Visual Web Task which looks like it does the job. If it was the same one I used it worked very well pulling data and pictures from a page.

Lencom Software Inc Software Informer: Latest Lencom Software Inc software updates and reviews: Fast Email Extractor,...
Leave a comment:
stek replied

5 September 2013, 13:03
wget or curl
Leave a comment:
RasputinDude started a topic Website crawling

5 September 2013, 13:03
Website crawling

I need to get some data from a web site that has a report with a fixed maximum of 20 lines per page. There are about 20k pages.

I need a tool that will download the data into some format - CSV, MySQL, I don't care - and then autmatically follow the link to the next page and repeat.

I could write one, but has anyone used a tool that does this?

Going straight to the database is not an option, I'm afraid. This is the only way to the data.
Tags: None