Creating a Combined Job Database.

[MarillionFan]

I had asked AtW the Web Guru on this, but he said he was stumped, so anyone cleverer?

If I wanted to populate a SQL Server Job Database I'm creating with a number of automatic feeds from Monster, Jobsite, Jobserve etc (and assuming I define the format) what methods are there? I only tend to use Web Queries from Microsoft so I'm not sure what I should be looking at to make it happen.

Ignoring RSS feeds that may be available how can it be done?

[Durbs]

Why ignoring RSS feeds? thats the way to do it. It'd just be a scheduled Windows Service/Console app to loop through a DB of feed URL's and parse and consume them all in sequence?

Doesn't necessarily have to be RSS format but its certainly a candidate for XML Feed/Scheduled service app?

do you have the feed providers on-board so you can extend the RSS spec with the MarillionJob-A-Tron namespace and get them to feed according to your schema?

[fullyautomatix]

Are you assuming that all the major job boards adhere to some standard web service definition of the current live jobs ? I somehow doubted it.

Aktuellste Jobangebote von JobDirectory.ch von Banken, Finanzinstituten, Versicherungen, Krankenkassen. uses a custom web crawler. And I know of many other job aggregators who just use custom crawlers which are customised for each job board.

[MarillionFan]

Are you assuming that all the major job boards adhere to some standard web service definition of the current live jobs ? I somehow doubted it.

Aktuellste Jobangebote von JobDirectory.ch von Banken, Finanzinstituten, Versicherungen, Krankenkassen. uses a custom web crawler. And I know of many other job aggregators who just use custom crawlers which are customised for each job board.

So thats what Im looking at I think. Is there something I can purchase or use, and then scrape on a site by site basis, or is it something I can script?

[Spacecadet]

So thats what Im looking at I think. Is there something I can purchase or use, and then scrape on a site by site basis, or is it something I can script?

you can use SSIS
A few years ago I did build a custom data source which scraped a web site... i'll see if i can dig it out.

[Scrag Meister]

you can use SSIS
A few years ago I did build a custom data source which scraped a web site... i'll see if i can dig it out, rewrite parts and beat you to market.

ftfy

[vetran]

I'd like a copy of that custom data source if possible please.

Like the idea of a combined job search engine I have a memory there was one out there already?

[Spacecadet]

so many jobs appear on all the job sites anyway you'll just end up with a pile of duplicates

[vetran]

which is handy as you can pick your agency. You will also find each advert may supply more information different rates or summary.

[MarillionFan]

you can use SSIS
A few years ago I did build a custom data source which scraped a web site... i'll see if i can dig it out.

Could you? Thats sounds like the type of thing.

Anyway, the idea would be to scrape/store adverts from across multiple sites. Dedepulicate and actually work out 'real age' on adverts.Then using some kind of additional scoring process, to grade adverts and then only send very specific contract roles to be via SMS or EMAIL(as I tend to ignore email during the day, SMS would be the preferred type. In addition, even getting it to automatically apply for the role.

Because I wouldn't own the data, its not something I would look to commercialise, but I dont see why as part of CUK (if it works) I dont open it up to a number of our users as a more sophisticated alert system, and allow access to the archived data for any that need it.