Want a database for completed auction sales.

[newbury]

I'm a dedicated follower of several auction sites for personal use. One of them, Publicsurplus has feature that allows me to "save" an auction listing.
Thus I have an online "database" dating back to March 3013 of items I've been seriously looking at. This had proved very valuable because it allows me to look back and compare items like tractor implements that have been sold over the years.

I'd like to be able to grab the listings and put them in a database do I can easily refer to them later.
I want to capture things like auction date, description, final price, successful bidder, location and most important the pictures of the equipment.

I used to program a bit in dbaseII then dBASEIII, but that was a long time ago. I might be able to gin up something in Open Office but if there is something out there already built it would be a lot quicker.

So does anyone know of something out there or some links to where I could get more pointers on how to build one?

 

[quicksandfarmer]

Since you used dBase I'll assume you're a PC guy. Have you looked at Visual Basic? The Express version is free, Basic is pretty easy to learn, and it includes built-in objects for retrieving files from the internet and storing data locally. It also interfaces with Microsoft Excel if you want to go that way.

Here's how I would attack it:
Give your program the URL of the auction site. Have it pull down the page you want in HTML. (built in VB functionality). Parse it as XML into an internal table (built-in). Go through that table and pull out the info you want. If you trust the site you only have to pull the links; if you want to keep the underlying data on your own computer pull down the linked pages and store them. To store the data, you could either write it out to disk as xml, or into Excel (or both, Excel can read/write XML).

Pretty much all of the above can be done in any modern programming language.

Note that you're almost certainly violating the terms of use and copyright of the site. They may detect you downloading lots of pages and investigate.