Memory prices

[PerlOfWisdom]

About 70,000 - the unix words file. Used about 1K per word.

 

[AtW]

About 70,000 - the unix words file. Used about 1K per word.

Ok old chap, I won't be disrespectful to your achievement because I am wiser now, but read the following thread very carefully: technical challenge

I don't want to spoil it for you, but here is the most important last message that I posted there:

Ok, a bit disappointing not to see any estimates, but I had pretty low expectations in the first place.

Anyway, here are the results of benchmarking (on AMD Athlon x2 3800 - single core used):

0) no caching tricks - search pattern is expected to be evenly distributed
1) 100 mln unique strings of 20 bytes each, data size: 2 GB
2) Indexing takes 30 mins
3) Generated index size is ~810 MB.
4) Running searches for 10,000 randomly selected strings, with 100 runs (total 10 mln searches) results in a sustained performance of ~232,000 (that's thousands) searches per second.

The system supports multiple indices, so it's not like everything is done for 100 mln strings: scalability is perfect because by adding 2nd CPU you will get double the speed, same goes with extra machines which will not offer extra speed but also load balancing and redundancy.

Could you do better than that? I doubt it. Not because I think I am so ****ing amazing, but because low level algorithms used in the system are such that mathematically pretty much perfect: you can't cheat them with probabilistic modelling because searches will be evenly distributed so caching tricks are off.

So old chap, you had 70k words and I had 100 mln words - performance is pretty similar, but I had ~1500 (about one thousand five hundred) times more data - algorithm is such that size of word does not matter, could have been 10k bytes each.

Now you do understand that when I buy more RAM I know exactly what I am doing?

 

[TheMonkey]

Bought 8x1 GB PC 3200 DIMMs Crucial branded... from dabs.com, about 10 quid cheaper than I would have gotten them from Crucial itself - they sure do good memory, they are Micron daughter company or something.

Yeah but buying dabs is always a risk. Crucial from Crucial ALWAYS arrives the next morning, even if you order it at 6pm the day before!

 

[MrsGoof]

Yeah but buying dabs is always a risk. Crucial from Crucial ALWAYS arrives the next morning, even if you order it at 6pm the day before!

Amazon can be just as bad. Order something at 7pm for 2 to 5 day free delivery. Then at 8am the next day some bloody courier is banging on the door with the delivery. Selfish buggers!

 

[AtW]

I buy stuff from dabs.com about once a month, I think twice products were faulty and they replaced it all quickly enough for me. They ship next day even though I don't pay extra for it. Prices for disks are very keen too, and in this case crucial RAM was 10 quid cheaper.

 

[Forumbore]

The price of memory these days eh? Its those alzheimer sufferers I feel sorry for

 

[AtW]

I remember time when 1 MB cost £40.

 

[TheMonkey]

I remember time when 1 MB cost £40.

That's pathetic. I remember when 64k EPROMs cost that much, and I'm not THAT old as techies go...

 

[AtW]

I remember when getting 1 concrete block to the pyramids cost 40 slaves.

threaded.

 

[TheMonkey]