Reply to: Technical challenge

[AtW]

Keep faith old chap, fortune favours prepared minds.

[MarillionFan]

Very good AtW. Its all very good, but so far your search engine returns even less matches than www.onlyscanned5sites.com

[AtW]

Ok, a bit disappointing not to see any estimates, but I had pretty low expectations in the first place.

Anyway, here are the results of benchmarking (on AMD Athlon x2 3800 - single core used):

0) no caching tricks - search pattern is expected to be evenly distributed
1) 100 mln unique strings of 20 bytes each, data size: 2 GB
2) Indexing takes 30 mins
3) Generated index size is ~810 MB.
4) Running searches for 10,000 randomly selected strings, with 100 runs (total 10 mln searches) results in a sustained performance of ~232,000 (that's thousands) searches per second.

The system supports multiple indices, so it's not like everything is done for 100 mln strings: scalability is perfect because by adding 2nd CPU you will get double the speed, same goes with extra machines which will not offer extra speed but also load balancing and redundancy.

Could you do better than that? I doubt it. Not because I think I am so ****ing amazing, but because low level algorithms used in the system are such that mathematically pretty much perfect: you can't cheat them with probabilistic modelling because searches will be evenly distributed so caching tricks are off.

[AtW]

Provided it's on a PDP11/44.

Go on then - so far I have not heard better offers and I am about to benchmark my stuff at 100 mln unique strings index, we shall see now if theoretical calculations were correct...

[xoggoth]

I used to be a wiz at estimating hardware requirements in my sales support days so I can help. Provided it's on a PDP11/44.

[TheOmegaMan]

Benchmarking code is done, generating 100 mln unique strings index,actually slightly higher as the app need to remove duplicates that may or may not be present, but in the original task I thought to go easy on you

Hey AtW I can do it 10% quicker than your application. How do I know ? Easy, I am a fking genious and anything I do would be much, much better.

[AtW]

Benchmarking code is done, generating 100 mln unique strings index,actually slightly higher as the app need to remove duplicates that may or may not be present, but in the original task I thought to go easy on you

[darmstadt]

Have a look at the operating system TPF to do this or IMS running under zOS on a z9 EC 54 way. These systems do these type of things 24/7.

[sasguru]

At this point I gave up. There is no point in me living any more. Some fat moustachiod Russian immigrant in a bedsit in Birmingham has the answer to all the worlds problems and it is impossible for anyone to be better than him.

If only I could have such a great life.

Next ....

[AtW]

I will have my benchmarks in just a few minutes, hurry up with your own estimates because I will actually get it all working

[PRC1964]

don't worry about getting non-optimal answer as it is pretty much impossible to get it done better than my approach.

At this point I gave up. There is no point in me living any more. Some fat moustachiod Russian immigrant in a bedsit in Birmingham has the answer to all the worlds problems and it is impossible for anyone to be better than him.

If only I could have such a great life.

[threaded]

All very well AtW, wondering about the system, but you ought to know: most of us here could've got the client to pay us more to do the same thing.

threaded in "and here endeth the first lesson" mode

[AtW]

There is no current system - new system developed by me is totally brand new, which is in any case irrelevant - what is relevant is that you know what the system should do and I want your own estimate on the basis of your own experience.

How would you even approach this problem - use database?

Anything goes - don't worry about getting non-optimal answer as it is pretty much impossible to get it done better than my approach.

[DaveB]

ok, here is what you have: 100 mln unique strings - 20 bytes each

Your job is to create a system that would allow to either confirm that a given string does not exist in the list of those unique strings you have, or, if it does exist, return unique numeric ID that you can associate with each of those unique strings - database equivalent of RowID.

In terms of performance the system should allow for at least 50 searches per second, or in other words one search should not take more than 0.020 sec.

How fast my system performs (writing benchmark code now) is irrelevant since I want to know what kind system would YOU need to build in order to get around that performance. It does not have to be exact, say 10 servers with X GB ram and Y CPUs each using Oracle etc: estimated cost £Z.

MarillionFan: if you can't give me estimate of work then you ain't getting contract in the first place because clearly you would not have a clue how to do execute it in a way that puts my interests first, ie: very cost effective solution.

Current performance is very relavent since we have no idea of what the software is capable of on it's current platform so we have no indication of the loads it is likely to put on the hardware.

If you are getting X% of your desired performance on a particular hardware platform then we can start to predict performance on other configurations taking into account OS overheads, disk speeds etc.

Is the idea of this excersise to show how much cheaper your solution is to an oracle based one?

Anyway, I'm of to do something more interesting with my free time than take on unpaid systems design work

[sasguru]

ok, here is what you have: 100 mln unique strings - 20 bytes each

Your job is to create a system that would allow to either confirm that a given string does not exist in the list of those unique strings you have, or, if it does exist, return unique numeric ID that you can associate with each of those unique strings - database equivalent of RowID.

In terms of performance the system should allow for at least 50 searches per second, or in other words one search should not take more than 0.020 sec.

How fast my system performs (writing benchmark code now) is irrelevant since I want to know what kind system would YOU need to build in order to get around that performance. It does not have to be exact, say 10 servers with X GB ram and Y CPUs each using Oracle etc: estimated cost £Z.

MarillionFan: if you can't give me estimate of work then you ain't getting contract in the first place because clearly you would not have a clue how to do execute it in a way that puts my interests first, ie: very cost effective solution.

Remember what I said earlier about Aspergers Syndrome? This is a classic symptom, not having the basic empathy to understand that your post is quite probably the most boring ever to be posted on CUK ...

If Carlsberg did boring threads, this would be it ...