Reply to: oh dear: Panic selling shuts £2bn fund

[AtW]

You should get the Nobel prize for services to the medical industry atw.

I would if I find a way to cure you from excess sarcasm

Typical of sas not to get real answer

[sasguru]

Problem is chappy, your indian programmer would not know what to do unless you explain him in great detail, and you don't really know what "multiway in-place merging" is, do you? This topic is actually very poorly described on the Internet and books, it took a moment of brilliance for me to actually get it done.

Let me just explain to you chappy - one of my SKA subsystems just finished deduplicating around 300 bln strings (long urls), with mapping relationships (one pointing to the other) between them at around 930 bln. I'd like to see rent-a-coder do it cheaper and faster than me - if they do (with reasonable hardware usage) I might as well admit you are right

OR:

Take your favourite database and create this table:

create table UrlMap
(
TargetURL varchar(255),
SourceURL varchar(255),
Flags int
)

Then load into this table 1 bln (that's 1,000,000,000) of rows, with average URL length of 60 bytes - take a note how long it takes. Then create clustered index on TargetURL - you will be querying on that key. Take a note how long index building takes. Then run a few queries when you are done just to see how long it takes. Then take a note on cost of database and harware you will need to handle at least 10 searches per second.

Then consider that my SKA handled almost 1000 times more in around 8 days on one server

Forget SKA, I'm going to market this post as cure to the insomniacs of the world. You should get the Nobel prize for services to the medical industry atw.

[milanbenes]

very good

Milan.

[AtW]

Almost at that point old chap, almost.

[milanbenes]

Alexey,

we all love your SKA, and want you to succeed,

isn't about time you spent some of your hard earned on the services of a PR Company to do some shouting and get your product out there into the world ?

It's a wonder what a bit of PR and Marketing can achieve.

Milan.

[AtW]

I could hire some Indian programmer to do it for a few pounds.

Problem is chappy, your indian programmer would not know what to do unless you explain him in great detail, and you don't really know what "multiway in-place merging" is, do you? This topic is actually very poorly described on the Internet and books, it took a moment of brilliance for me to actually get it done.

Let me just explain to you chappy - one of my SKA subsystems just finished deduplicating around 300 bln strings (long urls), with mapping relationships (one pointing to the other) between them at around 930 bln. I'd like to see rent-a-coder do it cheaper and faster than me - if they do (with reasonable hardware usage) I might as well admit you are right

OR:

Take your favourite database and create this table:

create table UrlMap
(
TargetURL varchar(255),
SourceURL varchar(255),
Flags int
)

Then load into this table 1 bln (that's 1,000,000,000) of rows, with average URL length of 60 bytes - take a note how long it takes. Then create clustered index on TargetURL - you will be querying on that key. Take a note how long index building takes. Then run a few queries when you are done just to see how long it takes. Then take a note on cost of database and harware you will need to handle at least 10 searches per second.

Then consider that my SKA handled almost 1000 times more in around 8 days on one server

[DiscoStu]

You're right. I'm sure it wouldn't take me more than a few minutes rather than the years it took you, though. And why would I be interested anyway - I could hire some Indian programmer to do it for a few pounds.

Now there's an idea. We have a whip and stick a bid up on rentacoder for a few hundred quid for someone to build something similar to AtW's SKA before he finishes it.

[sasguru]

I moved away from hashing now. I used it initially and improved it considerably, but the volume of data forced me to think of a much more effective approach that I have successfully implemented about 24 months ago. It is the kind of secret sauce that allows me to handle terabytes of data quickly on small scale hardware processing it all in parallel. Here is the new term for you sas: "multiway in-place merging", not that I expect you to understand it, but then again you don't really know how hashing works either.

You're right. I'm sure it wouldn't take me more than a few minutes rather than the years it took you, though. And why would I be interested anyway - I could hire some Indian programmer to do it for a few pounds.

[AtW]

atw just shut it. You're embarassing yourself again. Isn't there some hashing algorithm you can perfect or something, there's a good boy.

I moved away from hashing now. I used it initially and improved it considerably, but the volume of data forced me to think of a much more effective approach that I have successfully implemented about 24 months ago. It is the kind of secret sauce that allows me to handle terabytes of data quickly on small scale hardware processing it all in parallel. Here is the new term for you sas: "multiway in-place merging", not that I expect you to understand it, but then again you don't really know how hashing works either.

[milanbenes]

yep wantacontract,

funny thing is with the property fund, I guess they marketed the fund as being just like holding property and now for the poor investors in the fund it is just like holding property and they can't get out

Milan.

[wantacontract]

ho ho ho...property funds......

milan, thats my attitude towards my investments and pension, i'll rather lose the money myself then have someone else lose the money for me!!

[sasguru]

atw just shut it. You're embarassing yourself again. Isn't there some hashing algorithm you can perfect or something, there's a good boy.

[AtW]

what amazes me is so little responce to this

People are bricking (brick and mortars eh?) it and are in denial.

[AtW]

If a fund can't pay what it is due then it should be put into administration just like any other company - execs go to jail for 20+ years, lawyers who signed them off go for 30, regulators who were responsible for that fund/bank go to jail for 40 years and Gordon Brown goes for 100 to GULAG.

[milanbenes]

what amazes me is so little responce to this

it's because in the UK we've seen all this all our lives and become accustomed to it

you go onto the mainland and tell people that in the UK your pension which you put money into all your life can be pulled from underneath you and they won't believe you !!

I think the Europeans have been molly coddled

Milan.