Big Data

[GB9]

Anyway, it's the Predictive Analytics that gives the value.

Clustering such as K-means, dimensionality reduction (PCA anyone?) and regression that provides the value. Anyone doing PCA using SQL Server? What about Bayes inference of future trends?

Anyone?

Right. So you have come up with some genuine big (volume) examples. Thank you. I'll throw in online gaming analytics and NASA spacex for good measure.

However, I stand by my point that these are the exception not the norm. Your average company doesn't need anything like these examples. The company I mentioned before that used sql server was one of the country's top online retailers and it had no need for big data solutions.

As for predictive analytics, again very specialist. I've done s load of stuff previously with insurers and pharmaceuticals and they managed well with SAS.

In memory is my current pain point.

[AtW]

CUK got Big Data - TykeMerc's sockie reports

[DimPrawn]

The last bit is true. Fred's Fish and Chip Shop doesn't need it for sure. Government, military, big global companies do.

E.g. analyse every cell phone conversation on the planet. Speech recognition in every known language, and then looks for trends in terms of military threats.

Look for patterns, predict likely probabilities. In real time.

Then correlate with every CCTV video feed in the world, and every GPS tracking in every phone in the World, with voice recognition to every person in the World, in realtime.

Then predict what they are thinking, and arrest them for crimes they have hardly even thought about committing yet.

[GB9]

Then correlate with every CCTV video feed in the world, and every GPS tracking in every phone in the World, with voice recognition to every person in the World, in realtime.

Then predict what they are thinking, and arrest them for crimes they have hardly even thought about committing yet.

Sounds like you could make a film out of that.

And still isn't what your average company does.

[DimPrawn]

Sounds like you could make a film out of that.

And still isn't what your average company does.

No, 99% of companies aren't there yet. But Predictive Analytics is used by big retailers, insurance companies, financial services, medical companies, even if they don't call it Big Data.

Shoot marketing people, they are @rseholes.

[OwlHoot]

... and pharmaceuticals and they managed well with SAS. ...

Oh dear, I suspect this thread will soon be taking a turn for the worse. ...

[DimPrawn]

Oh dear, I suspect this thread will soon be taking a turn for the worse. ...

I'm logging in now....you lot are thick as mince

[oracleslave]

That's where you need a data scientist and statistician.

You mean like a sasguru?

[DimPrawn]

You mean like a sasguru?

For £2K a day, he's worth every penny.

[jamesbrown]

Yes, I work with Big Data pretty regularly. Currently looking at some model outputs that are accruing at ~60TB per day, with retrospective runs on the order of several PB. However, none of this involves DBs, other than for metadata. The models run on supercomputers and the analysis is mainly there, but the less challenging stuff is done via distributed computing with a mix of Java and R (using Spark etc.).