Big Data

[DimPrawn]

Yes, I work with Big Data pretty regularly. Currently looking at some model outputs that are accruing at ~60TB per day, with retrospective runs on the order of several PB. However, none of this involves DBs, other than for metadata. The models run on supercomputers and the analysis is mainly there, but the less challenging stuff is done via distributed computing with a mix of Java and R (using Spark etc.).

Yes, that's Big Data. It's got nothing whatsoever to do with databases, it's map reduce, distributed computing, predictive analytics, data science. It's not a SQL Server DB.

[PurpleGorilla]

That's not big data. That's analysing structured data in a DW.

Big data is analysing something like every tweet in realtime and obtaining understanding of how the World sees your products.

BTW there's 9100 tweets per second to analyse, 24/7/365. Try that in your SQL server DB.

So the NSA then.

[AtW]

That's not big data. That's analysing structured data in a DW.
Big data is analysing something like every tweet in realtime and obtaining understanding of how the World sees your products.
BTW there's 9100 tweets per second to analyse, 24/7/365. Try that in your SQL server DB.

Crawling and analysing 81000 web pages (each 100 times bigger than max tweet) per second here ....

[jamesbrown]

Yes, that's Big Data. It's got nothing whatsoever to do with databases, it's map reduce, distributed computing, predictive analytics, data science. It's not a SQL Server DB.

Absolutely. I'm not aware of any DBs that could handle this, although DBs aren't my area (we simply don't use them, except for metadata). I do the modeling/analytics. As I say, distributed computing is used quite a lot, but all of the models themselves (and some of the analytics) run on supercomputers, with an intermediary swap (disk) space for analytics, and then tapes for archiving.

[MarillionFan]

We use SAP Hana for data collection. We're tracking usage against 5000 cloud instances, who's inserted, deleted records etc, growth of application usage.

This is all then tied back using Licensing decoder I designed to make sense of it to the business. Now using for product recommendations.

But it's all business SME plus data scientists. We're hampered by the fact that IT decided to chuck the data into SAP Hana in a more unstructured manner as they reckoned it was so quick that it would handle unstructured data for analysis.

Guess what they were wrong and now having to build out a proper set of star schemas with additional
Business input.

Most companies don't need it but call it out as Big data.

[MarillionFan]

Contract I have now taken is a Big Data project. There are going to be some quite useful use cases at clientco, but initially they're buying the kit thinking it will do every form of operational, management & analytical reporting they want (including financial) just by chucking the data in.

I am still skeptical, but it's a greenfield site so will have a certain free hand. I shall report back on any real use cases over classic BI.

[original PM]

Contract I have now taken is a Big Data project. There are going to be some quite useful use cases at clientco, but initially they're buying the kit thinking it will do every form of operational, management & analytical reporting they want (including financial) just by chucking the data in.

I am still skeptical, but it's a greenfield site so will have a certain free hand. I shall report back on any real use cases over classic BI.

Please do as I think we are going to be in the same boat soon.

[jamesbrown]

Contract I have now taken is a Big Data project. There are going to be some quite useful use cases at clientco, but initially they're buying the kit thinking it will do every form of operational, management & analytical reporting they want (including financial) just by chucking the data in.

I am still skeptical, but it's a greenfield site so will have a certain free hand. I shall report back on any real use cases over classic BI.

Sounds familiar. When you buy a new hammer, everything starts to look like a nail, including your finger. What sort of kit are they buying? Hopefully starting sensibly and then scaling...

[portseven]

Been on a Big Data project for the last year, just think of it as a huge bucket. Great at having lots of data in one place that you can run analysis against, because its distributed compute, parallelism jobs are the way to go.

Good use case is machine logs, chuck them all in there, and analyise later.

[GB9]

So what is the grain you guys are capturing at?