I was recently involved in a discussion where old AS/400 based folks were debating the merits of relational database and SQL. It was a bit unbelievable since Dr Codd first published his papers on relational database architecture in 1970, 42 years ago! In the course of discussion the issue of modern alternatives to relational database systems came up.
This brings us into the topic of "Big Data". "Big Data" is defined by Volume, Velocity, and Variety. We are seeing data volumes in Petabytes (1000 terabytes) and Exabytes (1000 petabytes) materializing regularly.
The concept of "Big Data" first materialized in scientific computing circles where measurements in weather, aeronautics (i.e. wind or airflow on an airframe), medicine, and other industrial or scientific applications produced massive amounts of data that needed to be captured and analyzed, often in real time.
We then began to see social media sites like Facebook Google, Amazon, and others create unbelievably huge repositories of data. Facebook reports that it operates over 30,000 computers and captures log files of over 25 terabytes of data each day...
Today we are seeing Big Data move into commercial business enterprises. Consider the following:
- Marketing applications that search the Internet for hits on a product or company name and then collect and analyze the content of everything found to determine the sentiment or attitude of the public toward the product or company.
- Consider huge international retailers who capture and monitor every sales transaction in every store (millions of transactions daily) to analyze product movement and revenue opportunities.
- Consider a major retailer who captures everything that a user does in their e-Commerce web sites, including products they look at but do not buy, searches they perform, every single keystroke and then analyze why visitors behave the way they do...
I would really like your help in pursuing this line of thought and exploring the use of "Big Data" by commercial business enterprises. Send me your experience at: firstname.lastname@example.org. Let me know if I can use your name or company name or if I should keep that info confidential. I'd really love to hear from you.