I am sure most of us would have heard of the term “Big Data”. But do we all really know what Big Data is. For most of us it is just a term which signifies a “lot of data”. But Big Data has lot more than just “lot of data”. Big data is a buzzword which is used to describe massive amount of data (both structured and unstructured data). This data is so huge that it is difficult to process it using traditional database and software techniques.
Now some of you might be wondering how Big Data is different from another famous Buzz word “Business Intelligence”. Well, 10 years back data was not as massive as it is today. Thus conventional techniques such as querying and reporting (Internal data) formed a major part of Business Intelligence. But now the world is moving into Web and Social Media where we have thousands of data majorly in the external form which has no defined structure. Processing this data using the conventional Business Intelligence techniques is not possible. This external data which is very massive forms a part of Big Data. The science of pre-processing, storing, analyzing, and predicting patterns is called “Data Science” or "Business Analytics".
After giving a broad picture of what Big data is, the next question which arises is how is Big Data useful. Who are the users, the business needs, its applications in real world. As mentioned in the previous paragraph, data is increasing massively. Companies like Google who is leading in the Search Engine market deal with large amount of data on the web. To query and provide search results against petabytes (1,024 terabytes) or exabytes of data quickly and efficiently, a lot of intelligence needs to be applied. Google came up with very useful algorithms such as Hadoop, Map Reduce and its variations to manage their data. This process is continuous and challenging where the algorithms has to be updated to manage the exponentially growing data.
Apart from Google, there are other various other companies who are moving towards Big Data. The Health care industry demands maintaining large amount data consisting of doctor information, patient records and the insurance details. A simple conventional database will not suffice the purpose. To increase revenue, these companies are trying to predict various patterns on the diseases which can possibly occur and the required prevention to be taken using Data Science techniques. Even companies such as Amazon, eBay, PayPal who are into e-commerce are using Big Data techniques to improve their business. Th recommendations which appear after one purchases a product from these e-commerce sites are examples of how Big Data is being used. Apart from web based companies, other companies who had their focus on standalone applications are moving towards Big Data and Analytics. The best example for this would be Adobe Systems who had their business focused on Flash and Flex during 2010. But now even they are moving towards Big Data industry capitalizing on Digital Marketing (Acquisition of Omniture in 2009) and using all their applications ( Adobe Reader, Adobe Photoshop ) on Creative Cloud.
We have got a general idea of what Big Data is and how it is used in today's world. But, the knowledge one posses on data can never be complete as data is growing endlessly. It is a very interesting and challenging space to conquer. Stay tuned to my next post where I will discuss more about Big Data and its related applications.