INSITE Big Data Symposium - Wonderful Learning Experience !!

I can't believe that I am already 8 weeks into the semester. I must say time files very fast. Business Intelligence course has been a really good learning experience with engaging lectures by Dr. Sudha Ram. The mention of Big Data symposium was made on the first day of class itself. I was looking forward to it since then and the day (October 10th 2013) had finally arrived. It was a nice and bright Thursday morning. I reached the UofA student Union - south ballroom fifteen minutes before 8 AM. Before entering,  I was handed over my name badge and the agenda for the day. It was a really long agenda with several talented and erudite speakers from big companies such as IBM, SAP, Macy's etc. I was eagerly looking forward to their sessions.



The sessions started with Opening remarks by our Professor Dr. Sudha Ram where the speakers were introduced and the major goal of the symposium was addressed. The first session was by Brain Gentile where he spoke about the rise of Big Data, the myths regarding Big Data and Big Data Transformation. Certain concepts introduced such as 4th V or veracity with respect to data were extremely interesting and intriguing. The wide range of applications which JasperSoft offers was also another nice learning. The second session was also yet another interesting session by Brenda Dietrich from IBM where the term analytics was expressed seamlessly. After a short break, session on how Big Data is used to make better business decisions was taken. The speakers were Darren Stoll and Kerem Tomak from Macy's. Several novel concepts such as Big Data Ecosystem layers were discussed in detail and how they can be used to make better business decisions. The concept where Analytics is the pillar of business was emphasized by giving real-time examples. Finally, Tim Hood from SAP presented on the SAP HANA tool and its uses before the lunch break. The entire morning was a fruitful one where I learnt a lot of novel and intriguing concepts. I was looking forward for more insightful presentations.

The afternoon sessions were equally interesting and useful. Extremely important applications of Big Data such as how Big Data is used to solve security concerns were discussed by David Cowart. These concepts and ideas were new to me and it was a very good learning experience. After security, applications of Big Data in Healthcare industry were discussed. Healthcare has always been my passion and it was amazing to learn how Big Data was used to predict patterns and detect outliers from clusters. After security and Healthcare, applications of Big Data in dynamic pricing of tickets were discussed by Zaheer Benjamin. Finally the sessions ended with a extremely flawless presentation by our professor Dr. Sudha Ram who discussed Big Data research being performed in the university using smart card. The Visuals created were extremely appalling and inspiring. The day ended with a very good closing note and it was truly a extremely informative session.

Lessons learnt from the symposium

• Big Data is not structured or unstructured. Big Data can have more than one type of data which consists of structured, semi-structured and un-structured data. This can be referred to Multi-structured data
• Big Data is more than data from Social Media. The classic factors of production entitled land, labor and capital. But today's world, Time and Speed constitutes a major portion
• Big Data is undergoing a series of transformations where the focus is moving towards predictive analysis, 100% of the users being data users, data being controlled by systems where concerns regarding privacy may arise and cost will be extremely low
• The fact of external data being more prevalent in today's world the concept of internal data does not cease to exist. It has its own importance
• Data to be used to take business decisions should be real-time or the time taken to execute the data should be minimized
• The major reason for the emergence of Big Data is lower costs with increased efficiency
• Data is not useful unless there is a clear goal on how the data should be utilized. The concept of using Big Data against a framework where the customer focus is achieved is encouraged
• Big Data serves as a Glue different parts of the organization which enables to perform better execution

The symposium was indeed a great learning experience. I would like to thank my professor Dr. Sudha Ram for organizing this symposium and imparting great knowledge of  today's view of Big Data. 

Applications of BigData in Health Care Industry

In my previous post, I discussed about Big Data and how it is being used in today’s world. In this post, I will discuss more on how Big Data is being used in a particular industry such as healthcare. These days terms such as NoSQL, MongoDB, CouchDB are being very frequently used. Have many of us wondered why they are being used or what are its benefits? As discussed in my last post, data from external sources such as social media data is becoming prominent rather than the internal data which companies used to rely on 10 years back. It is a challenging task to restructure the database schema and data warehouse to fit the external data. It is not a feasible option as well since external data comes from various sources and they do not have any specific format. How can this data be tracked and stored to analyze results? To store these kind of data, we require a non-relational database.

The terms MongoDB, CouchDb are types of non-relational database which does not have the relational structure. Healthcare is a major industry which uses BigData and the necessary applications to track and analyze patient records. While many companies use Data warehouse and star schemas to perform prediction and reporting, there are few who use the traditional NoSql databases. One example to illustrate this point is the use of MUMPS database in IT Healthcare companies such as Epic Systems. 

MUMPS database is a traditional database which was used back in 1950. Its prevalence was lost because of the invention of SQL and RDBMS. But recently since external data is increasing these traditional sources have gained their importance. It is a hierarchical database unlike the relational database. It is very useful in the case of healthcare industry since it helps them to maintain the data efficiently without placing a constraint that the database should be in 3NF. To explain this point more clearly, if the hospital wants to track the behavior of patients along with their frequency and cause of visits. Using the relational schema, we could have a relationship between patient and visit. Each visit is related to a procedure. It is possible that a patient can have more than one visits and each visit can have multiple procedures. Maintaing these in a relational database will require the tables to be in the 3rd Normal form. Because of this constraint, it becomes difficult to maintain multiple details such as phone numbers of the patients. Creating separate attributes for the phone numbers can solve the issue but it becomes a cumbersome task maintaining the NULL values for phone numbers (patients who do not have multiple phone numbers). Creating a hierarchical database can be a better option since it does not require the tables to be in 1NF.  

This post just gave a brief overview of why traditional databases such as MUMPS are being widely used in the healthcare industry and their use.  In my following posts, I will discuss more on Big Data, Business Intelligence and their applications. 

References

http://healthcaresecurity.wordpress.com/2010/10/12/why-are-hierarchical-databases-like-mumps-still-popular-in-healthcare/

BIG DATA - "The buzzword"

I am sure most of us would have heard of the term “Big Data”. But do we all really know what Big Data is. For most of us it is just a term which signifies a “lot of data”. But Big Data has lot more than just “lot of data”. Big data is a buzzword which is used to describe massive amount of data (both structured and unstructured data). This data is so huge that it is difficult to process it using traditional database and software techniques.

Now some of you might be wondering how Big Data is different from another famous Buzz word “Business Intelligence”. Well, 10 years back data was not as massive as it is today. Thus conventional techniques such as querying and reporting (Internal data) formed a major part of Business Intelligence. But now the world is moving into Web and Social Media where we have thousands of data majorly in the external form which has no defined structure. Processing this data using the conventional Business Intelligence techniques is not possible. This external data which is very massive forms a part of Big Data. The science of pre-processing, storing, analyzing, and predicting patterns is called “Data Science” or "Business Analytics".

After giving a broad picture of what Big data is, the next question which arises is how is Big Data useful. Who are the users, the business needs, its applications in real world. As mentioned in the previous paragraph, data is increasing massively. Companies like Google who is leading in the Search Engine market deal with large amount of data on the web. To query and provide search results against petabytes (1,024 terabytes)  or exabytes of data quickly and efficiently, a lot of intelligence needs to be applied. Google came up with very useful algorithms such as Hadoop, Map Reduce and its variations to manage their data. This process is continuous and challenging where the algorithms has to be updated to manage the exponentially growing data. 

Apart from Google, there are other various other companies who are moving towards Big Data. The Health care industry demands maintaining large amount data consisting of doctor information, patient records and the insurance details. A simple conventional database will not suffice the purpose. To increase revenue, these companies are trying to predict various patterns on the diseases which can possibly occur and the required prevention to be taken using Data Science techniques. Even companies such as Amazon, eBay, PayPal who are into e-commerce are using Big Data techniques to improve their business. Th recommendations which appear after one purchases a product from these e-commerce sites are examples of how Big Data is being used. Apart from web based companies, other companies who had their focus on standalone applications are moving towards Big Data and Analytics. The best example for this would be Adobe Systems who had their business focused on Flash and Flex during 2010. But now even they are moving towards Big Data industry capitalizing on Digital Marketing (Acquisition of Omniture in 2009) and using all their applications ( Adobe Reader, Adobe Photoshop ) on Creative Cloud.

We have got a general idea of what Big Data is and how it is used in today's world. But, the knowledge one posses on data can never be complete as data is growing endlessly. It is a very interesting and challenging space to conquer. Stay tuned to my next post where I will discuss more about Big Data and its related applications. 

References:

http://www.webopedia.com/TERM/B/big_data.html

http://www.linkedin.com/groups/What-is-difference-between-Busiiness-1814785.S.210009923