WWW.KELLYTECHNO.COMPage 1 INTRODUCTION TO HADOOP What is Big Data? Every day, we create 2.5 quintillion bytes of data so much that 90% of the data in the world today has been created in the last two years alone. Gartner defines Big Data as high volume, velocity and variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.According to IBM, 80% of data captured today is unstructured, from sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals, to name a few. All of this unstructured data is Big Data.Big data spans three dimensions: Volume, Velocity and Variety. Volume: Enterprises are awash with ever-growing data of all types, easily amassing terabytes - even petabytes - of information. Turn 12 terabytes of Tweets created each day into improved product sentiment analysis Convert 350 billion annual meter readings to better predict power consumption Velocity: Sometimes 2 minutes is too late. For time-sensitive processes such as catching fraud, big data must be used as it streams into your enterprise in order to maximize its value. Scrutinize 5 million trade events created each day to identify potential fraud Analyze 500 million daily call detail records in real-time to predict customer churn faster Variety: Big data is any type of data - structured and unstructured data such as text, sensor data, audio, video, click streams, log files and more. New insights are found when analyzing these data types together. Monitor 100s of live video feeds from surveillance cameras to target points of interest Exploit the 80% data growth in images, video and documents to improve customer satisfaction What does Hadoop solve? Organizations are discovering that important predictions can be made by sorting through and analyzing Big Data. WWW.KELLYTECHNO.COMPage 2 However, since 80% of this data is "unstructured", it must be formatted (or structured) in a way that that makes it suitable for data mining and subsequent analysis. Hadoop is the core platform for structuring Big Data, and solves the problem of making it useful for analytics purposes. The Importance of Big Data and What You Can Accomplish The real issue is not that you are acquiring large amounts of data. It's what you do with the data that counts. The hopeful vision is that organizations will be able to take data from any source, harness relevant data and analyze it to find answers that enable 1) cost reductions, 2) time reductions, 3) new product development and optimized offerings, and 4) smarter business decision making. For instance, by combining big data and high-powered analytics, it is possible to: Determine root causes of failures, issues and defects in near-real time, potentially saving billions of dollars annually. Optimize routes for many thousands of package delivery vehicles while they are on the road. Analyze millions of SKUs to determine prices that maximize profit and clear inventory. Generate retail coupons at the point of sale based on the customer's current and past purchases. Send tailored recommendations to mobile devices while customers are in the right area to take advantage of offers. Recalculate entire risk portfolios in minutes. Quickly identify customers who matter the most. Use clickstream analysis and data mining to detect fraudulent behavior. Challenges Many organizations are concerned that the amount of amassed data is becoming so large that it is difficult to find the most valuable pieces of information. What if your data volume gets so large and varied you don't know how to deal with it? Do you store all your data? Do you analyze it all? WWW.KELLYTECHNO.COMPage 3 How can you find out which data points are really important? How can you use it to your best advantage? Until recently, organizations have been limited to using subsets of their data, or they were constrained to simplistic analyses because the sheer volumes of data overwhelmed their processing platforms. But, what is the point of collecting and storing terabytes of data if you can't analyze it in full context, or if you have to wait hours or days to get results? On the other hand, not all business questions are better answered by bigger data. You now have two choices: Incorporate massive data volumes in analysis. If the answers you're seeking will be better provided by analyzing all of your data, go for it. High-performance technologies that extract value from massive amounts of data are here today. One approach is to apply high-performance analytics to analyze the massive amounts of data using technologies such as grid computing, in-database processing and in-memory analytics. Determine upfront which data is relevant. Traditionally, the trend has been to store everything (some call it data hoarding) and only when you query the data do you discover what is relevant. We now have the ability to apply analytics on the front end to determine relevance based on context. This type of analysis determines which data should be included in analytical processes and what can be placed in low-cost storage for later use if needed. Technologies A number of recent technology advancements enable organizations to make the most of big data and big data analytics: Cheap, abundant storage. Faster processors. Affordable open source, distributed big data platforms, such as Hadoop. Parallel processing, clustering, MPP, virtualization, large grid environments, high connectivity and high throughputs. Cloud computing and other flexible resource allocation arrangements. The goal of all organizations with access to large data collections should be to harness the most relevant data and use it for better decision making. Three Enormous Problems Big Data Tech Solves But whats less commonly talked about is why Big Data is such a problem beyond size and computing power. The reasons behind the conversation are the truly interesting part and need to be understood. WWW.KELLYTECHNO.COMPage 4 Here you gothere are three trends that are driving the discussion and should be made painfully clear instead of lost in all the hype: Were digitizing everything. This is big datas volume and comes from unlocking hidden data from common things all around us that were known before but werent quantified, stored, compared and correlated. Suddenly, theres enormous value in the patterns of what was recently hidden from our view. Patterns offer understanding and a chance for prediction of what will happen next. These each are important and together are remarkably powerful. Theres no time to intervene. This is big datas velocity. All of that digital data creates massive historical records but also rich streams of information that are flowing constantly. When we take the patterns discovered in historical information and compare it to everything happening right now, we can either make better things happen or prevent the worst. This is revenue generating and life saving and all of the other wonderful things we hear about, but only if we have the systems in place to see it happening in the moment and do something about it. We cant afford enough human watchers to do this, so the development of big data systems is the only way to get to better things when the data gives humans insufficient time to intervene. Variation creates instability. This is big datas variety. Data was once defined by what we could store and relate in tables of columns and rows. A world thats digitized ignores those boundaries and is instead full of both structured and unstructured data. That creates a very big problem for systems that were built upon the old definition, which comprise just about everything around us. Suddenly, theres data available that cant be consumed or generated by a database. We either ignore that information or it ends up in places and formats that are unreadable to older systems. Gone is the ability to correlate unstructured information with that vast historical (but highly structured) data. When we cant analyze and correlate well, we introduce instability into our world. Were missing the big picture unless we build systems that are flexible and dont require reprogramming the logic for every unexpected (and there will be many) change. There you have it The underlying reasons that big data matters and isnt just hype (though theres plenty of that). The digitization, lack of time for intervention and instability that big data creates leads us to develop whole new ways of managing information that go well beyond Hadoop and distributed computing. Its why big data presents such enormous challenge and opportunity for software vendors and their customers, but only if these three challenges are the drivers and not opportunism. BI vs. Big Data vs. Data Analytics By Example Business Intelligence (BI) encompasses a variety of tools and methods that can help organizations make better decisions by analyzing their data. Therefore, Data Analytics falls under BI. Big Data, if used for the purpose of Analytics falls under BI as well. WWW.KELLYTECHNO.COMPage 5 Lets say I work for the Center for Disease Control and my job is to analyze the data gathered from around the country to improve our response time during flu season. Suppose we want to know about the geographical spread of flu for the last winter (2012). We run some BI reports and it tells us that the state of New York had the most outbreaks. Knowing that information we might want to better prepare the state for the next winter. Theses types of queries examine past events, are most widely used, and fall under the Descriptive Analytics category. Now, we just purchased an interactive visualization tool and I am looking at the map of the United States depicting the concentration of flu in different states for the last winter. I click on a button to display the vaccine distribution. There it is; I visually detected a direct correlation between the intensity of flu outbreak with the late shipment of vaccines. I noticed that the shipments of vaccine for the state of New York were delayed last year. This gives me a clue to further investigate the case to determine if the correlation is causal. This type of analysis falls under Diagnostic Analytics (discovery).We go to the next phase which is Predictive Analytics. PA is what most people in the industry refer to as Data Analytics. It gives us the probability of different outcomes and it is future-oriented. The US banks have been using it for things like fraud detection. The process of distilling intelligence is more complex and it requires techniques like Statistical Modeling. Back to our examples, I hire a Data Scientist to help me create a model and apply the data to the model in order to identify causal relationships and correlations as they relate to the spread of flu for the winter of 2013. Note that we are now taking about the future. I can use my visualization tool to play around with some variables such as demand, vaccine production rate, quantityto weight the pluses and minuses of different decisions insofar as how to prepare and tackle the potential problems in the coming months. The last phase is the Prescriptive Analytics and that is to integrate our tried-and-true predictive models into our repeatable processes to yield desired outcomes. An automated risk reduction system based on real-time data received from the sensors in a factory would be a good example of its use case. Finally, here is an example of Big Data. Suppose its December 2013 and it happens to be a bad year for the flu epidemic. A new strain of the virus is wreaking havoc, and a drug company has produced a vaccine that is effective in combating the virus. But, the problem is that the company cant produce them fast enough to meet the demand. Therefore, the Government has to prioritize its shipments.Currently the Government has to wait a considerable amount of time to gather the data from around the country, analyze it, and take action. The process is slow and inefficient. The following includes the contributing factors. Not having fast enough computer systems capable of gathering and storing the data (velocity), not having computer systems that can accommodate the volume of the data pouring in from all of the medical centers in the country (volume), and not having computer systems that can process images, i.e, x-rays (variety). Big Data technology changed all of that. It solved the velocity-volume-variety problem. We now have computer systems that can handle Big Data.The Center for Disease Control may receive the data WWW.KELLYTECHNO.COMPage 6 from hospitals and doctor offices in real-time and Data Analytics Software that sits on the top of Big Data computer system could generate actionable items that can give the Government the agility it needs in times of crises.