Policy Paper : Need for focussed Big Data & Analytics skillset

building through NSDC(National Skill Development Corporation)

& creation of Bigdata & Analytics focussed IT Parks

Author:

Ritesh Shrivastava, Group Project Manager,SAP

HCL Technologies Ltd.Cell:+91-9818342252

Linkedin: https://in.linkedin.com/in/ riteshshrivastava Twitter: @Ritz1975

TABLE OF CONTENTS

1. INTRODUCTION.......................................................................................................................................... 4

1.1. EVOLUTION OF BIG DATA................................................................................................................................61.2. HOW BIG IS BIG DATA?..................................................................................................................................81.3. WHY USE BIG DATA?.....................................................................................................................................9

1.1. INTRODUCTION

Big Data is an unprecedented phenomenon happening current in the world of ICT (Information AND Communications Technology) field which focuses on tools and techniques to make sense of the large volume, velocity, and variety of sources of the creation of new data and make meaningful inferences productive to the organization /enterprises based on the same.

Organizations, both public and private, will need to adopt in order to manage, make sense of, and obtain Economic and Social value from this vast quantity of newly generated data generated in this era of social media and mobile computing with access to technology available at affordable price-points for all customers

BIG DATA can be defined as structured, semi-structured and unstructured information from demographic and psychographic information about consumers to product reviews and commentary, blogs, content on social media websites, and data streamed 24/7 from mobile devices, sensors and technical devices. (Refer attached Figure /Exhibit 1)

In fact, lots of time, these emerging disruptive trends are referred to as SMAC phenomenon (Social, Mobile, Analytics and Cloud Computing).Exabytes (1018) of new data are created every single day. Much of this information is transported over Internet protocol (IP) networks. First described by Clive Humby as the “new oil,” this data growth is fueling knowledge economies, sparking innovation, and unleashing waves of creative destruction. But most of these data are unstructured and underutilized, flowing at a volume and velocity that is often too large and too fast to analyze. If data do, in fact, comprise the new raw material of business, on par with economic inputs such as capital and labor, then deriving insight and added value from this new input will require targeted transmission, processing, and analysis. A rising share of this data growth is flowing over IP networks as more people, places, and things connect to this Internet of Everything (IoE). Proprietary networks, built on industry-siloed standards such as those in manufacturing or electric utilities, are increasingly migrating to IP networks, facilitating the growth of big data, and fast becoming the key link among data generation, processing, analysis, and utilization.

Big Data can solve those problems. With the appropriate software and tools, companies can comb through the nonproductive, non-revenue-producing data in storage and analyze it to identify business trends and reveal new opportunities. Big Data is different from ordinary database information because of the massive volume, the variety (structured, semi-structured, and unstructured), and the velocity. Now information such as product reviews, e-mail, video, blogs, web log files, and Tweets are in the data mix. Social media feedback has become a very useful research tool for businesses but sophisticated analysis of the massive quantities was not possible before analytic tools were developed specifically for Big Data. Additionally, Big Data encompasses data generated by machines such as sensors. Data have always had strategic value, but with the magnitude of data available today—and our capability to Process them—they have become a new form of asset class. In a very real sense, data are now the equivalent Of oil or gold. And today we are seeing a data boom rivaling the Texas oil boom of the 20th century and the San Francisco gold rush of the 1800s. It has spawned an entire support industry and has attracted a great deal Of business press in recent years. This new asset class of big data is commonly described by what we call the “three Vs.” Big data is high volume, high velocity, and includes a high variety of sources of information. Next to those traditional three Vs. we could add a fourth: value. This is what everyone is looking for, and this is why big data today gets so much attention. In the quest for value, the challenge facing us is how to reduce the complexity and unwieldiness of big data so that it becomes truly valuable.

Big data can take the form of structured data such as financial transactions or unstructured data such as photographs or blog posts. It can be crowd-sourced or obtained from proprietary data sources. Big data has been fueled by both technological advances (such as the spread of radio-frequency identification, or RFID, chips) and social trends (such as the widespread adoption of social media). Our collective discussions, comments, likes, dislike, and networks of social connections are now all data and their scale are massive. What did we search for? What did we read? Where did we go? With whom do we associate? What do we eat? What do we purchase? In short, almost any imaginable human interaction can be captured and studied within the realm of big data.

Several major transitions in technology—each important in its own right—are combining to make the Internet of Everything possible. These include the emergence of cloud and mobile computing, the growth of big data and analytics, and the explosive development of the Internet of Things (IoT). These transitions are changing the role of information technology (IT), with Internet protocol (IP) networks playing an increasingly central part by seamlessly connecting disparate IT Environments. The explosive expansion of IoT, or connections between context-aware machines and other physical objects, is changing how we utilize devices to improve our daily lives. And the shift in data and analytics— from being centralized, structured, and static to being distributed, mixed structured and unstructured, and real time— is leading to a new era of real-time processing and decision-making. More industries are moving their systems and processes to IP networks, and the rapid growth of IP connected devices is driving exponential increases in data traffic. The migration to IP networks and the ability to turn “Big Data” into valuable, actionable information have demonstrable benefits—both economic and social—as well as positive financial impacts for firms.

The Big Data market is at $5.1 billion as of 2013 and is expected to grow to $32.1 billion by 2015 and to $53.4 billion by 2017.We create 2.5 quintillion bytes of data daily .90% of this data has been created in last two years due to rapid enhancement in ICT and

Mobile Computing.62% of companies believe Big data has significant potential to create competitive advantage.Almost every business collects data, and the majority of it is about their customers. Over the past several years the cost of storing data has decreased substantially, thanks mostly to cloud solutions. Now companies gather huge amounts of information and store it just in case it may turn out to be useful someday. Accumulating and storing data is easy. Organizing and analyzing data and figuring out what is valuable are much more complex tasks. And knowing how to extract meaningful, actionable insights is even more difficult, but is crucial to bolstering the bottom line. Big Data can solve those problems. With the appropriate software and tools, companies can comb through the nonproductive, non-revenue-producing data in storage and analyze it to identify business trends and reveal new opportunities.

Big Data is different from ordinary database information because of the massive volume, the variety (structured, semi-structured, and unstructured), and the velocity. Now information such as product reviews, e-mail, video, blogs, web log files, and Tweets are in the data mix. Social media feedback has become a very useful research tool for businesses but sophisticated analysis of the massive quantities was not possible before analytic tools were developed specifically for Big Data. Additionally, Big Data encompasses data generated by machines such as sensors.

Big data has arrived. It is changing our lives and changing the way we do business. But succeeding with big data requires more than just data. Data-based value creation requires the identification of patterns from which predictions can be inferred and decisions made. Businesses need to decide which data to use.The data each business owns might be as different as the businesses themselves; these data range from log files and GPS data to customer- or machine-to-machine data. Each business will need to select the data source it will use to create value. Moreover, creating this value will require the right way of dissecting and then analyzing those data with the right analytics. It will require knowing how to separate valuable information from hype. (Refer Attached Figure-1)

This world of big data has also become a source of concern. The consequences of big data for issues of privacy and other areas of society are not yet fully understood. Some prominent critics, such as Jaron Lanier, call on us to be cautious about readily believing any result created by the “wisdom of the crowd.” Moreover, applications of big data in military intelligence have created a growing concern for privacy around the world. Indeed, we are now living in a world where anything and everything can be measured. “Data” could become a new ideology. We are just at the beginning of a long journey where, with the proper principles and guidelines, we should be able to collect, measure, and analyze more and more information about everyone and everything in order to make better decisions.

1.2. HOW BIG IS BIG DATA?

In 2000, 800,000 petabytes of data were stored worldwide. (One thousand terabytes equals one Petabyte.) Today, Facebook alone generates ten terabytes of data every day and Twitter comes in At seven terabytes. About 90 percent of the available data in the world has been generated in the past two years.

1.3. WHY USE BIG DATA?

Big Data analytic tools facilitate the examination of large amounts of different types of data to Reveal hidden patterns and correlations that are not otherwise easily discernible. IBM’s 2012 Big Data @ Work Survey of 1144 professionals found that 63 percent of respondents reported that the use of information including Big Data and analytics is creating a Competitive advantage for their organizations. While over 50 percent of respondents stated that they had not yet begun Big Data analysis, 47 percent planned to do so. In another study, The Deciding Factor: Big Data & Decision Making, conducted by the Economist Intelligence Unit for Cap Gemini, two-thirds of the executives note in the study that they consider their organizations to be “data-driven,” meaning that data collection and analysis are the Foundation of their firms’ business strategies and day-to-day decision-making.The percentages are highest in the energy, financial services, healthcare and Pharma. More than half of the respondents say management decisions that are based purely on intuition Or experiences are increasingly regarded as questionable. And 65 percent affirm that more Management decisions are based on validated analytic information. That figure rises to 73 percent for the financial services sector.Big Data allows companies to make better decisions using existing and new data sources. It offers the capability to better understand and predict consumer behavior. Real time data capture enables faster decision-making, such as when a customer is on a website or on the telephone speaking with a customer service representative. The use of Big Data has improved the performance of businesses by 26 percent on average and that influence will grow to 41 percent over the next three years, according to a Cap Gemini study.The creation of immense job opportunities by mass skill-building and creation of Bigdata & Analytics focused IT parks targeted for startups and boutique firms , just like what is done in Hyderabad by Telengana state government, can create a second mass-scale employment generation just like the mid 1990s IT Revolution.