Big Data using NoSQL Technologies

Big Data using NoSQL Technologies

Amit Kr. Singh

Senior Developer, Ericsson

December 14, 2012

Slide title minimum 32 pt

(32 pt makes 2 rows

Text and bullet level 1 minimum 24 pt

Bullets level 2-5minimum 20 pt

!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´¶·¸¹º»¼½ÀÁÂÃÄÅÆÇÈËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĀāĂăąĆćĊċČĎďĐđĒĖėĘęĚěĞğĠġĢģĪīĮįİıĶķĹĺĻļĽľŁłŃńŅņŇňŌŐőŒœŔŕŖŗŘřŚśŞşŠšŢţŤťŪūŮůŰűŲųŴŵŶŷŸŹźŻżŽžƒȘșˆˇ˘˙˚˛˜˝ẀẁẃẄẅỲỳ–—‘’‚“”„†‡•…‰‹›⁄€™−≤≥fifl

ĀĀĂĂĄĄĆĆĊĊČČĎĎĐĐĒĒĖĖĘĘĚĚĞĞĠĠĢĢĪĪĮĮİĶĶĹĹĻĻĽĽŃŃŅŅŇŇŌŌŐŐŔŔŖŖŘŘŚŚŞŞŢŢŤŤŪŪŮŮŰŰŲŲŴŴŶŶŹŹŻŻȘș

ΆΈΉΊΌΎΏΐΑΒΓΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΪΫΆΈΉΊΰαβγδεζηθικλνξορςΣΤΥΦΧΨΩΪΫΌΎΏ

ЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁЂЃЄЅІЇЈЉЊЋЌЎЏѢѢѲѲѴѴҐҐә ẀẁẂẃẄẅỲỳ№ǽ

Do not add objects or text in the footer area

My Background

Part of Java and Open Source Practice Area. Driving technology initiatives in LockBox project. Part of System-X development team. Contributing in JOSP Competence Development & Training.


(32 pt makes 2 rows








Big Data


(32 pt makes 2 rows








Big Data

Ericsson defines:

“People,devices and things are constantly generating massive volumes of data. At work people create data, as do children at home, students at school, people and things on the move, as well as objects that are stationary. Devices and sensors attached to millions of things take measurements from their surroundings, providing up-to-date readings over the entire globe – data to be stored for later use by countless different applications.”


(32 pt makes 2 rows








Big Data

IBM defines:

“Every day, we create 2.5 quintillion bytes of data — so much that 90% of the data in the world today has been created in the last two years alone. This data comes from everywhere: sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals to name a few. This data is big data.”


(32 pt makes 2 rows








Big Data

Wikipedia defines:

Big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools. The challenges include capture, curation, storage, search, sharing, analysis and visualization.


(32 pt makes 2 rows








Big Data

Why so many definitions? I am really confused.


(32 pt makes 2 rows








Big Data

In simple words, A set of technology advances that have made capturing and analyzing data at high scale and speed vastly more efficient.


(32 pt makes 2 rows








Twitter


(32 pt makes 2 rows








Facebook


(32 pt makes 2 rows








Six insights from Facebook's former Head of Big Data

Analytics on 900M users

25PB of compressed data – 125 uncompressed New technologies has shifted the conversions from “what data to store” to

“what can we do with more data”. Simplify data anlytics for end users. More users means data analytics system have to be more robust. Social networking works for Big Data. No single infrastructure can solve all Big Data problems. Building software is hard, but running a service is even harder.


(32 pt makes 2 rows








Big Data in Retail


(32 pt makes 2 rows








Big Data in IT


(32 pt makes 2 rows








Big Data in Customer Services


(32 pt makes 2 rows








Big Data Dimensions


(32 pt makes 2 rows








The Three Vs of Big Data

Volume – big data comes in one size XXL and available storage cannot handle these volumes.

Velocity – data needs to be used quickly to maximize business benefit before the value of the information is lost.

Variability – data can be structured, unstructured, semi-structured or a mix of all three. It comes in many forms including text, audio, video, click streams and log files.


(32 pt makes 2 rows








Big Data Technologies

Big-data technologies are usually engineered from the bottom up with two things in mind: scale and availability. Most solutions are distributed in nature and introduce new programming models for working with large volumes of data.

Technologies such as Not only SQL (NoSQL), characterized by its non-adherence to the RDBMS model, used in a wide variety of industry applications. These technologies have the flexibility to handle Big Data.


(32 pt makes 2 rows








Scalability

Scalability refers to the ability of an application or product to increase in size as demand warrants. The base concept is consistent – the ability for a business or technology to accept increased volume without impacting the business settings.

Scale horizontally (scale out) Scale vertically (scale up)


(32 pt makes 2 rows








Scalability

Scale vertically (scale up)

Extra capacity can be obtained by adding more hardware to a specific computer or by moving applications to larger computers – a process known as vertical scaling. One limitation of this approach is the risk of outgrowing the capacity of the largest computer; this will eventually affect cost. Vendor lock-in is a potential risk, and vertically scaled solutions can become prohibitively expensive.


(32 pt makes 2 rows








Scalability

Scale horizontally (scale out)

Adding computers in parallel can also increase capacity. This approach is known as horizontal scaling, and Big Data technologies tend to favor it because it supports network expansion. Systems that are built in this way are more flexible, and because commodity computers can be operated together in parallel, the risk associated with single vendor solutions is reduced. Also horizontal scaling is built for Cloud.


(32 pt makes 2 rows








Availability

Availability is a guarantee that every request receives a response about whether it was successful or failed.

Users want their systems (Facebook, Twitter, Telecom app, etc) to be ready to serve them at all times. If a user cannot access the system, it is said to be unavailable. Generally, the term downtime is used to refer to periods when a system is unavailable.


(32 pt makes 2 rows








NoSQL

What NoSQL databases can: Serve as an online processing database, so that it becomes the primary datasource/operational datastore for online applications. Use data stored in primary source systems for real-time, batch analytics, and enterprise search operations. Handle “big data” use cases that involve data velocity, variety, volume, and complexity. Excel at distributed database and multi-data center operations. Offer a flexible schema design that can be changed without downtime or service disruption. Accommodate structured, semi-structured, and non-structured data. Easily operate in the cloud and exploit the benefits of cloud computing.


(32 pt makes 2 rows








Is NoSQL replacing the RDBMS?

The answer is both yes and no, considering that the choice between the two depends on the Use Case.

NoSQL doesn't take advantage of ACID properties. Applications which depend on transaction support (Banking, Airlines etc) will continue to work with RDBMS while Social Media applications which mostly deal with unstructured data will look at alternative NoSQL solutions. However hybrid architecture may prove beneficial as well where the power of both RDBMS and NoSQL can be leveraged.


(32 pt makes 2 rows








Is NoSQL replacing the RDBMS?

However many enterprises are choosing to leave some legacy RDBMS systems in place, while directing new development towards NoSQL databases. This is especially the case when the applications in question demand high write throughput, need flexible schema designs, process large volumes of data, and are distributed in nature.

Technology aside, another reason many new development and/or migration efforts are being directed towards NoSQL databases is the high cost of legacy RDBMS vendors versus NoSQL software. In general the fact is that, NoSQL software is a fraction of what vendors such as IBM and Oracle charge for their databases.


(32 pt makes 2 rows








RDBMS & Big Data

Tactics to extend the useful scope of RDBMS technology Sharding Denormalizing Distributed caching


(32 pt makes 2 rows








Sharding

If the data for an application will not fit on a single server or, more likely, if a single server is incapable of maintaining the I/O throughput required to serve many users simultaneously, then a tactic known as sharding is frequently employed.

Database sharding is the process of splitting up a database across multiple machines to improve the scalability of an application.


(32 pt makes 2 rows








Sharding

This does work to spread the load but there are some undesirable consequences to the approach.

When you fill a shard, you have to change the sharding strategy in the application itself. For example, placing user profile information on one database server, friend lists on another and a third for user generated content like photos and blogs. The main problem with this approach is that if the site experiences additional growth then it may be necessary to further shard a feature specific database across multiple servers.

You lose some of the most important benefits of the relational model. You can’t do “joins” across shards. In addition, you can’t do cross-node locking when making updates.


(32 pt makes 2 rows








Denormalizing

Denormalization is the process of attempting to optimise the read performance of a database by adding redundant data or by grouping data. In some cases, denormalisation is a means of addressing performance or improving the scalability in relational database software.

Most of the time denorm is application-specific and needs to be re-evaluated if

the application changes. Denorm can increase the size of tables.


(32 pt makes 2 rows








Distributed Caching

Another tactic used to extend the useful scope of RDBMS technology is to employ distributed caching technologies, such as Memcached. Today, Memcached is a key ingredient in the data architecture behind 18 of the top 20 largest (by user count) Web applications, including Google, Wikipedia, Twitter, YouTube and Facebook.

Memcached “sits in front” of an RDBMS system, caching recently accessed data in memory and storing that data across any number of servers or virtual machines. When an application needs access to data, rather than going directly to the RDBMS, it first checks Memcached to see if the data is available there; if it is not, then the database is read by the application and stored in Memcached for quick access next time it is needed.


(32 pt makes 2 rows








Distributed Caching


(32 pt makes 2 rows








Distributed Caching

Memcached and similar distributed caching technologies used for this purpose are no magic and can even create problems of their own:

Memcached was designed to accelerate the reading of data by storing it in main memory, but it was not designed to permanently store data. Memcached stores data in memory. If a server is powered off or otherwise fails, or if memory is filled up, data is lost.

Again another tier to manage. It should be obvious that inserting another tier of infrastructure into the architecture to address some (but not all) of the failings of RDBMS technology in the modern interactive software use case can create its own set of problems: more capital costs, more operational expense, more points of failure and more complexity.


(32 pt makes 2 rows








NoSQL Technologies

Sharding, Denormalizing, Distributed Caching and other tactics are all attempt to paper over one simple fact: RDBMS technology is a forced fit for modern interactive software systems. Because vendors of RDBMS technology have little incentive to disrupt a technology generating billions of dollars for them annually.

Few application developers from Google (Big Table) and Amazon (Dynamo) took initiatives and invented, developed No SQL database technologies.


(32 pt makes 2 rows








NoSQL Characteristics:

No schema required. Data can be inserted in a NoSQL database without first defining a rigid database schema. As a corollary, the format of the data being inserted can be changed at any time, without application disruption. This provides immense application flexibility, which ultimately delivers substantial business flexibility.

Auto-sharding. A NoSQL database automatically spreads data across servers, without requiring applications to participate. Servers can be added or removed from the data layer without application downtime. Most NoSQL databases also support data replication, storing multiple copies of data across the cluster, and even across data centers, to ensure high availability and support disaster recovery.


(32 pt makes 2 rows








NoSQL Characteristics:

Distributed query support. “Sharding” an RDBMS can reduce, or eliminate in certain cases, the ability to perform complex data queries. NoSQL database systems retain their full query expressive power even when distributed across hundreds or thousands of servers.

Integrated caching. To reduce latency and increase sustained data throughput, advanced NoSQL database technologies transparently cache data in system memory. This behavior is transparent to the application developer and the operations team, in contrast to RDBMS technology where a caching tier is usually a separate infrastructure tier that must be developed to, deployed on separate servers, and explicitly managed by the ops team.


(32 pt makes 2 rows








Research activities in Big Data

The White House has recently announced a national "Big Data Initiative" for improving the ability to extract knowledge and insights from large and complex collections of digital data. This initiative will help US goverment in scientific discovery, environmental and biomedical research, education, and national security.

NASA is working on number of innovative approaches to advancing Big Data, including the Lunar Mapping and Modeling Activity


(32 pt makes 2 rows








Reference

www.couchbase.com www.datastax.com www.forbes.com/sites/davefeinleib/2012/07/09/the-3-is-of-big-data/ www.slideshare.net/bigdatalandscape/big-data-trends www.kunocreative.com/blog/bid/76907/Big-Data-Made-Simple-What- Marketers-Need-to-Know

Big Data using NoSQL Technologies

Technology

Transcript of Big Data using NoSQL Technologies