Ejemplo de sistemas escalables en entornos distribuidos: Hadoop (v2d)

Diseño de Sistemas Distribuidos Máster en Ciencia y Tecnología Informá<ca

Curso 2014-‐2015

Alejandro Calderón Mateos & Óscar Pérez Alonso Grupo de Arquitectura de Computadores [email protected]

Sistemas escalables en entornos distribuidos

Diseño

de Sistem

as Distrib

uido

s Alejandro Calderón

Mateo

s 2

Contenidos

– Mo<vación –  Introducción –  Hand-‐on –  Benchmarking

–  Ecosistema

hTp://www.siliconweek.es/wp-‐content/uploads/2013/08/BigData-‐datos-‐guardar-‐almacenamiento-‐fichero-‐archivo.jpg hTp://datameer2.datameer.com/blog/wp-‐content/uploads/2012/06/Hadoop-‐Ecosystem-‐Infographic-‐21.png

Diseño

de Sistem

as Distrib

uido


Mateo

s 3

Contenidos

– Mo#vación –  Introducción –  Hand-‐on –  Benchmarking

–  Ecosistema


Diseño

de Sistem

as Distrib

uido


Mateo

s 4

Repaso de opciones… (1/3)

hTp://www.scsi4me.com/images/1296909213_60.jpg

File System

Disk Disk

App

Ordenador

DAS Direct ATached Storage

•  HW: –  Ordenador con periféricos de

almacenamiento. •  SW:

–  Sistema de ficheros local o gestor de base de datos local.

•  V/I: –  Rapidez, simplicidad –  No compar<ción, crecimiento limitado

Diseño

de Sistem

as Distrib

uido


Mateo

s 5


hTp://www.jollynas.com/nas/img/wJollyNAS.jpg

NAS

File System

Disk Disk

Network

App

Cliente

NAS Network ATached Storage

NFS, CIFS

•  HW: –  Ordenador con periféricos de

comunicación. –  NAS: ordenador con periféricos de

comunicación y periféricos de almacenamiento.

•  SW: –  Sistema de ficheros remoto o

gestor de base de datos remoto.

•  V/I: –  Compar<ción (RO), crecimiento –  Red limita velocidad y crecimiento

Diseño

de Sistem

as Distrib

uido


Mateo

s 6


hTp://andysworld.org.uk/blog/wp-‐content/uploads/2010/05/ra4100.jpg

Disk Disk

SAN

Shared F.S.

App

RAID/JBOD

App. server iSCSI

SAN Storage Area Network •  HW:

–  Ordenador con periféricos de comunicación.

–  SAN: periféricos de almacenamiento con disposi<vo de comunicación.

•  SW: –  Gestor de base de datos o

sistema de ficheros, compar<do.

•  V/I: –  Mejor crecimiento-‐velocidad –  Complejidad, coste

Diseño

de Sistem

as Distrib

uido


Mateo

s 7

Combinación de opciones…

File System

Disk Disk

App

Ordenador

DAS Direct ATached Storage

NAS

File System

Disk Disk

Network

App

Cliente

NAS Network ATached Storage

NFS, CIFS

Disk Disk

SAN

Shared F.S.

App

RAID/JBOD

App. server

iSCSI

SAN Storage Area Network + +

Diseño

de Sistem

as Distrib

uido


Mateo

s 8


SSD AFA

SAN

NAS + SAN

Shared F.S.

App GP

FS, Lustre,…

iSCSI

Network

Shared F.S.

Disks Disk

SSD AFA Disks Disk

Diseño

de Sistem

as Distrib

uido


Mateo

s 9


SAN

Shared F.S.

App

Network

Shared F.S.

Blocks

Files Objs.

GPFS, Lustre,…

iSCSI

SSD AFA Disks Disk

Diseño

de Sistem

as Distrib

uido


Mateo

s 10


SAN

Shared F.S.

App

Network

Shared F.S.

Blocks

Files Objs.

GPFS, Lustre,…

iSCSI

S3, IL7, … NFS, CEPH, …

iSCSI, FCoE, …

Diseño

de Sistem

as Distrib

uido


Mateo

s 11


App

Blocks

Files Objs.

S3, IL7, … NFS, CEPH, …

iSCSI, FCoE, …

Gran almacenamiento con capacidad de “crecimiento”

Diseño

de Sistem

as Distrib

uido


Mateo

s 12

Big data is coming…

hTp://online.wsj.com/news/ar<cles/SB10001424127887324178904578340071261396666

Diseño

de Sistem

as Distrib

uido


Mateo

s 13

Big data is coming…


•  Ciencia •  Industria •  Logs de sistema •  …

Diseño

de Sistem

as Distrib

uido


Mateo

s 14

Big Data frequently used…

hTp://www.ndm.net/emcstore/storage/greenplum hTp://fcw.com/~/media/GIG/FCWNow/Topics/Big%20Data/Big_data.png

•  Text mining •  Index building •  Graph crea<on and

analysis •  PaTern recogni<on •  Predic<on model •  …

SSD AFA Disks Disk

Diseño

de Sistem

as Distrib

uido


Mateo

s 15

Problemas…

SAN

Shared F.S.

App

Network

Shared F.S.

GPFS, Lustre,…

iSCSI

•  Gran can<dad de tráfico por movimiento de datos.

SSD AFA Disks Disk

Diseño

de Sistem

as Distrib

uido


Mateo

s 16

Opciones para almacenamiento…

SAN

Shared F.S.

App

Network

Shared F.S.

GPFS, Lustre,…

iSCSI


•  Posible disminuir si se acerca parte del cómputo al almacenamiento.

A0 A1 A2 A3

A4 A5

SSD AFA Disks Disk

Diseño

de Sistem

as Distrib

uido


Mateo

s 17


SAN

Shared F.S.

App

Network

Shared F.S.

GPFS, Lustre,…

iSCSI



A0 A1 A2 A3

A4 A5

SSD AFA Disks Disk

Diseño

de Sistem

as Distrib

uido


Mateo

s 18


SAN

Shared F.S.

App

Network

Shared F.S.

GPFS, Lustre,…

iSCSI



A0 A1 A2 A3

A4 A5

hTp://rsoltd.com/wp-‐content/uploads/2014/04/expensive.jpg

Diseño

de Sistem

as Distrib

uido


Mateo

s 19

Buscar/crear la herramienta adecuada…

hTp://storageio.com/images/SIO_ToolBox.png

Diseño

de Sistem

as Distrib

uido


Mateo

s 20

The Google File System

•  Google presenta MapReduce y Google File System (GFS) – MapReduce es una propuesta para aplicar la misma función a par<ciones de datos (map) y luego se <ene el resultado procesando los resultados parciales (reduce)

– GFS es una propuesta para almacenar petabytes de datos en muchas máquinas comunes, tratando con fallos, distribución, etc.

hTp://sta<c.googleusercontent.com/media/research.google.com/en//archive/gfs-‐sosp2003.pdf

SSD AFA Disks Disk

Diseño

de Sistem

as Distrib

uido


Mateo

s 21

De esta opción…

SAN

Shared F.S.

App

Network

Shared F.S.

GPFS, Lustre,…

iSCSI

A0 A1 A2 A3

A4 A5

Diseño

de Sistem

as Distrib

uido


Mateo

s 22

…a esta opción

GFS

Disk

App

A0

GFS

Disk

A1

GFS

Disk

A3

GFS

Disk

A4

GFS

Disk

A5

GFS

Disk

A6

GFS

Disk

A7

GFS

Disk

A8

Network

Diseño

de Sistem

as Distrib

uido


Mateo

s 23

Hadoop Breve historia

•  Doug Cusng trabajando en Yahoo! e inspirado por estas tecnologías, inicia el desarrollo de Hadoop. – El proyecto usa Java, implanta las ideas detrás de GFS y MapReduce.

– El logo<po se basa en el elefante amarillo que era el juguete favorito de su hijo.

– Doug Cusng pasó a trabajar a Cloudera.

hTp://www.enriquedans.com/2011/11/hadoop-‐el-‐elefante-‐omnipresente.html

Diseño

de Sistem

as Distrib

uido


Mateo

s 24

Hadoop Breve historia

•  Actualmente es un proyecto de código abierto bajo licencia Apache. –  Su licencia ha facilitado que sea adoptado por un importante número de empresas.

•  El apoyo de IBM, Oracle, EMC, etc. ha acelerado su implantación y su mejora en prestaciones. – Gran uso en proyectos <po big data. –  JPMorgan Chase: “We’re hiring, and we’re paying 10% more than the other guys.”

hTp://www.informa<onweek.com/soxware/informa<on-‐management/its-‐next-‐hot-‐job-‐hadoop-‐guru/d/d-‐id/1101209?

Diseño

de Sistem

as Distrib

uido


Mateo

s 25

hTp://hadoop.apache.org

•  Apache Hadoop es un proyecto soxware open source para computación distribuida escalable y de confianza (reliable).

•  Ofrece un framework que permite la computación distribuida de grandes conjuntos de datos mediante clusters de ordenadores usando modelos de programación simples. –  Diseñado para poder pasar de un solo servidor a miles de máquinas

(scale-‐up), donde cada una ofrece tanto computación como almacenamiento.

–  El framework está diseñado para detectar y tratar fallos a nivel de aplicación. Esto permite ofrecer servicios con alta disponibilidad sobre un cluster de ordenadores (aunque estos tengan fallos).

hTp://hadoop.apache.org/

Diseño

de Sistem

as Distrib

uido


Mateo

s 26

B.D.A. Workflow example…

hTp://www.infoivy.com/2013/12/5-‐steps-‐for-‐big-‐data-‐applica<on.html

Diseño

de Sistem

as Distrib

uido


Mateo

s 27

Big Opportuni<es…


Diseño

de Sistem

as Distrib

uido


Mateo

s 28

Big Opportuni<es…

hTp://www.indeed.com/jobtrends?q=Big-‐data&rela<ve=1

Diseño

de Sistem

as Distrib

uido


Mateo

s 29

Contenidos


–  Ecosistema


Diseño

de Sistem

as Distrib

uido


Mateo

s 30

Arquitectura

hTp://www.sachinpbuzz.com/2014/01/big-‐data-‐overview-‐of-‐apache-‐hadoop.html

Diseño

de Sistem

as Distrib

uido


Mateo

s 31

Despliegue

hTp://blog.csdn.net/suifeng3051/ar<cle/details/17288047

Diseño

de Sistem

as Distrib

uido


Mateo

s 32

Despliegue


Diseño

de Sistem

as Distrib

uido


Mateo

s 33

Despliegue


hdfs-‐site.xml:dfs.replica#on

:9000

:50010

Diseño

de Sistem

as Distrib

uido


Mateo

s 34

Despliegue


Diseño

de Sistem

as Distrib

uido


Mateo

s 35

Despliegue


Diseño

de Sistem

as Distrib

uido


Mateo

s 36

Despliegue


Diseño

de Sistem

as Distrib

uido


Mateo

s 37

Despliegue


Diseño

de Sistem

as Distrib

uido


Mateo

s 38

Contenidos


–  Ecosistema


Diseño

de Sistem

as Distrib

uido


Mateo

s 39

Hadoop: solo un nodo Prerequisitos Instalación Uso básico

alejandro@h1:~$ sudo addgroup hadoop Adding group `hadoop' (GID 1001) ... Done. alejandro@h1:~$ sudo adduser -‐-‐ingroup hadoop hduser Adding user `hduser' ... Adding new user `hduser' (1001) with group `hadoop' ... Creating home directory `/home/hduser' ... Copying files from `/etc/skel' ... Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully Changing the user information for hduser Enter the new value, or press ENTER for the default

Full Name []: Room Number []: Work Phone []: Home Phone []: Other []:

Is the information correct? [Y/n]

hTp://www.bogotobogo.com/Hadoop/BigData_hadoop_Install_on_ubuntu_single_node_cluster.php

Diseño

de Sistem

as Distrib

uido


Mateo

s 40


alejandro@h1:~$ sudo apt-‐get install ssh rsync Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: rsync ssh … alejandro@h1:~$ sudo apt-‐get install default-‐jdk Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: libice-‐dev libpthread-‐stubs0-‐dev libsm-‐dev libx11-‐dev libx11-‐doc libxau-‐dev libxcb1-‐dev libxdmcp-‐dev libxt-‐dev openjdk-‐7-‐jdk …

hTp://wiki.apache.org/hadoop/HadoopJavaVersions

Diseño

de Sistem

as Distrib

uido


Mateo

s 41


alejandro@h1:~$ su hduser Password: hduser@h1:/home/alejandro$ ssh-‐keygen -‐t rsa -‐P "" Generating public/private rsa key pair. Enter file in which to save the key (/home/hduser/.ssh/id_rsa): … Your public key has been saved in /home/hduser/.ssh/id_rsa.pub. … The key's randomart image is: +-‐-‐[ RSA 2048]-‐-‐-‐-‐+ | =+B+o. | | ..B.o+. | …

Diseño

de Sistem

as Distrib

uido


Mateo

s 42


hduser@h1:/home/alejandro$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

hduser@h1:/home/alejandro$ ssh localhost The authenticity of host 'localhost (127.0.0.1)' can't be established. ECDSA key fingerprint is eb:51:89:99:49:42:6a:6e:78:5d:79:6c:69:2a:8c:45. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts. Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-‐36-‐generic x86_64) … hduser@h1:~$ exit logout hduser@h1:/home/alejandro$ exit exit

Diseño

de Sistem

as Distrib

uido


Mateo

s 43


alejandro@h1:~$ wget http://apache.rediris.es/hadoop/common/current/hadoop-‐2.5.2.tar.gz … 2014-‐09-‐26 21:57:25 (1,12 MB/s) -‐ ‘hadoop-‐2.5.2.tar.gz’ saved [138656756/138656756]

alejandro@h1:~$ tar zxf hadoop-‐2.5.2.tar.gz alejandro@h1:~$ ls -‐las hadoop-‐2.5.2 total 60 4 drwxr-‐xr-‐x 9 alejandro alejandro 4096 jun 21 08:38 . 4 drwxr-‐xr-‐x 16 alejandro alejandro 4096 sep 27 21:58 .. 4 drwxr-‐xr-‐x 2 alejandro alejandro 4096 jun 21 08:05 bin 4 drwxr-‐xr-‐x 3 alejandro alejandro 4096 jun 21 08:05 etc 4 drwxr-‐xr-‐x 2 alejandro alejandro 4096 jun 21 08:05 include 4 drwxr-‐xr-‐x 3 alejandro alejandro 4096 jun 21 08:05 lib 4 drwxr-‐xr-‐x 2 alejandro alejandro 4096 jun 21 08:05 libexec 16 -‐rw-‐r-‐-‐r-‐-‐ 1 alejandro alejandro 15458 jun 21 08:38 LICENSE.txt 4 -‐rw-‐r-‐-‐r-‐-‐ 1 alejandro alejandro 101 jun 21 08:38 NOTICE.txt 4 -‐rw-‐r-‐-‐r-‐-‐ 1 alejandro alejandro 1366 jun 21 08:38 README.txt 4 drwxr-‐xr-‐x 2 alejandro alejandro 4096 jun 21 08:05 sbin 4 drwxr-‐xr-‐x 4 alejandro alejandro 4096 jun 21 08:05 share

Diseño

de Sistem

as Distrib

uido


Mateo

s 44


alejandro@h1:~$ sudo mv hadoop-‐2.5.2 /usr/local/hadoop alejandro@h1:~$ sudo chown -‐R hduser:hadoop /usr/local/hadoop/ alejandro@h1:~$ cd /usr/local/hadoop/ alejandro@h1:/usr/local/hadoop$ ls -‐las total 60 4 drwxr-‐xr-‐x 9 hduser hadoop 4096 jun 21 08:38 . 4 drwxr-‐xr-‐x 11 root root 4096 sep 27 22:02 .. 4 drwxr-‐xr-‐x 2 hduser hadoop 4096 jun 21 08:05 bin 4 drwxr-‐xr-‐x 3 hduser hadoop 4096 jun 21 08:05 etc 4 drwxr-‐xr-‐x 2 hduser hadoop 4096 jun 21 08:05 include 4 drwxr-‐xr-‐x 3 hduser hadoop 4096 jun 21 08:05 lib 4 drwxr-‐xr-‐x 2 hduser hadoop 4096 jun 21 08:05 libexec 16 -‐rw-‐r-‐-‐r-‐-‐ 1 hduser hadoop 15458 jun 21 08:38 LICENSE.txt 4 -‐rw-‐r-‐-‐r-‐-‐ 1 hduser hadoop 101 jun 21 08:38 NOTICE.txt 4 -‐rw-‐r-‐-‐r-‐-‐ 1 hduser hadoop 1366 jun 21 08:38 README.txt 4 drwxr-‐xr-‐x 2 hduser hadoop 4096 jun 21 08:05 sbin 4 drwxr-‐xr-‐x 4 hduser hadoop 4096 jun 21 08:05 share

Diseño

de Sistem

as Distrib

uido


Mateo

s 45


•  Configurar variables de entorno: –  Encontrar los componentes de Hadoop

•  ~/.bashrc –  Encontrar en Hadoop a JAVA_HOME

•  /usr/local/hadoop/etc/hadoop/hadoop-‐env.sh

•  Configurar los componentes de Hadoop: –  Configurar hadoop.tmp.dir y fs.default.name

•  /usr/local/hadoop/etc/hadoop/core-‐site.xml –  Configurar qué framework usar para mapreduce

•  /usr/local/hadoop/etc/hadoop/mapred-‐site.xml –  Configuración de los directorios para namenode y datanode

•  /usr/local/hadoop/etc/hadoop/hdfs-‐site.xml

Diseño

de Sistem

as Distrib

uido


Mateo

s 46


alejandro@h1:/usr/local/hadoop$ su hduser Password:

hduser@h1:/usr/local/hadoop$ update-‐alternatives -‐-‐config java There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/

jvm/java-‐7-‐openjdk-‐amd64/jre/bin/java Nothing to configure.

hduser@h1:/usr/local/hadoop$ cat >> ~/.bashrc

export JAVA_HOME=/usr/lib/jvm/java-‐7-‐openjdk-‐amd64 export HADOOP_INSTALL=/usr/local/hadoop export PATH=$PATH:$HADOOP_INSTALL/bin export PATH=$PATH:$HADOOP_INSTALL/sbin export HADOOP_MAPRED_HOME=$HADOOP_INSTALL export HADOOP_COMMON_HOME=$HADOOP_INSTALL export HADOOP_HDFS_HOME=$HADOOP_INSTALL export YARN_HOME=$HADOOP_INSTALL export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native export HADOOP_OPTS="-‐Djava.library.path=$HADOOP_INSTALL/lib“

Diseño

de Sistem

as Distrib

uido


Mateo

s 47


hduser@h1:/usr/local/hadoop$ grep JAVA_HOME /usr/local/hadoop/etc/hadoop/hadoop-‐env.sh … export JAVA_HOME=${JAVA_HOME}

hduser@h1:/usr/local/hadoop$ cat >> /usr/local/hadoop/etc/hadoop/hadoop-‐env.sh export JAVA_HOME="/usr/lib/jvm/java-‐7-‐openjdk-‐amd64"

Diseño

de Sistem

as Distrib

uido


Mateo

s 48


•  Configurar variables de entorno: –  Encontrar los componentes de Hadoop

•  ~/.bashrc –  Encontrar en Hadoop a JAVA_HOME

•  /usr/local/hadoop/etc/hadoop/hadoop-‐env.sh

•  Configurar los componentes de Hadoop: –  Configurar hadoop.tmp.dir y fs.default.name

•  /usr/local/hadoop/etc/hadoop/core-‐site.xml –  Configurar qué framework usar para mapreduce

•  /usr/local/hadoop/etc/hadoop/mapred-‐site.xml –  Configuración de los directorios para namenode y datanode

•  /usr/local/hadoop/etc/hadoop/hdfs-‐site.xml

Diseño

de Sistem

as Distrib

uido


Mateo

s 49


alejandro@h1:~$ sudo mkdir –p /hadoop/tmp ; sudo chown hduser:hadoop /hadoop/tmp/

hduser@h1:/usr/local/hadoop$ cat > /usr/local/hadoop/etc/hadoop/core-‐site.xml <?xml version="1.0" encoding="UTF-‐8"?> <?xml-‐stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>hadoop.tmp.dir</name> <value>/hadoop/tmp</value> <description>A base for other temporary directories.</description> </property> <property> <name>fs.default.name</name> <value>hdfs://localhost:54310</value> <description>The name of the default file system.</description> </property> </configuration>

Diseño

de Sistem

as Distrib

uido


Mateo

s 50


hduser@h1:/usr/local/hadoop$ cat > /usr/local/hadoop/etc/hadoop/mapred-‐site.xml <?xml version="1.0"?> <?xml-‐stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>mapred.job.tracker</name> <value>localhost:54311</value> <description>The host and port that the MapReduce job tracker runs at. </description> </property> </configuration>

Diseño

de Sistem

as Distrib

uido


Mateo

s 51


alejandro@h1:~$ sudo mkdir -‐p /usr/local/hadoop_store/hdfs/namenode ; sudo mkdir -‐p /usr/local/hadoop_store/hdfs/datanode ; sudo chown -‐R hduser:hadoop /usr/local/hadoop_store

hduser@h1:/usr/local/hadoop$ cat > /usr/local/hadoop/etc/hadoop/hdfs-‐site.xml <?xml version="1.0" encoding="UTF-‐8"?> <?xml-‐stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property><name>dfs.replication</name> <value>1</value> </property> <property><name>dfs.namenode.name.dir</name> <value>file:/usr/local/hadoop_store/hdfs/namenode</value> </property> <property><name>dfs.datanode.data.dir</name> <value>file:/usr/local/hadoop_store/hdfs/datanode</value> </property> </configuration>

Diseño

de Sistem

as Distrib

uido


Mateo

s 52

Posible configuración adicional…

hTp://grepalex.com/2012/11/26/hadoop-‐shuffle-‐configurables/

Diseño

de Sistem

as Distrib

uido


Mateo

s 53


inactivo

start-‐all.sh

activo

stop-‐all.sh

hdfs namenode -‐format

inicial

<hdfs>

<mapReduce>

<monitorizar>

Diseño

de Sistem

as Distrib

uido


Mateo

s 54


inactivo

start-‐all.sh

activo

stop-‐all.sh


inicial

<hdfs>

<mapReduce>

<monitorizar>

Diseño

de Sistem

as Distrib

uido


Mateo

s 55


hduser@h1:~$ hdfs namenode -‐format 14/09/25 23:02:59 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = h1/127.0.1.1 STARTUP_MSG: args = [-‐format] STARTUP_MSG: version = 2.5.2 … 14/09/27 23:07:07 INFO blockmanagement.BlockManager: encryptDataTransfer = false 14/09/27 23:07:07 INFO namenode.FSNamesystem: fsOwner = hduser (auth:SIMPLE) … 14/09/25 23:03:04 INFO util.ExitUtil: Exiting with status 0 14/09/25 23:03:04 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at h1/127.0.1.1 ************************************************************/

Diseño

de Sistem

as Distrib

uido


Mateo

s 56


inactivo

start-‐all.sh

activo

stop-‐all.sh


inicial

<hdfs>

<mapReduce>

<monitorizar>

Diseño

de Sistem

as Distrib

uido


Mateo

s 57


hduser@h1:~$ start-‐all.sh This script is Deprecated. Instead use start-‐dfs.sh and start-‐yarn.sh 14/09/28 13:31:35 WARN util.NativeCodeLoader: Unable to load native-‐hadoop library for your platform... using builtin-‐java classes where applicable

Starting namenodes on [localhost] localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-‐hduser-‐namenode-‐h1.out localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-‐hduser-‐datanode-‐h1.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-‐hduser-‐secondarynamenode-‐h1.out 14/09/28 13:32:03 WARN util.NativeCodeLoader: Unable to load native-‐hadoop library for your platform... using builtin-‐java classes where applicable

starting yarn daemons starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-‐hduser-‐resourcemanager-‐h1.out localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-‐hduser-‐nodemanager-‐h1.out

Diseño

de Sistem

as Distrib

uido


Mateo

s 58


hduser@h1:~$ jps 28026 ResourceManager 28147 NodeManager 27877 SecondaryNameNode 27564 NameNode 28448 Jps 27683 DataNode

hduser@h1:~$ nmap localhost … PORT STATE SERVICE 22/tcp open ssh 631/tcp open ipp 8031/tcp open unknown 8042/tcp open fs-‐agent 8088/tcp open radan-‐http

Diseño

de Sistem

as Distrib

uido


Mateo

s 59


hduser@h1:~$ stop-‐all.sh This script is Deprecated. Instead use stop-‐dfs.sh and stop-‐yarn.sh 14/09/28 13:33:22 WARN util.NativeCodeLoader: Unable to load native-‐hadoop library for your platform... using builtin-‐java classes where applicable

Stopping namenodes on [localhost] localhost: stopping namenode localhost: stopping datanode Stopping secondary namenodes [0.0.0.0] 0.0.0.0: stopping secondarynamenode 14/09/28 13:33:47 WARN util.NativeCodeLoader: Unable to load native-‐hadoop library for your platform... using builtin-‐java classes where applicable

stopping yarn daemons stopping resourcemanager localhost: stopping nodemanager no proxyserver to stop

Diseño

de Sistem

as Distrib

uido


Mateo

s 60


inactivo

start-‐all.sh

activo

stop-‐all.sh


inicial

<hdfs>

<mapReduce>

<monitorizar>

•  NameNode: hTp://localhost:50070/


Diseño

de Sistem

as Distrib

uido


Mateo

s 61

•  SecondaryNameNode: hTp://localhost:50090/


Diseño

de Sistem

as Distrib

uido


Mateo

s 62

•  DataNode: hTp://localhost:50075/


Diseño

de Sistem

as Distrib

uido


Mateo

s 63

Diseño

de Sistem

as Distrib

uido


Mateo

s 64


inactivo

start-‐all.sh

activo

stop-‐all.sh


inicial

<hdfs>

<mapReduce>

<monitorizar>

Diseño

de Sistem

as Distrib

uido


Mateo

s 65


: crear un directorio hduser@h1:~$ hadoop fs -‐mkdir -‐p /user/hduser

: copiar un fichero de local a hadoop hduser@h1:~$ echo "hdfs test" > hdfsTest.txt hduser@h1:~$ hadoop fs -‐copyFromLocal hdfsTest.txt hdfsTest.txt

: ver contenido de un directorio hduser@h1:~$ hadoop fs -‐ls

: ver contenido de un archivo hduser@h1:~$ hadoop fs -‐cat /user/hduser/hdfsTest.txt

: copiar un fichero de hadoop a local hduser@h1:~$ hadoop fs -‐copyToLocal /user/hduser/hdfsTest.txt hdfsTest2.txt

: borrar un fichero hduser@h1:~$ hadoop fs -‐rm hdfsTest.txt

Diseño

de Sistem

as Distrib

uido


Mateo

s 66


hduser@h1:~$ wget http://www.gutenberg.org/files/2000/old/2donq10.txt … 2014-‐10-‐04 12:53:30 (1,10 MB/s) -‐ ‘2donq10.txt’ saved [2143292/2143292]

hduser@h1:~$ dos2unix -‐n 2donq10.txt dq.txt dos2unix: converting file 2donq10.txt to file dq.txt in Unix format ...

hduser@h1:~$ hadoop fs –copyFromLocal -‐f dq.txt /user/hduser/dq.txt

hduser@h1:~$ hadoop fs -‐ls /user/hduser … Found 1 items -‐rw-‐r-‐-‐r-‐-‐ 3 hduser supergroup 2143292 2014-‐10-‐04 13:09 /user/hduser/dq.txt

Diseño

de Sistem

as Distrib

uido


Mateo

s 67


inactivo

start-‐all.sh

activo

stop-‐all.sh


inicial

<hdfs>

<mapReduce>

<monitorizar>

Diseño

de Sistem

as Distrib

uido


Mateo

s 68


Na#vo Encapsulado

Java Perl, Python, …

Diseño

de Sistem

as Distrib

uido


Mateo

s 69


hduser@h1:~$ hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-‐mapreduce-‐examples-‐2.5.2.jar pi 2 5

Number of Maps = 2 Samples per Map = 5 … Job Finished in 11.536 seconds Estimated value of Pi is 3.60000000000000000000

hTp://www.bogotobogo.com/Hadoop/BigData_hadoop_Running_MapReduce_Job.php

Diseño

de Sistem

as Distrib

uido


Mateo

s 70


package org.myorg; import java.io.IOException; import java.util.*; import org.apache.hadoop.fs.Path; import org.apache.hadoop.conf.*; import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.*; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

hTp://wiki.apache.org/hadoop/WordCount

1

Diseño

de Sistem

as Distrib

uido


Mateo

s 71


public class WordCount { public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map (LongWritable key, Text value, Context context)

throws IOException, InterruptedException { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } }


2

Diseño

de Sistem

as Distrib

uido


Mateo

s 72


public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce (Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } context.write(key, new IntWritable(sum)); } }


3

Diseño

de Sistem

as Distrib

uido


Mateo

s 73


public static void main (String[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf, "wordcount"); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.waitForCompletion(true); } } // class WordCount


4

Diseño

de Sistem

as Distrib

uido


Mateo

s 74


hduser@h1:/usr/local/hadoop$ hadoop jar share/hadoop/mapreduce/hadoop-‐mapreduce-‐examples-‐*.jar wordcount /user/hduser/dq.txt /user/hduser/counterj

14/10/04 16:33:36 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 14/10/04 16:33:37 INFO input.FileInputFormat: Total input paths to process : 1 14/10/04 16:33:37 INFO mapreduce.JobSubmitter: number of splits:1 14/10/04 16:33:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local835374884_0001 …

File Input Format Counters Bytes Read=2106143 File Output Format Counters Bytes Written=454722

hduser@h1:/usr/local/hadoop$ hadoop fs -‐cat /user/hduser/counterj/* | sort -‐n -‐k 2 -‐r|head -‐5

… que 19429 de 17986 y 15887 la 10199 a 9502

hTp://www.michael-‐noll.com/tutorials/running-‐hadoop-‐on-‐ubuntu-‐linux-‐single-‐node-‐cluster/

Diseño

de Sistem

as Distrib

uido


Mateo

s 75


Na#vo Encapsulado

Java Perl, Python, …

Diseño

de Sistem

as Distrib

uido


Mateo

s 76

Hadoop Streaming API

hTp://www.michael-‐noll.com/tutorials/wri<ng-‐an-‐hadoop-‐mapreduce-‐program-‐in-‐python/

mapper.sh

en 1 un 1 lugar 1 …

En un lugar… STDOUT STDIN

awk '{i=1; while (i<=NF) {gsub(/[\.,;]/,"",$i); print tolower($i)" "1; i++;}}'

Diseño

de Sistem

as Distrib

uido


Mateo

s 77

Hadoop Streaming API

hTp://www.michael-‐noll.com/tutorials/wri<ng-‐an-‐hadoop-‐mapreduce-‐program-‐in-‐python/

reducer.sh


sed 's/ 1$//g' |uniq -‐c| awk '{print $2" "$1}'|sed 's/^$//g'


STDOUT STDIN

Diseño

de Sistem

as Distrib

uido


Mateo

s 79


hduser@h1:/usr/local/hadoop$ hadoop jar share/hadoop/tools/lib/hadoop-‐streaming-‐2.5.2.jar -‐file ./mapper.sh -‐mapper ./mapper.sh -‐file ./reducer.sh -‐reducer ./reducer.sh -‐input /user/hduser/ -‐output /user/hduser/counter

packageJobJar: [./mapper.sh, ./reducer.sh] [] /tmp/streamjob724842872862965882.jar tmpDir=null 14/10/04 15:48:02 INFO Configuration.deprecation: session.id is deprecated. Instead, use

dfs.metrics.session-‐id …

File Input Format Counters Bytes Read=2106143 File Output Format Counters Bytes Written=320124

14/10/04 15:48:46 INFO streaming.StreamJob: Output directory: /user/hduser/counter

hduser@h1:/usr/local/hadoop$ hadoop fs -‐cat /user/hduser/counter/part-‐00000|sort -‐n -‐k 2 -‐r|head -‐5

… que 20545 de 18154 y 18053 la 10338 a 9779

hTp://hadoop.apache.org/docs/r1.1.2/streaming.html#Hadoop+Streaming

Diseño

de Sistem

as Distrib

uido


Mateo

s 80

Contenidos


–  Ecosistema


Diseño

de Sistem

as Distrib

uido


Mateo

s 81

Benchmarking

hTp://m.oschina.net/blog/74201

•  TestDFSIO

•  TeraSort benchmark suite –  Yahoo! 2009: 1 PB de datos en 16 horas

•  NameNode benchmark (nnbench)

•  MapReduce benchmark (mrbench)

Diseño

de Sistem

as Distrib

uido


Mateo

s 82

TeraSort (2009, 500GB)

hTps://developer.yahoo.com/blogs/hadoop/hadoop-‐sorts-‐petabyte-‐16-‐25-‐hours-‐terabyte-‐62-‐422.html

Diseño

de Sistem

as Distrib

uido


Mateo

s 83

Contenidos


–  Ecosistema


Diseño

de Sistem

as Distrib

uido


Mateo

s 84

Ecosistema

hTp://ambuj4bigdata.blogspot.com.es/2014_05_01_archive.html

Diseño

de Sistem

as Distrib

uido


Mateo

s 85

Ecosistema


Diseño

de Sistem

as Distrib

uido


Mateo

s 86

Almacenamiento

hTp://www.nextree.co.kr/p2865/

Diseño

de Sistem

as Distrib

uido


Mateo

s 87

Procesamiento


Diseño

de Sistem

as Distrib

uido


Mateo

s 88

Ecosistema


Diseño

de Sistem

as Distrib

uido


Mateo

s 89

CDH (Cloudera Hadoop Distribu<on)

hTp://www.theregister.co.uk/2012/06/05/cloudera_cdh4_hadoop_stack/

Diseño

de Sistem

as Distrib

uido


Mateo

s 90

AEMR (Amazon Elas<c MapReduce)

hTp://aws.amazon.com/es/elas<cmapreduce/

Diseño

de Sistem

as Distrib

uido


Mateo

s 91

Ecosistema


Diseño

de Sistem

as Distrib

uido


Mateo

s 92

Ecosistema (funcionalidad)

hTp://bigdataconsultants.blogspot.com.es/2013/09/big-‐data-‐glossary.html hTps://www.joyent.com/content/08-‐company/03-‐press/105-‐the-‐hadoop-‐ecosystem-‐the-‐welcome-‐elephant-‐in-‐the-‐room/hadoop-‐infographic-‐gigaom.jpg

Diseño

de Sistem

as Distrib

uido


Mateo

s 93

Ecosistema (estructura)


Diseño

de Sistem

as Distrib

uido


Mateo

s 94

Almacenamiento


Diseño

de Sistem

as Distrib

uido


Mateo

s 95

Procesamiento


purchases = LOAD "/data/purchases" AS (id, client_id, product_id); bigpurchases = FILTER purchases BY price > 1000; …

SELECT * FROM purchases WHERE price > 1000 ORDER BY client_id

Diseño

de Sistem

as Distrib

uido


Mateo

s 96

Almacenamiento: Integración


Diseño

de Sistem

as Distrib

uido


Mateo

s 97

Procesamiento: orquestación


Diseño

de Sistem

as Distrib

uido


Mateo

s 98

Ej.: Flume-‐Hive-‐oozie

hTp://www.datadansandler.com/2013/03/big-‐data-‐and-‐hadoop-‐assets-‐available-‐on.html

Diseño

de Sistem

as Distrib

uido


Mateo

s 99

Ecosistema


Diseño

de Sistem

as Distrib

uido


Mateo

s 100

•  Velocidad –  Ejecuta 100 veces más rápido en memoria, 10 veces más rápido en disco

•  Facilidad de uso –  En Java, Scala, Python

•  Polivalente –  Combina SQL, streaming, analí<co… –  Trabaja con Hadoop, HBase, S3, …

hTps://spark.apache.org/

Google Dataflow Diseño

de Sistem

as Distrib

uido


Mateo

s 101

•  Data pipelines for helping to ingest, transform and, analyze data.

hTp://techcrunch.com/2014/06/25/google-‐launches-‐cloud-‐dataflow-‐a-‐managed-‐data-‐processing-‐service/

Diseño

de Sistem

as Distrib

uido


Mateo

s 102

Contenidos


–  Ecosistema


Diseño

de Sistem

as Distrib

uido


Mateo

s 103

Bibliogra�a: tutoriales

•  Página Web oficial: –  hTp://hadoop.apache.org/

•  Introducción a cómo funciona Hadoop: –  hTp://blog.csdn.net/suifeng3051/ar<cle/details/17288047

•  Tutorial de cómo instalar y usar Hadoop: –  hTp://www.bogotobogo.com/Hadoop/

BigData_hadoop_Install_on_ubuntu_single_node_cluster.php –  hTp://www.bogotobogo.com/Hadoop/

BigData_hadoop_Running_MapReduce_Job.php

Diseño

de Sistem

as Distrib

uido


Mateo

s 104

Bibliogra�a: libro

•  Hadoop: The Defini<ve Guide, 3rd Edi<on: –  hTp://shop.oreilly.com/product/0636920021773.do –  hTps://github.com/tomwhite/hadoop-‐book/

Diseño

de Sistem

as Distrib

uido


Mateo

s 105

Bibliogra�a: TFG

•  Extracción de información social desde TwiTer y análisis mediante Hadoop. –  Autor: Cris<an Caballero Mon<el –  Tutores: Daniel Higuero Alonso-‐Mardones y

Juan Manuel Tirado Mar�n –  hTp://e-‐archivo.uc3m.es/handle/10016/16784

•  Adapta<on, Deployment and Evalua<on of a Railway Simulator in Cloud Environments –  Autora: Silvina Caíno Lores –  Tutor: Alberto García Fernández

Diseño

de Sistem

as Distrib

uido


Mateo

s 106

Agradecimientos

•  Por úl<mo pero no por ello menos importante, agradecer al personal del Laboratorio del Departamento de Informá<ca todos los comentarios y sugerencias para esta presentación.

Diseño de Sistemas Distribuidos Máster en Ciencia y Tecnología Informá<ca

Curso 2014-‐2015

Alejandro Calderón Mateos & Óscar Pérez Alonso Grupo de Arquitectura de Computadores [email protected]

Sistemas escalables en entornos distribuidos

Ejemplo de sistemas escalables en entornos distribuidos: Hadoop (v2d)

Education

Transcript of Ejemplo de sistemas escalables en entornos distribuidos: Hadoop (v2d)