Integration of LVM with HADOOP

• What is LVM?

LVM is a tool for logical volume management which includes allocating disks, striping, mirroring and resizing logical volumes.

With LVM, a hard drive or set of hard drives is allocated to one or more physical volumes. LVM physical volumes can be placed on other block devices which might span two or more disks.

The physical volumes are combined into logical volumes, with the exception of the /boot partition. The /boot partition cannot be on a logical volume group because the boot loader cannot read it. If the root (/) partition is on a logical volume, create a separate /boot partition which is not a part of a volume group.

Step-1: Attach a Hard Disks of any size

First we have to attach Hard disks and to do so we have to go in the Settings of our VM and under storage we find an option of Add hard disk. Now create an Hard disk by clicking the create button and give the size to the hard disk. Attach the Hard disk,In my case I have created the Hard Disks of 10Gib. Now that the Disks are attach we are good to go.Power on the OS.

Step-2 Create a Logical volume

After the OS is power on check whether the disk is successfully attached or not and hence to do so use the below command:

fdisk -l

We can see that the disk is successfully connected of 10GiB.

Now we will create a physical volume [PV].To create a Physical volume use the following command:

pvcreate /dev/sdb

Now that we have successfully created the physical volume we will now create a volume group [VG]. Give a relevent name to the volume group so that it is easily identified.Use the below command to create a volume group:

vgcreate name /dev/xvd1

Now we have successfully created a Volume group.

We can use the below command to create a logical volume,

lvcreate --size 9G --name (any name) vg-name

Now it is the time to format the Hard disk to do so we have to use the below mentioned command:

mkfs.ext4 /dev/xvdf1

Now we have to mount the Logical volume created.First we have to make a directory in the root folder.Use the following command to make a directory

mkdir /folder
ls /

Now that the directory is created we can mount the logical volume for this we have use a command and to check if logical volume is correctly mounted use the df -hT command as below:

mount /dev/xvdf1 /folder
df -hT

Step-3: Create a Hadoop slave node

Now we have to create a slave node for that we have to go in the the /etc/hadoop directory and make the require changes in the hdfs-site.xml and core-site.xml files respectively.

Now start the datanode services by using the below commands as below:

hadoop-daemon.sh start datanode
jps

Now switch to the master node and run the the command below to know if the slave is connected to the master and see if the storage is provided by the logical volume:

hadoop dfsadmin -report

Hence by the output we will see that slave node is successfully connected to the master node and is providing 8.8GB storage which is approximately equal to that of the logical volume that we have created which was of 10GB respectively.

Increase or Decrease the Size of Static Partition in Linux.

Now we can increase or decrease the size of the static partation by following the commands below in the images:

Hence, size of the partation can increased or decreased accordingly.