Elasticsearch and Log Data Disk Migration
Disk Preparation and Mount Configuration
Scan for newly attached disks
Use following three commands at once instead of using one by one.
for host in `ls /sys/class/scsi_host/`;do
echo "- - -" >/sys/class/scsi_host/${host}/scan;
done
Forces the system to rescan all SCSI hosts to detect newly attached storage disks.
List all disks and partitions
lsblk
Displays block devices and their mount points to identify the new disks.
Hot Data Volume (for Elasticsearch)
pvcreate /dev/sdb
Initializes the new disk /dev/sdb as a Physical Volume for LVM.
It might not always be sdb or sdc. Edit here with care.
pvs
Lists all existing Physical Volumes to confirm creation.
vgcreate hotdata-vg /dev/sdb
Creates a Volume Group named hotdata-vg using the new PV.
vgs
Lists all Volume Groups to verify creation.
lvcreate -L 100GB -n hotdata-lvm hotdata-vg
Creates a Logical Volume named hotdata-lvm with 100GB size inside hotdata-vg. ℹ 100GB here is symbolic. After the next command full disk space is going to be used.
lvextend -l +100%FREE /dev/hotdata-vg/hotdata-lvm
Extends the logical volume to use all remaining free space in the volume group.
lvs
Lists all Logical Volumes to confirm size and status.
mkdir /hot_data
Creates the mount point directory.
mkfs.ext4 /dev/hotdata-vg/hotdata-lvm
Formats the logical volume with the EXT4 filesystem.
mount /dev/hotdata-vg/hotdata-lvm /hot_data
Mounts the filesystem to /hot_data.
df -h
Verifies that the mount was successful.
nano /etc/fstab
Opens the fstab file to make the mount persistent across reboots.
Add this line:
/dev/mapper/hotdata--vg-hotdata--lvm /hot_data ext4 defaults 0 1
Use Ctrl + X then hit Enter.
mount -a df -h
Tests and verifies that the fstab entry works properly.
Cold Data Volume (for Archive & Signed logs)
Raw Persist and Parsed Persist services are scheduled to work after midnight. Do not do these steps around this time.
pvcreate /dev/sdc
It might not always be sdb or sdc. Edit here with care.
pvs
vgcreate colddata-vg /dev/sdc
vgs
lvcreate -L 100GB -n colddata-lvm colddata-vg
lvextend -l +100%FREE /dev/colddata-vg/colddata-lvm
lvs
mkdir /cold_data
mkfs.ext4 /dev/colddata-vg/colddata-lvm
mount /dev/colddata-vg/colddata-lvm /cold_data
df -h
nano /etc/fstab
Add this line:
/dev/mapper/colddata--vg-colddata--lvm /cold_data ext4 defaults 0 1
Use Ctrl + X then hit Enter.
mount -a
df -h
Move and Symlink Elasticsearch Directory
systemctl stop elasticsearch
Stops Elasticsearch service before moving its data directory.
mv /var/lib/elasticsearch /hot_data/elasticsearch
Moves the existing Elasticsearch data directory to the new hot data disk.
ln -s /hot_data/elasticsearch /var/lib/elasticsearch
Creates a symbolic link so that /var/lib/elasticsearch now points to
/hot_data/elasticsearch.
systemctl start elasticsearch
Restarts Elasticsearch with the new data path.
Move and Symlink Archive & Signed Directories
mv /opt/var/log/archive /cold_data/archive
mv /opt/var/log/signed /cold_data/signed
Moves both directories to the cold data disk.
ln -s /cold_data/archive /opt/var/log/archive
ln -s /cold_data/signed /opt/var/log/signed
Creates symlinks so applications still write to the old paths, but data is stored on /cold_data.