- September 3, 2015
- Posted by: Surender Kumar
- Category: Linux
Disk Storage Management in Linux
Table of Contents
The hard disk is a very important hardware resource in every computer since it stores operating system as well as all user data. How to manage them effectively is directly related to performance of the whole system. In Windows there is a utility called disk management which allows the administrator to create and manage the partition on hard disk drive. Similarly, Linux also gives some tools which you can use to manage the disk.
What is a Partition?
Partition is a logical container which is used to house filesystems, where operating systems, applications, and data is stored on. A single partition may span the entirety of a physical storage device.
What is a File System?
A file system provides a way of separating the data on the drive into individual pieces, which are the files. It also provides a way to store data about these files for example, their filenames, permissions, and other attributes. The file system also provides an index which is a list of the files on the drive and where they are located on the drive, so the operating system can see what’s on the drive in one place rather than combing through the entire drive to find a file.
Your operating system needs to understand a file system so it can display its contents, open files, and save files to it. If your operating system doesn’t understand a file system, you may be able to install a file system driver that provides support or you just can’t use that file system with that operating system.
Why Partition your Disk?
Consider that your company is moving into a new building that is setup like a warehouse. There are no internal walls, it is just one big room for everybody to work in. The first thing the boss is going to do is build an office for himself (he wouldn’t want to associate with you workers). So, he will have a partition set up, an internal wall to separate him from you. Next, each worker will get a cabin in the big room with smaller partitions to separate you from the person next to you.
Hard drives work the same way. The disk gets partitioned into smaller, separate pieces which can belong to different owners. In our case, Windows might own some and Linux will own others. On hard drives, the word “partition” does not refer to the wall itself, it refers to the logically separated space.
After creating a partition, the partition is formatted with a file system — like the NTFS file system on Windows drives, FAT32 file system for removable drives, HFS+ file system on Mac computers, and the ext3, ext4 file system on Linux. Files are then written to that file system on the partition.
Benefits of Multiple Partitions
Creating more than one partition has the following advantages:
- Separation of the operating system and program files from user files. This allows image backups (or clones) to be made of only the operating system and installed software.
- Having a separate area for operating system virtual memory swapping/paging.
- Keeping frequently used programs and data near each other.
- Having cache and log files separate from other files. These can change size dynamically and rapidly, potentially making a file system full.
- Multiple partitions allow user to install multiple operating systems in one computer. For example, one could install Linux, Mac OS X, Microsoft Windows etc. on different partitions of the same HDD and have a choice of booting into any compatible operating system at power-up.
- Protecting or isolating files, to make it easier to recover a corrupted file system or operating system installation. If one partition is corrupted, other file systems may not be affected.
- Raising overall computer performance on systems where smaller file systems are more efficient. For instance, large HDDs with only one NTFS file system typically have a very large sequentially accessed Master File Table (MFT) and it generally takes more time to read this MFT than the smaller MFTs of smaller partitions.
Partitioning is good however it also has some disadvantages like increased fragmentation and wasted disk space. The more partitions you create, the more usable disk space you waste. So it is always a good idea to create 2-3 partitions if you have only one operating system installed.
On Windows, the disk partitioning scheme is pretty clearly cut: OS lives on your disk, usually on one partition, and that’s all. If you have other drives, and they have a compatible file system, then it will read them as well. If not, it will usually ignore them, or offer you the ability to reformat. But in Linux and Unix world, it doesn’t work that way.
The way Linux works is that it puts everything onto a hierarchical tree. If you have another partition or disk, it gets “mounted” as a branch in a specific folder, usually /media or /mnt. The directory that a partition gets mounted to is called a “mount point”. This method works better with Linux’s tree structure, and you can mount partitions as folders nearly anywhere. In Windows, this is not so easily done; new partitions generally show up as separate drives. In addition, Linux can work with many more types of file systems natively than Windows.
The older hard disks use sectors of 512-byte size and the MBR (Master Boot Record) partition table. The limitation of MBR disk is that it allows 4 primary partitions only on one disk. If you want more partitions, you need to create an extended partition which reduces the number of primary partitions to 3, and then create logical partitions inside extended partition. The limit of MBR based disks is that it does not support drives of more than 2TB capacity. If you have a drive with 4TB capacity, you can only use 2TB of space in MBR partitioning scheme or you have to convert the disk to GPT.
To address the limitations of MBR disk, the GPT (GUID Partition Table) is introduced which uses the sectors of 4096-byte.
Disk Partitioning Scheme
There are no strict rules for partitioning a hard drive, although one may follow the general guidance given below. A disk partitioning scheme is determined by various factors such as desired flexibility, speed, security, multi-boot requirement as well as the limitations imposed by available disk space. It is essentially personal preference.
I will favor to stick with the default partitioning scheme especially for beginners or if you are installing Linux on your desktop for personal use. If you are installing a server that will serve multiple users simultaneously and provide a lot of services, you might need to create the seven or more partitions. But for a healthy Linux installation, I would recommend four partitions: root, boot, swap, and home.
root (/) Partition
The root file system is represented by a forward slash (/). It is the top of the directory tree, and contains Linux and everything that you install with Linux. This is roughly equivalent to your “C:” drive under Windows. You must create a partition for the root directory.
The size of your root partition will vary depending on what you install or plan to install. The best bet is to check your distribution’s documentation, and reserve enough space for a maximum installation, plus at least 1GB more for temporary space and installation of new software. If you plan to download and try out lots of software, leave more space. If you have a small hard drive, you can trim back on your installed packages to save space. Since hard drives are getting massive and cheap these days, this should not be a problem. Isn’t it?
A 3.0 GB root (/) partition allows you to install a minimal installation, while a 5.0 GB root partition lets you perform a full installation, choosing all package groups but I would recommend 10 – 15 GB. I know it sounds a lot but opting for a really small size for root partition can give problems later. The /root directory is the home directory of the user account for system administration.
/boot Partition
The /boot partition contains the operating system kernel which allows your system to boot the Linux, along with files used during the bootstrap process. For most users, a 250 MB boot partition is sufficient. Note that normally the /boot partition is created automatically by the installer. However, if the /root partition is larger than 2 TB and UEFI is used for booting, you need to create a separate /boot partition that is smaller than 2 TB to boot the machine successfully.
Swap Partition
You need one partition that will be used as Linux swap space. This is space on your hard drive that can be used as virtual memory. Virtual memory allows your computer to run large programs and perform complex tasks even if it does not have enough physical RAM to do the job. It is lot slower than physical RAM, but it still works.
Historically, the general rule for swap partition size was to allocate twice the amount of physical RAM. But as computers have now gained larger memory capacities, this rule is outdated. The basic idea is that if you are short on RAM with plenty of drive space, go large. If you’re short on drive space and have lots of RAM, go small.
/home Partition
This partition will hold /home directory of your Linux system. This is the place where all the user files are stored. It is roughly equivalent to the “My Documents” folder on a Microsoft Windows desktop. Each user will have her own subdirectory under /home.
Frankly speaking, it is not necessary to create a separate partition for /home. If you do not create, it will reside on the root partition like everything else.
But I will recommend creating a separate /home partition. Whether you are new or experienced user of Linux, you will be going to play with files and make some experiments. And as you know in Linux, everything is a file; if not, it is a process. In your experiments you might break something so badly that you will need to reinstall the operating system, or you might want to reinstall in order to try a different Linux distribution. Having a separate /home partition makes it very easy to wipe out and reinstall Linux without losing any of your data.
Creating Disk Partitions
Linux consider every thing as files. For example if there is one hard disk in a Linux system then it will be represented as hda (harddisk ‘A’). The second hard disk will be represented as hdb (harddisk ‘B’). The CD drive is represented as cdrom. Modern SATA and USB drives are represented like sda, sdb etc. Linux stores every device in /dev directory. So, the absolute path for first SATA hard drive will be /dev/sda. The path to cdrom will be /dev/cdrom. The first partition will be represented as /dev/sda1, the second as /dev/sda2, and so on.
You can use the fdisk command line utility to create a new partition and to do other disk management operations. In GUI, disk partitioning in Linux is similar to that of Disk Management in Microsoft Windows. If you have experience, Windows also had fdisk utility based on MS DOS but that was not as much powerful as fdisk utility found in Linux.
To list the current Partition Table of disk, use fdisk -l command.
[root@centos ~]# fdisk -l Disk /dev/sda: 64.4 GB, 64424509440 bytes, 125829120 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: gpt # Start End Size Type Name 1 2048 411647 200M EFI System EFI System Partition 2 411648 1435647 500M Microsoft basic 3 1435648 1845247 200M Microsoft basic 4 1845248 83765247 39.1G Microsoft basic 5 83765248 88025087 2G Linux swap [output cut]
The above command will list all the partitions on disk. If you have multiple hard disk drives in your system you can add the name of hard drive at the end to list the partition information for specific drive and the command will look like fdisk -l /dev/sdb.
To create partition on a hard drive, use fdisk /dev/sda command, where sda is your drive name. Once you enter this command, it will enter its own configuration mode which provide you different options to perform various functions like create new partition, delete existing partition etc.
- Start fdisk using the following command:
fdisk /dev/sda
where
/dev/sda
stands for the hard drive that you want to partition. - In fdisk, to create a new partition, type the n option:
Command (m for help): n Partition type: p primary (1 primary, 0 extended, 3 free) e extended Select (default p): Using default response p. Partition number (2-4, default 2): First sector (1026048-854745087, default 1026048): *Enter* Last sector, +sectors or +size{K,M,G,T,P} (1026048-854745087, default 854745087): +500M Created a new partition 2 of type 'Linux' and of size 500 MiB.
- When prompted to specify the Partition type, type p to create a primary partition or e to create an extended one. There may be up to four primary partitions. If you want to create more than four partitions, make the last partition extended, and it will be a container for other logical partitions.
- When prompted for the Number, in most cases, type 2 because a typical Linux partition type is 2.
- When prompted for the start cylinder, type a starting cylinder number or press Return to use the first cylinder available.
- When prompted for the last cylinder, press Return to allocate all the available space or specify the size of a new partition in cylinders if you do not want to use all the available space. for example, for 500MB, type + 500M and press Enter. You can use +10G for 10GB, +50G for 50GB and so on.
By default, fdisk creates a partition with a System ID 83. If you’re unsure of the partition’s System ID, use the
l
command to check it.
- Use the
w
command to write the changes to the partition table.
- Restart the virtual system by entering the
reboot
command.
- When restarted, create a file system on the new partition. We recommend that you use the same file system as on the other partitions. In most cases it will be either the ext3 or ext4 file system. For example, to create the ext3 file system on new partition /dev/sda3, enter the following command:
mkfs -t ext3 /dev/sda3
- Create a directory that will be a mount point for the new partition. For example, create a new directory with the name /data:
mkdir /data
- Mount the new partition to the directory you have just created by using the following command:
mount /dev/sda3 /data
- Edit the /etc/fstab file using vim /etc/fstab command. The fstab is a configuration file that contains information of all the partitions and storage devices in your computer and every partition gets mounted automatically by reading the configuration information from this file when system is started.
Add the following line at the end of file:/dev/sda3 /data ext3 defaults 0 0
The /dev/sda3 is the partition you have just created, /data is a mount point for the new partition, ext3 is the file type of the new partition.
- Save the /etc/fstab file.
The fdisk utility works well with MBR based disks but it doesn’t handle GPT partition tables. To work with GPT based disk, you can use another tool known as parted.
Create Partitions with parted Utility
To start the parted utility, type the parted command.
[root@centos ~]# parted GNU Parted 3.1 Using /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted)
To view the list of all available commands, type help and press Enter.
(parted) help
align-check TYPE N check partition N for TYPE(min|opt)
alignment
help [COMMAND] print general help, or help on
COMMAND
mklabel,mktable LABEL-TYPE create a new disklabel (partition
table)
mkpart PART-TYPE [FS-TYPE] START END make a partition
name NUMBER NAME name partition NUMBER as NAME
print [devices|free|list,all|NUMBER] display the partition table,
available devices, free space, all found partitions, or a particular
partition
quit exit program
rescue START END rescue a lost partition near START
and END
rm NUMBER delete partition NUMBER
select DEVICE choose the device to edit
disk_set FLAG STATE change the FLAG on selected device
disk_toggle [FLAG] toggle the state of FLAG on selected
device
set NUMBER FLAG STATE change the FLAG on partition NUMBER
toggle [NUMBER [FLAG]] toggle the state of FLAG on partition
NUMBER
unit UNIT set the default unit to UNIT
version display the version number and
copyright information of GNU Parted
(parted)
To list all the disks and partitions, type print all and press Enter.
(parted) print all Model: Msft Virtual Disk (scsi) Disk /dev/sda: 64.4GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 211MB 210MB fat16 EFI System Partition boot 2 211MB 735MB 524MB xfs 3 735MB 945MB 210MB xfs 4 945MB 42.9GB 41.9GB xfs 5 42.9GB 45.1GB 2181MB linux-swap(v1) 6 45.1GB 47.2GB 2147MB linux-swap(v1) 7 47.2GB 64.4GB 17.2GB xfs 12 64.4GB 64.4GB 1032kB Error: /dev/sdb: unrecognised disk label Model: Msft Virtual Disk (scsi) Disk /dev/sdb: 85.9GB Sector size (logical/physical): 512B/512B Partition Table: unknown Disk Flags:
Notice that I have two disks. First is /dev/sda with GPT partition table which has operating system installed. I will not mess up with this one. The other drive is /dev/sdb which is showing as unrecognised disk label and Partition table is unknown. For this lab, I will create partitions on /dev/sdb.
To select the /dev/sdb disk, type select /dev/sdb command
(parted) select /dev/sdb Using /dev/sdb (parted)
To create a GPT partition table on the /dev/sdb disk, type mktable gpt command. If you want to use MBR partition table, use mktable msdos command.
(parted) mktable gpt
(parted)
Since this was new unrecognised disk, I did not get any error or warning. If the disk was to be any recognised one, I would have got the following Warning:
Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? y
In this case, type y and hit Enter if you wish to format the disk and destroy all the data.
Now, use print command to list the disk information.
(parted) print
Model: Msft Virtual Disk (scsi)
Disk /dev/sdb: 85.9GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
Notice that disk is now showing as Msft Virtual Disk (scsi) with GPT Partition Table.
To create a primary partition with the ext4 type (starting at 1MB and finishing at 10GB), type mkpart primary ext4 1MB 10GB command.
(parted) mkpart primary ext4 1MB 10GB
Note1: Specifying ext4 doesn’t format the partition with ext4 file system, it only tags it as ext4 partition.
Note2: The partition doesn’t start at 0 but 1MB to avoid disk alignment problems.
Note3: To specify all the remaining space, use -1 as end position.
Note4: In parted, 1GB=1000MB.
To create a swap partition with a size of 2GB (here starting at 10GB and finishing at 12GB), type the following command:
(parted) mkpart primary linux-swap 10GB 12GB
Note1: parted checks that both partitions don’t overlap.
Note2: If, at a later stage, you want to change the type of partition, don’t drop and recreate the partition: format the partition as you want and parted will normally detect the new type.
To print the new partition information use print command.
(parted) print Model: Msft Virtual Disk (scsi) Disk /dev/sdb: 85.9GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 10.0GB 9999MB primary 2 10.0GB 12.0GB 2000MB primary (parted)
To set the first partition as bootable, type
(parted) set 1 boot on (parted) print Model: Msft Virtual Disk (scsi) Disk /dev/sdb: 85.9GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 10.0GB 9999MB primary boot 2 10.0GB 12.0GB 2000MB primary (parted)
To remove the swap partition (partition number 2), type rm 2 command as shown below:
(parted) rm 2 (parted) print Model: Msft Virtual Disk (scsi) Disk /dev/sdb: 85.9GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 10.0GB 9999MB primary boot
To exit the parted prompt, type q and press Enter.
The changes you have made to disk are not seen by Linux kernel yet. So, you need to run the following command:
[root@centos ~]# partprobe /dev/sdb
Now create a file system on the new partition. We recommend that you use the same file system as on the other partitions. In most cases it will be either the ext3 or ext4 file system. For example, to create the ext4 file system on new partition /dev/sdb1, enter the following command:
[root@centos ~]# mkfs.ext4 /dev/sdb1
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
610800 inodes, 2441216 blocks
122060 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2151677952
75 block groups
32768 blocks per group, 32768 fragments per group
8144 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
Partition is now formatted with ext4. You need to create a directory that will be a mount point for the new partition. For example, create a new directory with the name /project:
[root@centos ~]# mkdir /project
Mount the new partition to the directory you have just created by using the following command:
[root@centos ~]# mount /dev/sdb1 /project
Edit the /etc/fstab file using vim /etc/fstab command to allow kernel to read the mount point information at boot. Add the following line at the end of file:
/dev/sdb1 /project ext4 defaults 0 0
The /dev/sdb1 is the partition you have just created, /project is a mount point for the new partition, ext4 is the file type of the new partition.
Now save the /etc/fstab file.
The newly created partition can now be used on /project directory. Whatever you will store into /project directory will go into /dev/sdb1 partition.
Monitoring Disk Usage
Linux also gives you the tools which can help you monitor the disk usage and do the proactive disk maintenance tasks. These tools are “df” and “du”.
1. df
The df command can get how much space is occupied and how much is available space left, it can also display the usage information for both inode and disk blocks. The following are most commonly used options with df command:
-h, --human-readable print sizes in human readable format (e.g., 1K 234M 2G) -H, --si likewise, but use powers of 1000 not 1024 -i, --inodes list inode information instead of block usage -k like --block-size=1K -t, --type=TYPE limit listing to file systems of type TYPE -T, --print-type print file system type
Example: The following output shows the use of df command without any option:
[root@centos ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda7 16793600 10073968 6719632 60% /
devtmpfs 495872 0 495872 0% /dev
tmpfs 504168 80 504088 1% /dev/shm
tmpfs 504168 6916 497252 2% /run
tmpfs 504168 0 504168 0% /sys/fs/cgroup
/dev/sda3 201388 93964 107424 47% /boot
/dev/sda1 204580 9760 194820 5% /boot/efi
/dev/sdb1 9480384 6198296 2777464 70% /project
[root@centos ~]#
Notice that the command gives you the 6 column output such as Filesystem, 1K-Blocks (Size), Used, Available, Use% and Mounted on. Everything seems OK except Used and Available space is showing in Bytes which is hard to figure out for everyone. I mean you have to manually convert these values to MB or GB everytime you will run this command. Fortunately, df command gives you -h option so that the size will be shown in human readable format.
[root@centos ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda7 17G 9.7G 6.5G 60% / devtmpfs 485M 0 485M 0% /dev tmpfs 493M 80K 493M 1% /dev/shm tmpfs 493M 6.8M 486M 2% /run tmpfs 493M 0 493M 0% /sys/fs/cgroup /dev/sda3 197M 92M 105M 47% /boot /dev/sda1 200M 9.6M 191M 5% /boot/efi /dev/sdb1 9.1G 6.0G 2.7G 70% /project
Notice that df -h command gives us the more comfortable snapshot of disk usage. You can read the Size, Used space and Available space in MB and GB for every partition which is pretty nice. One thing that you may consider missing is actual name of file system. To list the name of file system as well, use df -Th command.
[root@centos ~]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/sda7 xfs 17G 9.7G 6.5G 60% / devtmpfs devtmpfs 485M 0 485M 0% /dev tmpfs tmpfs 493M 80K 493M 1% /dev/shm tmpfs tmpfs 493M 6.8M 486M 2% /run tmpfs tmpfs 493M 0 493M 0% /sys/fs/cgroup /dev/sda3 xfs 197M 92M 105M 47% /boot /dev/sda1 vfat 200M 9.6M 191M 5% /boot/efi /dev/sdb1 ext4 9.1G 6.0G 2.7G 70% /project
This time yo can also see the type of file system for every partition. Notice the file system of partition /dev/sdb1 is ext4 which we have previously created using parted tool.
2. du
The du is the abbreviation of “disk usage”, this command will display how many blocks are occupied by looking progressively into each subdirectory of the specified directory. If not given a specified directory, then will show the statistics of the current directory.
The following are most commonly used options with du command:
-a, --all write counts for all files, not just directories -b, --bytes equivalent to '--apparent-size --block-size=1' -c, --total produce a grand total -h, --human-readable print sizes in human readable format (e.g., 1K 234M 2G) -k like --block-size=1K -l, --count-links count sizes many times if hard linked -m like --block-size=1M
For example to check the disk space usage for /project directory, use the following command:
[root@centos ~]# du -h /project/ 16K /project/lost+found 23M /project/RPMS/repodata 5.9G /project/RPMS 5.9G /project/
This command listed the disk usage for every subdirectory under /project directory. If you want to view the usage including size of every file, use the following command:
[root@centos ~]# du -ah /project/ 16K /project/lost+found 72K /project/RPMS/libacl-devel-2.2.51-12.el7.i686.rpm 164K /project/RPMS/apache-rat-javadoc-0.8-13.el7.noarch.rpm 20K /project/RPMS/texlive-multido-svn18302.1.42-32.el7.noarch.rpm 32K /project/RPMS/libmusicbrainz-devel-2.1.5-17.el7.i686.rpm 176K /project/RPMS/ibus-devel-1.5.3-11.el7.i686.rpm 120K /project/RPMS/sblim-smis-hba-1.0.0-10.el7.x86_64.rpm 404K /project/RPMS/sox-14.4.1-6.el7.i686.rpm 28K /project/RPMS/gdm-devel-3.8.4-27.el7.x86_64.rpm 28M /project/RPMS/gimp-help-nl-2.8.1-1.el7.noarch.rpm 28K /project/RPMS/libcap-devel-2.22-8.el7.x86_64.rpm 20K /project/RPMS/texlive-paralist-svn15878.2.3b-32.el7.noarch.rpm 6.6M /project/RPMS/libreoffice-sdk-doc-4.1.4.2-3.el7.x86_64.rpm [output cut]
The above command shows the list of each and every file with absolute path and size in human readable format (MB, GB etc).
To view the top 5 directories in the decreasing order of their size, use the following command:
[root@srv1 ~]# du -Sh /project/ | sort -rh | head -5 4.5G /project/project1 1.1G /project/project2 755M /project/project3 689M /project/project4 501M /project/project5 [output cut]
The disk partitioning scheme discussed in this section is the traditional method of disk partitioning. It does not offer the flexibility of managing the storage space. Also it does not provide any data redundancy in case of drive failures.
If you need redundancy in case of drive failures and flexibility of managing the storage space, take a look at Logical Volume Management (LVM) and RAID Configuration section. LVM and RAID Configuration is often needed if you are running Linux Servers.
1 Comment
Comments are closed.
Thanks alot for this usefull article .