Managing Storage and Disks on Ubuntu Server: File Systems, Partitioning, Formatting, and Optimization

by

Introduction

Managing storage and disks is a fundamental skill for Ubuntu server administrators, particularly in environments like cloud platforms (AWS, Azure, Google Cloud) or virtualization platforms (VMware, Hyper-V, Proxmox). Scenarios such as OS disk expansion or data disk initialization are increasingly common, and while automation mechanisms often handle these tasks, understanding the underlying process is essential for troubleshooting and manual intervention.

This guide covers file system types, partitioning, formatting, and optimizing disk usage, with practical examples for Ubuntu servers in both physical and virtualized environments.

Understanding File System Types

A file system defines how data is stored and accessed on a disk. Choosing the right file system is crucial for performance, reliability, and compatibility. Below are the most common file systems for Ubuntu servers:

Common File Systems for Ubuntu

  1. Ext4 (Fourth Extended File System)
    • Default file system for Ubuntu servers.
    • Features journaling, large file support, and excellent performance.
    • Ideal for general-purpose workloads.
    • Command to format: sudo mkfs.ext4 /dev/sdX1
  2. XFS
    • High-performance journaling file system designed for large files and high I/O workloads.
    • Commonly used in enterprise and cloud environments.
    • Command to format: sudo mkfs.xfs /dev/sdX1
  3. Btrfs (B-Tree File System)
    • Advanced file system offering snapshot support, data compression, and RAID capabilities.
    • Suitable for complex workloads and environments requiring frequent backups.
  4. NTFS
    • Used primarily for compatibility with Windows environments.
    • Useful for dual-boot systems or shared disks between Linux and Windows.
  5. FAT32 and exFAT
    • Lightweight file systems used for removable drives and cross-platform compatibility.

Choosing the Right File System

  • Ext4: General-purpose workloads.
  • XFS: High-performance workloads in enterprise or cloud environments.
  • Btrfs: Advanced features like snapshots and compression.
  • NTFS/FAT32: Cross-platform compatibility.

Partitioning Disks on Ubuntu Server

Partitioning divides a physical disk into logical sections, enabling better data organization and management. This is particularly useful for separating system files, backups, and application data.

Steps to Partition a Disk on Ubuntu Server

  • Identify the Disk:
    Use the lsblk command to list all available disks and their partitions: lsblk
  • Create a Partition:
    Use fdisk or parted to create a new partition. Example using fdisk: sudo fdisk /dev/sdX
    • Press n to create a new partition.
    • Specify the partition size and type.
    • Press w to write changes to the disk.
  • Verify the Partition:
    Run lsblk again to confirm the new partition has been created.

Formatting Data Disks on Ubuntu Server

Formatting prepares a partition for use by creating a file system.

Steps to Format a data Disk on Ubuntu Server

  • Install Necessary Tools:
    Ensure you have the mkfs utility installed (included by default in most distributions).
  • Format the Partition:
    Use the mkfs command to format the partition. Example: sudo mkfs.ext4 /dev/sdX1
  • Create a Mount Point:
    Important Note: In cloud environments, avoid using /mnt as the mount point. The /mnt directory is often reserved for temporary disks in cloud platforms (like Azure or AWS), which may be cleared during reboots or system updates. Instead, use a custom directory such as /data or /storage.Example:
sudo mkdir /data
  • Mount the Partition:
    Mount the formatted partition to the newly created directory: 
sudo mount /dev/sdX1 /data
  • Persist the Mount:
    To ensure the partition is automatically mounted on boot, add it to the /etc/fstab file: 
echo '/dev/sdX1 /data ext4 defaults 0 2' | sudo tee -a /etc/fstab

Persisting the Mount: Use UUID in Cloud Environments

To ensure the partition is automatically mounted on boot, adding it to the /etc/fstab file is essential. However, in cloud environments, it is recommended to use the UUID (Universally Unique Identifier) for mounting disks instead of directly referencing the device name (e.g., /dev/sdX1). Device names can change after a system reboot, especially in virtualized or cloud environments, potentially leading to mount failures. Using UUID ensures consistent and reliable mounting.

Steps to Persist the Mount Using UUID

  • Find the UUID of the Partition:
    Use the blkid command to retrieve the UUID of the partition:
sudo blkid
  • Example output:
/dev/sdX1: UUID="123e4567-e89b-12d3-a456-426614174000" TYPE="ext4"
  • Edit the /etc/fstab File:
    Add an entry to the /etc/fstab file using the UUID instead of the device name.
    Example: echo 'UUID=123e4567-e89b-12d3-a456-426614174000 /data ext4 defaults 0 2' | sudo tee -a /etc/fstab
  • Verify the Configuration:
    Test the configuration by unmounting the partition and remounting it using the mount -a command: 
sudo umount /data  
sudo mount -a
  • Reboot and Confirm:
    Reboot the system to ensure the disk is mounted correctly during startup: sudo reboot

Why Use UUID in Cloud Environments?

  • Consistency: Device names like /dev/sdX may change after a reboot due to dynamic device enumeration in cloud platforms.
  • Reliability: UUIDs are unique to each partition and remain constant, ensuring reliable mounting.
  • Best Practice: Cloud providers like AWS, Azure, and Google Cloud recommend using UUIDs or labels for mounting disks in production environments.

Cloud and Virtualization Scenarios: Disk Expansion and Initialization

Scenario 1: OS Disk Expansion

In cloud platforms like AWS, Azure, or Google Cloud, OS disks can be expanded from the physical layer. Many platforms automatically detect and expand the OS disk. However, manual intervention may be required to resize the file system to utilize the new space.

Steps to Expand OS Disk on Ubuntu Server:

  1. Check Disk Space:
    Verify the expanded disk space using lsblk: lsblk
  2. Resize the File System:
    Use resize2fs for Ext4 file systems: sudo resize2fs /dev/sdX1

Scenario 2: Data Disk Initialization, please see previous section

    Monitoring and Optimizing Disk Usage

    Monitoring Disk Usage

    1. Check Disk Space Usage:
      Use the df command to display disk space usage: df -h
    2. Analyze Directory Usage:
      Use the du command to check the size of specific directories: du -sh /var/log
    3. Advanced Monitoring:
      Install ncdu for interactive disk usage analysis: 
    sudo apt install ncdu 
ncdu /

    Optimizing Disk Usage

    • Remove Temporary Files:
    sudo apt-get autoclean  
sudo apt-get clean  
sudo apt-get autoremove
    • Find Large Files Consuming Disk Space
      Locate large files using the find command:
    sudo find / -type f -size +100M

    References

    In the next article, we will dive into Setting Up a Database Server on ubuntu servers. You’ll learn how to:

    • Install and configure MySQL or MariaDB for your applications.
    • Secure your database server to protect sensitive data.
    • Optimize database performance for production environments.

    Whether you’re deploying databases in the cloud or on-premises, this guide will provide actionable insights to help you effectively manage your database infrastructure.

    Don’t miss it—stay tuned!

    Leave a Comment