Understanding Linux File Systems: Ext4 And Beyond

Understanding Linux File Systems: Ext4 And Beyond

You can find the features of the ext4 file system used in Linux platforms, the differences with the ext3 file system, the future of previously used file systems and file systems in this article.

Most modern Linux distributions come with the ext4 file system by default. The file systems used by default in the past were ext3, ext2, and ext if they go back much more.

If you have just started to learn about the Linux platform or file systems, you may wonder what the ext4 file system has brought to the ext3 file system. You may also be wondering if ext4 is still being developed or alternative file systems like btrfs, XFS, ZFS.

It is not possible to gather everything about file systems in a single article. However, in this article, we will give you as much information as possible about the development of Linux file systems, where we are now, and what might happen in the future.

Ext File System History

MINIX File System

Before the Ext file system, there was the MINIX file system. MINIX was also a small UNIX-like operating system developed for IBM PC / AT microcomputers. The source code of this file system, developed by Andrew Tannenbaum for educational purposes, was shared in 1987 in writing.

Although it was then possible to access MINIX source codes, the operating system was not free. Publishers of Tannebaum's book were asking for a $ 69 license fee to run the MINIX operating system, which was included in the price of the book.

The conversion to the MINIX operating system was fast and exceeded teaching the coding of operating systems, Tannenbaum's main goal. It was possible to find this operating system in many universities around the world until the 1990s and even afterward. Linus Torvalds, the genius behind Linux, also developed the original Linux kernel, which was first announced in 1991 and released under the free software license in 1992, using the MINIX operating system.

If we talk a little more about the MINIX file system, this file system was used in the first Linux versions. This file system only supports file names of up to 14 characters and could only address 64 MB. In 1991, the standard was 40-140 MB of hard disk capacity. It was clear that Linux needed a better file system.

Exter

While Linus Torvalds was working on the new Linux kernel, Rémy Card worked on the first ext file system. Only 1 year after the announcement of Linux, the ext file system in 1992 solved the biggest problems of the MINIX file system.

The ext file system in 1992 used the new virtual file system layer in the Linux kernel. The Ext file system was able to address up to 2 GB and supported 255 character file names.

The reign of the Ext file system did not last very sadly. This file system with primitive time-stamping support only supported one-time stamp per file. In today's file systems, there are 3 timestamps (file creation, access, editing) per file. Rémy Card noticed the deficiencies in the ext file system and revealed the ext2 file system 1 year later.

ext2

The ext2 file system, unlike the ext file system, is designed on a commercial level like the Berkeley Fast File System of BSD. Ext2 has gigabytes as the maximum file size and terabytes as the file system. Due to these features, it was quickly owned. With the Linux kernel and MINIX, it has become available for use on macOS and Windows, even with third-party modules.

The Ext2 file system also had problems to resolve. Like other file systems in the 1990s, ext2 left the system unstable if the system crashed or power was interrupted while writing data to the disk. Dozens of files that were not related to the recorded files were damaged and even the use of the entire file system was compromised. Also, serious performance losses were experienced due to disk fragmentation (a single file is located in more than one area on the disk).

ext3

In 1998, Stephen Tweedie announced that he was working to significantly improve the ext2 file system. This was a new ext3 file system and the Linux 2.4.15 kernel, in 2001, he began to use with.

Ext3 and other file systems that emerged in the late 1990s (such as Microsoft's NTFS file system) started using the journaling system to prevent serious data loss that occurred during the writing we mentioned for ext2. The log is a special partition on the disk where data was stored during write transitions. When the process of burning to disk is finished, the location of the data in the daily part is allocated to the file system itself.

If the system crashes before this allocation, when the computer restarts, the system will treat it as an incomplete transition and undo the changes as if it never happened. In this way, even if the data studied is lost, the file system itself remains intact and other data is also secure.

Ext3 file system was made available in 3 levels while coding to Linux kernel; journal ordered, and writeback.

Journal is defined as the lowest disk mode. Both data and metadata are written to the journal section before being processed into the file system. Although this guaranteed the integrity of the written file and the entire file system, it significantly reduced its performance.

Ordered is the default mode used by many Linux distributions. In this mode, metadata is written to the journal section, but the data is directly linked to the file system. As the name implies, these operations take place in a fixed order. The metadata is written to the journal first, then the data is written to the file system, and the metadata associated with this data is linked to the latest file system. In this way, metadata about the incomplete writing process at the time of the crash remains in the journal section. It clears this incomplete writing when the file system returns to the journal. Thus, even if the data or data that are actively written at the time of the crash is corrupted, the file system itself and files that are not actively written are guaranteed.

Writeback is the third and most insecure journal mode. In this mode, no data is logged, only metadata is logged. Unlike Ordered mode, metadata or data are written in the order that will get maximum performance and there is no fixed order. In this way, the data is less secure even though there is a significant increase in performance. While the security of the file system is still protected, the data written before or during the crash becomes vulnerable to lose and corruption.

Like the ext2 file system, the ext3 file system uses 32-bit internal addressing. Thus, with a block size of 4 KiB, it supports a maximum of 2-byte file size and a maximum of 16-byte file size.

Ext4

Later, Theodore Ts'o, the chief developer of the ext3 project, announced the ext4 file system in 2006. It was launched 2 years later with the 2.6.28 version of the Linux kernel. To described ext4 as a temporary innovation. Although it adds some innovations according to Ext3, it is still using old technology. Ts’o expects this file system to be replaced by a new generation file system.

Differences Between Ext4 and Ext3 File Systems

Back Compatibility

The ext4 file system was designed to be backward compatible with the ext3 file system as much as possible. Thus, it was possible to upgrade directly to the ext4 file system in the Ext3 file system and allowed the ext4 driver to automatically connect the Ext3 file system in ext3 mode. In this way, it was not necessary to have two separate drivers.

Larger File System Support

As mentioned, the Ext3 file system uses 32-bit addressing and supports a maximum of 2-byte file size, 16-byte maximum file system size. However, this is possible with 4 KiB block sizes, and some ext3 file systems use smaller block sizes, so the maximum capacity is further limited.

The Ext4 file system uses 48-bit internal addressing. In this way, it theoretically supports a maximum file size of 16-byte byte and a 1,000,000-byte (1 EiB) file system. Although the initial adaptations of the ext4 file system were initially limited to the 16 TiB maximum file system, in 2011, e2fsprogs directly supported the creation of ext4 file systems larger than this size. For example, Red Hat Enterprise Linux officially provides support for file systems up to 50 TiB and does not recommend creating more than 100 TiB.

Space Allocation Enhancements

The ext4 file system has brought improvements to allocate storage blocks before they are written to disk. In this way, both reading and writing performance has increased significantly.

Extents: With this feature, adjacent physical blocks up to 128 MiB are defined. When these blocks are used, the number of file-related nodes is decreasing. This reduces fragmentation and improves performance when writing large files.

Multiple Block Allocation: An extender was used for each new block allocation in the Ext3 file system. Therefore, when more than one write operation took place at the same time, fragmentation occurred more. In Ext4, it is better adapted to co-authoring using the delayed allocation mechanism and makes a better decision about the allocation of the blocks before the write operation takes place.

Continuous Pre-Allocation: Most file systems have to write 0s when blocks are created to pre-allocate disk space for a file. The Ext4 file system guarantees the presence of the required space without the need to write before using the allocate () command. In this way, performance increases significantly both in writing and in reading the written data later.

Delayed Allocation: As we mentioned in the multi-block allocation section, it is expected to accumulate the data in the cache and provide more intelligent solutions related to the allocation of the blocks. However, the sync () command increases the risk of data loss, especially when it is not used.

Unlimited Subdirectories

The Ext3 file system was limited to a total of 32,000 subdirectories. The Ex4 file system makes it unlimited. Using the ext4 HTree system together with the 2.6.23 kernel reduces the performance loss that occurs during the processing of many sub-arrays.

Daily Supply

It was not possible to provide logs in the Ext3 file system. This could cause problems in the caches of devices or controllers, especially because the kernel did not have direct control. With the logs being supplied, the file system can recognize invalid or out-of-control entries after any crash. In this way, additional damage to the file system can be prevented.

Fast File System Control

In the Ext3 file system, when the fsck command was run, the entire file system (including deleted or empty files) was checked. In the Ext4 file system, it is possible to bypass unallocated blocks and sections.

Improved Timestamps

It used timestamps at the level of seconds along with Ext3. While this was sufficient for most uses, it was insufficient for applications that performed critical tasks. Ext4 file system offers time stamp support at the nanoseconds level. Also, the Ext3 file system does not store enough data for dates after January 18, 2038, but the Ext4 file system offers support until 2446.

Defragment Option With Disk Attached

Ext2 and ext3 file systems did not allow defragmentation while the disk was mounted. Although there were attempts to bring solutions for this in these systems, they were not very successful. The Ext4 file system brought kernel-supported, file-system-aware defragmentation support that allows defragmentation with the new e4defrag module on disk.

Ongoing ext4 Development Works

Although Ext4 has been put forward as a temporary solution by its developer, it will continue to maintain a significant amount of time since the next-generation systems that will replace this file system are not ready yet. That's why some innovations are working on possible new versions of ext4.

Metadata Provision

The Ext4 file system has a section called a superblock called the metadata of the file system. This superblock contains the number of blocks in the file system, the number of empty blocks, the time of linking, the time of writing, and much more. Since this data is important, its first copy is called primary superblock and is read when the file system is mounted. There is also a backup of this superblock in each block group.

However, this primary superblock is broken and users can't notice and correct it when replacement is required. Although new methods such as provisioning per block have been introduced in new generation file systems such as BTRFS and ZFS, it is being worked on providing metadata provisioning with the logic of “better than nothing” in the ext4 file system.

Enhanced Quota Support

Starting with the Ext2 file system, there was a quota feature that could restrict users' use of disk resources. However, we are working on making this quota support performance-oriented and more accurate and making it kernel supported.

Large Allocation Blocks

The Ext4 file system is currently limited to a 4K burst. Especially with the emergence of 8K hardware data blocks with some new SSDs, this limitation became much more evident. Larger blocks of data can reduce fragmentation and improve performance. Of course, this increase can be done by waiving the increase of free space in the blocks.

Practical Limits of Ext4 File System

Ext4 is a robust and stable operating system. That's why it has been used by many users even for the root directory. However, the ext4 file system has its limits and there are some things we should not expect from this file system.

Although Ext4 can theoretically address data of 1,000,000 TiB size, problems arise over 50-100 TiB and the availability of the file system decreases.

Ext4 cannot guarantee the integrity of your data. Although the logging system has made great progress since the ext3 file system, there is nothing this file system can do about the main reasons that may cause data loss. (Such as faulty equipment, the effect of rays, damage over time.)

If these two cases are considered, it would be more accurate to define ext4 as a pure file system. Ext4 is not a storage controller. Even if we have two identical disks, it is not possible for the ext4 file system to retrieve corrupt data from the other disk and use it.

Although it is theoretically possible to separate the file system and the storage control system into different layers and protect repair and damage detection mechanisms, today's storage systems are not designed in this way and are an obstacle for new designs.

Alternative File Systems

Before starting this part, a warning must be made. Alternative file systems may not be supported directly in the basic kernel of the Linux distribution.

Even if a file system is safe, any situation that may occur during the kernel upgrade when used in the root file system can cause serious troubles. If you do not like the idea of ​​dealing with any problems that may occur and you do not have enough experience, you should not use alternative file systems in the root directory.

You may have good reasons to use alternative file systems that your Linux distribution does not support. However, it is highly recommended to connect these units after the system is operational and available. For example, you may have the ext4 root file system and want to collect your other data in the zfs or btrfs pool.

XFS

XFS leads the way in non-ext file systems used under the Linux platform. This file system, which has 64-bit and logging features, has been included in the Linux kernel since 2001. It offers high performance in large file systems with a large number of writes.

As with RHEL 7 in Red Hat Enterprise Linux, the default file system has started to become XFS. It has disadvantages for homes and small businesses. For example, it is very difficult to resize the existing XFS file system. In many cases, it might even make more sense to create a new one and copy your data there.

Although the XFS file system is performance and stable, it doesn't make enough difference to be used by end-users instead of ext4. If you are dealing with data over 50 TiB, the situation may be different.

XFS cannot be described as the next-generation file system. As in the case of ext4, it is a temporary improvement that has been revealed until something better is revealed.

ZFS

ZFS was developed by Sun Microsystems and got this name because of zettabyte (1 trillion gigabytes). In theory, it is possible for this file system to address such a large amount of data.

ZFS, which can be described as a new generation file system, also offers unit management feature. So it is possible to address more than one storage device under the same file system. Apart from that, it features block-level cryptographic provisioning, automatic damage correction, high-speed cloud copying, internal compression and many more features that allow very high consistency damage detection.

The biggest problem of ZFS for Linux is the license issue. There are different opinions and thoughts regarding the use of ZFS within the Linux kernel, which uses a semi-permitted CDDL license that conflicts with Linux's GPL license on some issues. However, since 2016, Canonical (the company behind Ubuntu) has included the ZFS code in its kernel and has not encountered any legal trouble so far.

Btrfs

The Btrfs file system was announced by Chris Mason in 2007. Btrfs generally has the same goals as ZFS and offers similar features.

As of 2018, btrfs has become practical enough to be used as standard on single-disk file systems, but it is far from being reliable as a volume manager. Compared to Ext4, XFS, ZFS, there are performance problems. At the same time, the new generation of features can sometimes cause errors and this results in a range from seriously reduced performance to data loss.

The future status of Btrfs is uncertain. Although SUSE Enterprise Linux began to be used as the default file system in 2015, Red Hat announced in 2017 that it will not support btrfs starting from 7.4. Anyway, the support it offered before this release was for single disk uses.