ext4: two disks, same data - different sizes

Hi! I was hoping someone could explain this little mystery to me. Recently I got a new HDD for my NAS and began migrating data from one to another.

Old drive: WD Red 3TB (/dev/sdb1; mp: /mnt/WDRED30)
New drive: Seagate IronWolf 4TB (/dev/sdd1; mp: /mnt/STIW40)
Filesystem on both: ext4 (one partition on each)

I copied all the data to the new drive using rsync. Afterwards I checked file count and total size. Now, while file count matches:
Code:[root@xxxxx ~]$ find /mnt/WDRED30 -type f | wc -l
152182
[root@xxxxx ~]$ find /mnt/STIW40 -type f | wc -l
152182The total size of all files actually differs:
Code:[root@xxxxx ~]$ du -sk /mnt/WDRED30
2660608836 /mnt/WDRED30
[root@xxxxx ~]$ du -sk /mnt/STIW40
2692515672 /mnt/STIW40Code:2692515672-2660608836 = 31906836 kB = ~31 GBResult: The same data copied to the new drive take approx. 31 GB more space.

So I began investigating a bit. Both have the same sector size:
Code:[root@xxxxx ~]$ fdisk -l /dev/sdb1
Disk /dev/sdb1: 2.7 TiB, 3000591916544 bytes, 5860531087 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
[root@xxxxx ~]$ fdisk -l /dev/sdd1
Disk /dev/sdd1: 3.7 TiB, 4000785104896 bytes, 7814033408 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytesThe only differences I see are in Filesystem features and Reserved block count:
Code:[root@xxxxx ~]$ tune2fs -l /dev/sdb1
tune2fs 1.44.5 (15-Dec-2018)
Filesystem volume name: DATA2
Last mounted on: /mnt/WDRED30
Filesystem UUID: c6e57db2-888d-4972-a3c6-fd48bb0e8377
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 183148544
Block count: 732566385
Reserved block count: 0
Free blocks: 55859660
Free inodes: 182990738
First block: 0
Block size: 4096
Fragment size: 4096
Group descriptor size: 64
Reserved GDT blocks: 1024
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Flex block group size: 16
Filesystem created: Mon Feb 8 21:35:27 2016
Last mount time: Mon Jul 5 10:30:44 2021
Last write time: Mon Jul 5 10:30:44 2021
Mount count: 537
Maximum mount count: -1
Last checked: Mon Feb 8 21:35:27 2016
Check interval: 0 (<none>)
Lifetime writes: 2699 GB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: 5d94df22-0087-4fc8-98e8-b1539684dabb
Journal backup: inode blocksCode:[root@xxxxx ~]$ tune2fs -l /dev/sdd1
tune2fs 1.44.5 (15-Dec-2018)
Filesystem volume name: DATA1
Last mounted on: /mnt/STIW40
Filesystem UUID: 61f3f1a0-f9ec-46f2-beb6-d2f58f3e18c6
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 244195328
Block count: 976754176
Reserved block count: 48837708
Free blocks: 288018736
Free inodes: 244037521
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 791
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Flex block group size: 16
Filesystem created: Sat Jul 3 14:11:04 2021
Last mount time: Mon Jul 5 10:30:44 2021
Last write time: Mon Jul 5 10:30:44 2021
Mount count: 3
Maximum mount count: -1
Last checked: Sat Jul 3 14:11:04 2021
Check interval: 0 (<none>)
Lifetime writes: 2628 GB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 32
Desired extra isize: 32
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: 6e14190c-29fb-4072-88dd-5c1f576bc6b6
Journal backup: inode blocks
Checksum type: crc32c
Checksum: 0xf181afbcThe differences in Filesystem features:
Old: /dev/sdb1 - has 64bit and uninit_bg enabled
New: /dev/sdd1 - no 64bit, no uninit_bg but it has metadata_csum enabled

Manual says (https://man7.org/linux/man-pages/man5/ext4.5.html):
Code:metadata_csum
This ext4 feature enables metadata checksumming. This
feature stores checksums for all of the filesystem
metadata (superblock, group descriptor blocks, inode and
block bitmaps, directories, and extent tree blocks). The
checksum algorithm used for the metadata blocks is
different than the one used for group descriptors with the
uninit_bg feature. These two features are incompatible
and metadata_csum will be used preferentially instead of
uninit_bg.

uninit_bg
This ext4 file system feature indicates that the block
group descriptors will be protected using checksums,
making it safe for mke2fs(8) to create a file system
without initializing all of the block groups. The kernel
will keep a high watermark of unused inodes, and
initialize inode tables and blocks lazily. This feature
speeds up the time to check the file system using
e2fsck(8), and it also speeds up the time required for
mke2fs(8) to create the file system.So it actually makes sense that one is always present and not the other, as the two are incompatible. Could metadata_csum be the reason for the extra 31 GB of space used?

As for 64bit:
Code:64bit
Enables the file system to be larger than 2^32 blocks.
This feature is set automatically, as needed, but it can
be useful to specify this feature explicitly if the file
system might need to be resized larger than 2^32 blocks,
even if it was smaller than that threshold when it was
originally created. Note that some older kernels and
older versions of e2fsprogs will not support file systems
with this ext4 feature enabled.Do I understand it correctly that it just allows large size partitions and should not have any other implications? If so, this should not be the root cause I'm looking for. I just don't understand why the older smaller partition was created with it and not the new one but it should not matter.

As for Reserved block count:
Old: /dev/sdb1 - Reserved block count = 0
New: /dev/sdd1 - Reserved block count = 5% (default) = approx. 200 GB

Not sure why the old one has 0, I don't remember adjusting this but hey, it was a long time ago when I created that partition, so not really sure. I kinda understand now what the reserved blocks are about and this also should not be the cause for having 31 GB extra space used, correct? I will for sure lower the value (or set it to 0) as 200 GB seems way too much for my case.

Then there is the little difference in extra isize but that should not be the cause either as the extra size is taken from the inode size itself, right? Plus, the 4B difference does not fit the math either.

The partition on the new drive has been created using gparted in Debian 10.
The partition on the old drive was created 5 years ago in Debian 7, unfortunately I don't remember how (it was most probably either gparted or mke2fs).

I don't really care that much about those 31 gigs but I am really curious why. I never really paid much attention to all those fs features but this is showing me I probably should... :)

Thanks!