AUFS disk full, "copy up" does not happen

kranich · 9. Dezember 2015

So I started using OMV (1.19) a while ago and used Aufs via AUFS/MHDFS Plugin. The pool seemed to work since my smb share showed me the sum of all participating disks as free space.

First Pool
The first disk filled up and I received a "disk full" message writing via smb. So I read about other people's issues with not working balancing (cf. "AUFS" balancing). In the end I concluded I might have done an error in the setup.

So I deleted all .wh files, deleted the pool, created a new one and moved all previous data from the full disk via the new pool to another disk.

Current Pool
Balancing works
I noticed that on the top-level of the pool, the data is distributed. As I create a new folder on the root level, it correctly chooses the disk with the most free space. Now the other disk got full since aufs behaves like expected for the "pmfs" strategy the plugin set to my fstab. It files new content to the disk where the parent folder is stored.

Copy up does not
What I expect to happen with a full disk and a new file in a folder which is on the full disk: The "copy up" as described in the aufs manual. But this does not happen. When I write even as root via cp to the pool folder on the full disk, it stops after the disk is full (and gets really slow for the last MBs).

Is there configuration step for AUFS I missed or does anyone know how to overcome this behavior?

fstab:

Code

# >>> [openmediavault]
UUID=891a76a4-7ebe-4a46-a106-88e3965a658d /media/891a76a4-7ebe-4a46-a106-88e3965a658d ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota
.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
UUID=f891c173-7b31-44ed-bed8-af50f170fb47 /media/f891c173-7b31-44ed-bed8-af50f170fb47 ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota
.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
UUID=37b04043-7532-402e-9e34-bdaf13873373 /media/37b04043-7532-402e-9e34-bdaf13873373 ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota
.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
UUID=a8e423c4-9b4b-4763-9005-43b72cae24d8 /media/a8e423c4-9b4b-4763-9005-43b72cae24d8 ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota
.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
UUID=1b714067-99c1-4061-a43f-8bb2d6a33aee /media/1b714067-99c1-4061-a43f-8bb2d6a33aee ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota
.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
UUID=c19b0bcb-a32f-433f-a354-c6dda6bbec7b /media/c19b0bcb-a32f-433f-a354-c6dda6bbec7b ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota
.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
UUID=5fa381e1-65d9-4d41-8810-95d48a07ac8a /media/5fa381e1-65d9-4d41-8810-95d48a07ac8a ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota
.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
none /media/db00b4dd-d4a6-43ea-9f53-1c06ab5fa206 aufs br:/media/891a76a4-7ebe-4a46-a106-88e3965a658d=rw:/media/f891c173-7b31-44ed-bed8-af50f1
70fb47=rw:/media/37b04043-7532-402e-9e34-bdaf13873373=rw:/media/a8e423c4-9b4b-4763-9005-43b72cae24d8=rw:/media/1b714067-99c1-4061-a43f-8bb2d6
a33aee=rw,sum,create=pmfs 0 0
# <<< [openmediavault]

Alles anzeigen

mount:

Code

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=1019617,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=817104k,mode=755)
/dev/mapper/gnas-root on / type ext4 (rw,relatime,errors=remount-ro,user_xattr,barrier=1,data=ordered)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1634200k)
/dev/sdi2 on /boot type ext2 (rw,relatime,errors=continue)
/dev/sdi1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=utf8,shortname=mixed,errors=remount-ro)
tmpfs on /tmp type tmpfs (rw,relatime)
/dev/mapper/green1 on /media/891a76a4-7ebe-4a46-a106-88e3965a658d type ext4 (rw,noexec,relatime,user_xattr,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
/dev/mapper/green2 on /media/f891c173-7b31-44ed-bed8-af50f170fb47 type ext4 (rw,noexec,relatime,user_xattr,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
/dev/mapper/green3 on /media/37b04043-7532-402e-9e34-bdaf13873373 type ext4 (rw,noexec,relatime,user_xattr,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
/dev/mapper/old200 on /media/a8e423c4-9b4b-4763-9005-43b72cae24d8 type ext4 (rw,noexec,relatime,user_xattr,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
/dev/mapper/old500 on /media/1b714067-99c1-4061-a43f-8bb2d6a33aee type ext4 (rw,noexec,relatime,user_xattr,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
/dev/mapper/sea1 on /media/c19b0bcb-a32f-433f-a354-c6dda6bbec7b type ext4 (rw,noexec,relatime,user_xattr,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
/dev/mapper/red1 on /media/5fa381e1-65d9-4d41-8810-95d48a07ac8a type ext4 (rw,noexec,relatime,user_xattr,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
none on /media/db00b4dd-d4a6-43ea-9f53-1c06ab5fa206 type aufs (rw,relatime,si=b3797eb50ee8bbd6,create=pmfs,sum)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)

Alles anzeigen

df-output

Code

Dateisystem            1K-Blöcke    Benutzt  Verfügbar Verw% Eingehängt auf
rootfs                  21799476    2111048   18581076   11% /
udev                       10240          0      10240    0% /dev
tmpfs                     817104       6996     810108    1% /run
/dev/mapper/gnas-root   21799476    2111048   18581076   11% /
tmpfs                       5120          0       5120    0% /run/lock
tmpfs                    1634200          0    1634200    0% /run/shm
/dev/sdi2                 234153     188670      32991   86% /boot
/dev/sdi1                 497696        128     497568    1% /boot/efi
tmpfs                    4085512          0    4085512    0% /tmp
/dev/mapper/green1    2884281560   81571476 2802710084    3% /media/891a76a4-7ebe-4a46-a106-88e3965a658d
/dev/mapper/green2    2884281560     406180 2883875380    1% /media/f891c173-7b31-44ed-bed8-af50f170fb47
/dev/mapper/green3    2884281560 2884281560          0  100% /media/37b04043-7532-402e-9e34-bdaf13873373
/dev/mapper/old200     240360960     392000  239968960    1% /media/a8e423c4-9b4b-4763-9005-43b72cae24d8
/dev/mapper/old500     480718544     403028  480315516    1% /media/1b714067-99c1-4061-a43f-8bb2d6a33aee
/dev/mapper/sea1      2884281560 2858518064   25763496  100% /media/c19b0bcb-a32f-433f-a354-c6dda6bbec7b
/dev/mapper/red1      2884281560 2858518068   25763492  100% /media/5fa381e1-65d9-4d41-8810-95d48a07ac8a
none                  9373924184 2967054244 6406869940   32% /media/db00b4dd-d4a6-43ea-9f53-1c06ab5fa206

Alles anzeigen

ryecoaaron · 9. Dezember 2015

Some people have issues with aufs. Others don't. Never been able to figure out why (especially if removing the whiteout files doesn't fix it). Wondering if it is an issue with pmfs or the fact that you are using lvm?? You are the first I have seen use lvm with aufs. Unless you are getting the volumes from a SAN, using LVM is pointless with aufs.

I would try switching to mergerfs instead of aufs to see if it helps.

kranich · 10. Dezember 2015

The LVM was generated by the Debian installer automatically for an encrypted sytem environment. The disks for the AUFS pool are LUKS containers. Btw.: Where did you derive from there is any LVM involved at all?

I'll unmount the aufs pool next and give that mhddfs a try. If this does not work I will update to OMV2.X. And if that also does not help I will come back here and cry being lost in Weltschmerz.

ryecoaaron · 10. Dezember 2015

Where did I get the LVM involved? /dev/mapper maybe?

Try mergerfs not mhddfs. If you are still on OMV 1.x, upgrade to OMV 2.x.

kranich · 10. Dezember 2015

Ok - thanks so far - I updated to Stone-Burner and installed mergerfs. Now I have two problems.

The first one is a "ghost"-AUFS in my File System View, which requires me to keep the AUFS plugin installed since I otherwise only get an error message, that there is no backend for aufs when clicking that view. The ghost entry reads like "n/a n/a aufs n/a n/a missing". Some config file got out of sync here I guess. How to clean this up? The mount point is removed from fstab and no aufs is mounted anymore.
The second issue is that the mergerfs drive is not listed in the view of file systems in OMV. Following from that I cannot file any shared folders on the pool mountpoint since omv forces me to select a drive prior to selecting a folder.

mount

Code

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=1019617,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=817104k,mode=755)
/dev/mapper/gnas-root on / type ext4 (rw,relatime,errors=remount-ro,user_xattr,barrier=1,data=ordered)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1634200k)
/dev/sdi2 on /boot type ext2 (rw,relatime,errors=continue)
/dev/sdi1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=utf8,shortname=mixed,errors=remount-ro)
tmpfs on /tmp type tmpfs (rw,relatime)
/dev/mapper/green1 on /media/891a76a4-7ebe-4a46-a106-88e3965a658d type ext4 (rw,noexec,relatime,user_xattr,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
/dev/mapper/green2 on /media/f891c173-7b31-44ed-bed8-af50f170fb47 type ext4 (rw,noexec,relatime,user_xattr,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
/dev/mapper/green3 on /media/37b04043-7532-402e-9e34-bdaf13873373 type ext4 (rw,noexec,relatime,user_xattr,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
/dev/mapper/old200 on /media/a8e423c4-9b4b-4763-9005-43b72cae24d8 type ext4 (rw,noexec,relatime,user_xattr,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
/dev/mapper/old500 on /media/1b714067-99c1-4061-a43f-8bb2d6a33aee type ext4 (rw,noexec,relatime,user_xattr,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
/dev/mapper/sea1 on /media/c19b0bcb-a32f-433f-a354-c6dda6bbec7b type ext4 (rw,noexec,relatime,user_xattr,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
/dev/mapper/red1 on /media/5fa381e1-65d9-4d41-8810-95d48a07ac8a type ext4 (rw,noexec,relatime,user_xattr,barrier=1,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group,_netdev)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
pool on /media/pool type fuse.mergerfs (rw,relatime,user_id=0,group_id=0,allow_other)

Alles anzeigen

ryecoaaron · 11. Dezember 2015

You need to manually install mergerfs package - apt-get install mergerfs Once you do that, make a very small change to the pool and save/apply. That should everything but the n/a aufs stuff. That stuff won't hurt anything but we can get rid of it by manually editing the config.xml file. I will ask for output a bit later (busy right now).

kranich · 12. Dezember 2015

Ok - got it, for those who were as dumb as me. There was very recent development on integrating mergerfs into omv. You can read about it in this thread. Don't just grab a new version from github, but use the one from the omv-extras testing repo. You will also need a plugin to get the pool into the normal omv shared folders ecosystem.

Bottomline to use mergerfs in omv:

Update to OMV 2.X Stone Burner (cf. GUIDES forum section)
Enable testing repo in omv-extras plugin (links to a post where someone shows screenshots)
Manually install mergerfs as described by ryecoaaron above (apt-get install mergerfs)
You will need a very recent version of the openmediavault-unionfilesystems (1.8 is not enough). If you only see 1.8 as me, make sure your omv-extras package is up to date. I had to manually update it using apt-get.
Now finally you can enjoy using mergerfs as pooling option in the unionfilesystems plugin tab.

Thanks for the tip about this alternative. First simple tests seem promising.
PS: Removed the aufs entry (<mntent> to ... </mntent>) manually from /etc/openmediavault/config.xml - worked like a charm.

ryecoaaron · 12. Dezember 2015

No need to manually install mergerfs anymore in unionfilesystems-2.3 (mergerfs is a dependency). I also updated mergerfs to 2.8.

Sinedbret · 26. Dezember 2015

Hi all,

I also have the disk full symptoms.
Is it possible to migrate a pool from AUFS to MERGERFS without having to demantle everything?

If yes, does the MERGERFS will solve the problem without config modification?

Thank you for your answer.

Sined

ryecoaaron · 26. Dezember 2015

With the latest plugin, you can switch to mergesfs. Just reboot after doing it. No idea if it fixes your problem since I don't have any problems with aufs. But at least mergerfs is actively developed.

Sinedbret · 27. Dezember 2015

Thank you ryecoaaron for your reply.
I will check and keep you posted.
Best wishes.

Sined

Sinedbret · 9. Januar 2016

Hi all,

I switched to mergerfs but still get the same problem, Unable to write to the pool. Disk Full! Mergerfs does not switch to another disk.
I will try to reinstall everything from scratch to see if it solves the problem.

I will keep you updated.

Regards.

Sined

trapexit · 19. Januar 2016

Zitat von Sinedbret

I also have the disk full symptoms.
Is it possible to migrate a pool from AUFS to MERGERFS without having to demantle everything?

If yes, does the MERGERFS will solve the problem without config modification?

Thank you for your answer.

If the drive selected for the file to be written to is full that's the error that will be returned. If you use the moveonenospc feature in mergerfs it will try to move the file being written and then continue but only if there is sufficient space elsewhere.

I'm not familiar with how to set the value in OMV.

Jetzt mitmachen!