After upgrade to v. 1 - Disks do not match df

c0urier · 15. September 2014

Hi there

So I decided to do an upgrade to v. 1 - All went rather flawlessly except for a raid5 that died and most disks are misplaced - Or atleast looks like it.

Code

root@storage00:~# blkid
/dev/md0: UUID="592adfb1-ae68-451d-a700-f4258bde0415" TYPE="ext4"
/dev/md1: UUID="9c6d6c30-aacf-4c2b-8ec1-7db3dde5b4eb" TYPE="swap"
/dev/sdd: UUID="76f03b1e-c6f6-2c7e-ee09-750bb500b2d2" LABEL="storage00:CloudShare" TYPE="linux_raid_member" UUID_SUB="10ac9062-dfa9-290b-2932-b454fd49316b"
/dev/sdc: UUID="76f03b1e-c6f6-2c7e-ee09-750bb500b2d2" LABEL="storage00:CloudShare" TYPE="linux_raid_member" UUID_SUB="1ec2b85e-c9d9-8d86-2432-f4a003f942b3"
/dev/sdg: UUID="327ce2af-1c7b-0965-9c1e-f8a7c852eb72" LABEL="storage00:bCache00" TYPE="linux_raid_member" UUID_SUB="6c2b09eb-871b-14e1-84a1-841d74b6fcc7"
/dev/sdh: UUID="327ce2af-1c7b-0965-9c1e-f8a7c852eb72" LABEL="storage00:bCache00" TYPE="linux_raid_member" UUID_SUB="435bcd6b-2228-c3f2-dc76-213940befd06"
/dev/bcache0: UUID="ec15c4f4-64d9-4fb0-a16a-4b3c5e886a27" TYPE="ext4"
/dev/sdi: UUID="c47d2ea7-1762-a811-e3c3-2a01c1c501c0" LABEL="storage00:vmShare00" TYPE="linux_raid_member" UUID_SUB="2f20bdfa-12fa-cc53-b5bb-8742568ccbc6"
/dev/sdj: UUID="c47d2ea7-1762-a811-e3c3-2a01c1c501c0" LABEL="storage00:vmShare00" TYPE="linux_raid_member" UUID_SUB="569b060e-637e-604f-a5f2-4e4665265224"
/dev/sda: UUID="76f03b1e-c6f6-2c7e-ee09-750bb500b2d2" UUID_SUB="e271d618-1567-f94e-9d81-8f6fb2f163ad" LABEL="storage00:CloudShare" TYPE="linux_raid_member"
/dev/sde1: UUID="3f7580b6-09ac-b91a-531b-5b40dd2348f2" UUID_SUB="9344d3f0-d1f3-aaff-1ca9-f8b8d664d77e" LABEL="storage00:0" TYPE="linux_raid_member"
/dev/sde5: UUID="783f3fed-4432-e1c7-3421-e9155fff7b37" UUID_SUB="96341b04-9788-6862-11e8-ea14865296b2" LABEL="storage00:1" TYPE="linux_raid_member"
/dev/sdf1: UUID="3f7580b6-09ac-b91a-531b-5b40dd2348f2" UUID_SUB="343d2e2d-f60a-9dc3-eb7b-660ad86fccc4" LABEL="storage00:0" TYPE="linux_raid_member"
/dev/sdf5: UUID="783f3fed-4432-e1c7-3421-e9155fff7b37" UUID_SUB="e3fd0e18-c64d-82fd-b767-e63c63bc6fb4" LABEL="storage00:1" TYPE="linux_raid_member"
/dev/sdb: UUID="76f03b1e-c6f6-2c7e-ee09-750bb500b2d2" UUID_SUB="934dc79b-a736-a6bd-3bac-2d313607e554" LABEL="storage00:CloudShare" TYPE="linux_raid_member"
/dev/sdl: UUID="c47d2ea7-1762-a811-e3c3-2a01c1c501c0" UUID_SUB="f3eea358-d79d-0fa5-ace8-a3dbac955fac" LABEL="storage00:vmShare00" TYPE="linux_raid_member"
/dev/sdk: UUID="c47d2ea7-1762-a811-e3c3-2a01c1c501c0" UUID_SUB="83ab4847-bcae-1a7a-4ba3-e7d07d53ff4a" LABEL="storage00:vmShare00" TYPE="linux_raid_member"
/dev/md4: LABEL="CloudShare" UUID="7fe59b53-c96a-4513-bdbd-05560e24797b" TYPE="xfs"
/dev/sdq1: LABEL="c0urierMediaTMP" UUID="52294bc4-dd5c-450d-8057-0e1dd8a4a783" TYPE="ext4"
/dev/sdr1: LABEL="Media00" UUID="76598b8a-f8dd-4ca7-9193-00c834e020a2" TYPE="xfs"
/dev/sds1: UUID="12e77943-18bc-42d5-8aca-a8c031131651" TYPE="xfs"
/dev/sdt1: UUID="c9960dbc-36ef-4c9c-b161-502173503dc4" TYPE="xfs"

Alles anzeigen

Code

root@storage00:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          141G  8,4G  126G   7% /
udev             10M     0   10M   0% /dev
tmpfs           764M  620K  764M   1% /run
/dev/md0        141G  8,4G  126G   7% /
tmpfs           5,0M     0  5,0M   0% /run/lock
tmpfs           2,8G     0  2,8G   0% /run/shm
tmpfs           3,8G     0  3,8G   0% /tmp
/dev/bcache0    1,8T  290G  1,5T  17% /media/ec15c4f4-64d9-4fb0-a16a-4b3c5e886a27
/dev/sdm1       1,8T  381G  1,5T  21% /media/52294bc4-dd5c-450d-8057-0e1dd8a4a783
/dev/sdo1       2,8T  1,2T  1,6T  42% /media/12e77943-18bc-42d5-8aca-a8c031131651
/dev/sdp1       2,8T  2,8T  2,1G 100% /media/c9960dbc-36ef-4c9c-b161-502173503dc4
/dev/sdn1       2,8T  302G  2,5T  11% /media/76598b8a-f8dd-4ca7-9193-00c834e020a2
/dev/bcache0    1,8T  290G  1,5T  17% /export/vmShare00
/dev/bcache0    1,8T  290G  1,5T  17% /export/REAR-backup

Alles anzeigen

And a printscreen from within OMV disks.
http://prntscr.com/4n19zg

Another print from the RAID:
http://prntscr.com/4n1ao4

Any suggestions on how to get it aligned again. The Raid5 was recovered, but I wan't to add sdd, but that is not possible - It's not selectable in the recover menu.

ryecoaaron · 15. September 2014

Any time you have a highly customized setup (using bcache, etc), the upgrade is going to be tough because it is changing distro version as well (squeeze to wheezy).

I recommend booting into systemrescuecd (so array is not in use) and try:

mdadm --stop /dev/md4
mdadm --assemble /dev/md4/dev/sd[abcd] --verbose --force

c0urier · 15. September 2014

Understandable, but I don't understand the mismatch of the drives. Is it possible to get them aligned again?

The md4 dropped two disks, do I did the assemble worked fine, just it does not want to use sdd. Not in the gui atleast, even though it's seen just fine.

Another issue is when trying to enter Smart -> Devices I receive this: http://prntscr.com/4n1hqa

Sometimes it says sds sometimes sdr might be others, dont want to sit an reload the page.

ryecoaaron · 15. September 2014

Not sure what you mean by "aligned"?? Do you mean that they are changing letters? If so, that isn't a big deal. OMV uses the uuid instead of the letter.

If mdadm doesn't like the drive, I would try wiping it and re-adding it to the array. If it keeps happening, check for smart errors.

The smart error is probably because you must have setup a drive in smart that is no longer present. If you can't add the drive back, you will have to manually remove the smart job.

c0urier · 15. September 2014

Okay thanks - The drives were present, but I deleted all scheduled jobs and will readd them.

And yes by aligned I mean, it would be nice if omv showed what df does.

c0urier · 15. September 2014

Okay, deleted all jobs and still received:

Code

Error #6002:
exception 'OMVException' with message 'Device '/dev/sdr' not found' in /usr/share/openmediavault/engined/rpc/smart.inc:87
Stack trace:
#0 [internal function]: OMVRpcServiceSmart->enumerateDevices(NULL, Array)
#1 /usr/share/php/openmediavault/rpcservice.inc(125): call_user_func_array(Array, Array)
#2 /usr/share/openmediavault/engined/rpc/smart.inc(186): OMVRpcServiceAbstract->callMethod('enumerateDevice...', NULL, Array)
#3 [internal function]: OMVRpcServiceSmart->getList(Array, Array)
#4 /usr/share/php/openmediavault/rpcservice.inc(125): call_user_func_array(Array, Array)
#5 /usr/share/php/openmediavault/rpc.inc(79): OMVRpcServiceAbstract->callMethod('getList', Array, Array)
#6 /usr/sbin/omv-engined(501): OMVRpc::exec('Smart', 'getList', Array, Array, 1)
#7 {main}

Alles anzeigen

I did of course apply settings

The device is present in Storage -> Physical disks.

c0urier · 15. September 2014

Fun thing - When ever I reboot the box - The raid5 dies and I have to stop it and assemble the disks.

ryecoaaron · 15. September 2014

That tells me that a drive is failing. Not sure about the smart error. Might have to report that on bugtracker.

c0urier · 15. September 2014

Two drives at once? Not impossible I agree, but I can reassemble the raid5 and it works like a charm until I reboot. Currently it's doing a recover, since I can only attach 3 drives and then recover it.
The mismatch from Storage -> Physical drives are still on going, Storage Raidmanagement shows the correct drives they are aligned with df / blkid. File systems is OK as well.

Screen shots:
1: Physical drives: http://prntscr.com/4n22q7
2: Raid Management: http://prntscr.com/4n22vs
3: File systems: http://prntscr.com/4n230g

Blkid:

Code

root@storage00:~# blkid
/dev/md0: UUID="592adfb1-ae68-451d-a700-f4258bde0415" TYPE="ext4"
/dev/md1: UUID="9c6d6c30-aacf-4c2b-8ec1-7db3dde5b4eb" TYPE="swap"
/dev/sdd: UUID="327ce2af-1c7b-0965-9c1e-f8a7c852eb72" LABEL="storage00:bCache00" TYPE="linux_raid_member" UUID_SUB="435bcd6b-2228-c3f2-dc76-213940befd06"
/dev/sdc: UUID="327ce2af-1c7b-0965-9c1e-f8a7c852eb72" LABEL="storage00:bCache00" TYPE="linux_raid_member" UUID_SUB="6c2b09eb-871b-14e1-84a1-841d74b6fcc7"
/dev/sdg: UUID="76f03b1e-c6f6-2c7e-ee09-750bb500b2d2" LABEL="storage00:CloudShare" TYPE="linux_raid_member" UUID_SUB="1ec2b85e-c9d9-8d86-2432-f4a003f942b3"
/dev/sdh: UUID="76f03b1e-c6f6-2c7e-ee09-750bb500b2d2" LABEL="storage00:CloudShare" TYPE="linux_raid_member" UUID_SUB="67e4d0ba-d616-7a4c-1918-b4764070613f"
/dev/bcache0: UUID="ec15c4f4-64d9-4fb0-a16a-4b3c5e886a27" TYPE="ext4"
/dev/sdi: UUID="c47d2ea7-1762-a811-e3c3-2a01c1c501c0" LABEL="storage00:vmShare00" TYPE="linux_raid_member" UUID_SUB="2f20bdfa-12fa-cc53-b5bb-8742568ccbc6"
/dev/sdj: UUID="c47d2ea7-1762-a811-e3c3-2a01c1c501c0" LABEL="storage00:vmShare00" TYPE="linux_raid_member" UUID_SUB="569b060e-637e-604f-a5f2-4e4665265224"
/dev/sdl: UUID="c47d2ea7-1762-a811-e3c3-2a01c1c501c0" UUID_SUB="f3eea358-d79d-0fa5-ace8-a3dbac955fac" LABEL="storage00:vmShare00" TYPE="linux_raid_member"
/dev/sdk: UUID="c47d2ea7-1762-a811-e3c3-2a01c1c501c0" UUID_SUB="83ab4847-bcae-1a7a-4ba3-e7d07d53ff4a" LABEL="storage00:vmShare00" TYPE="linux_raid_member"
/dev/sda1: UUID="3f7580b6-09ac-b91a-531b-5b40dd2348f2" UUID_SUB="9344d3f0-d1f3-aaff-1ca9-f8b8d664d77e" LABEL="storage00:0" TYPE="linux_raid_member"
/dev/sda5: UUID="783f3fed-4432-e1c7-3421-e9155fff7b37" UUID_SUB="96341b04-9788-6862-11e8-ea14865296b2" LABEL="storage00:1" TYPE="linux_raid_member"
/dev/sdb1: UUID="3f7580b6-09ac-b91a-531b-5b40dd2348f2" UUID_SUB="343d2e2d-f60a-9dc3-eb7b-660ad86fccc4" LABEL="storage00:0" TYPE="linux_raid_member"
/dev/sdb5: UUID="783f3fed-4432-e1c7-3421-e9155fff7b37" UUID_SUB="e3fd0e18-c64d-82fd-b767-e63c63bc6fb4" LABEL="storage00:1" TYPE="linux_raid_member"
/dev/sde: UUID="76f03b1e-c6f6-2c7e-ee09-750bb500b2d2" UUID_SUB="e271d618-1567-f94e-9d81-8f6fb2f163ad" LABEL="storage00:CloudShare" TYPE="linux_raid_member"
/dev/sdf: UUID="76f03b1e-c6f6-2c7e-ee09-750bb500b2d2" UUID_SUB="934dc79b-a736-a6bd-3bac-2d313607e554" LABEL="storage00:CloudShare" TYPE="linux_raid_member"
/dev/sdq1: LABEL="c0urierMediaTMP" UUID="52294bc4-dd5c-450d-8057-0e1dd8a4a783" TYPE="ext4"
/dev/sdr1: LABEL="Media00" UUID="76598b8a-f8dd-4ca7-9193-00c834e020a2" TYPE="xfs"
/dev/sds1: UUID="12e77943-18bc-42d5-8aca-a8c031131651" TYPE="xfs"
/dev/sdt1: UUID="c9960dbc-36ef-4c9c-b161-502173503dc4" TYPE="xfs"
/dev/md4: LABEL="CloudShare" UUID="7fe59b53-c96a-4513-bdbd-05560e24797b" TYPE="xfs"

Alles anzeigen

c0urier · 7. Oktober 2014

After one of the later updates for OMV, my issue has vanished - I am able to boot without having to reassemble the raid5 array.

Furthermore I've upgraded to the latest stable kernel 3.17 and can say that bcache works like a charm with it.

Jetzt mitmachen!