TECHZEN Zenoss User Community ARCHIVE  
FORUMS / Configuration & Administration / DISCUSSION

serviced fails to start due to devicemapper error

Subject: serviced fails to start due to devicemapper error
Author: Jad Baz
Posted: 2018-10-30 09:42

I'm an error with Zenoss 6.2.1 (serviced 1.5.1) on a fresh Centos 7.5 with docker 17.09.0-ce.
The error is the same as this thread but appeared here on startup.

The weird thing is, it was working quite well until I logged in to check something and found that it was slowing down tremendously (I'm working on a 12 GB machine).
I rebooted the machine and started getting the error.
Oct 30 15:20:05 LDLBN019WMZNS01 serviced: E1030 13:20:05.438238 05007 devicemapper.go:477] Error mounting device "5DCUDNVNWnT9m0zluVxXQ1" on "/opt/serviced/var/volumes/9u3l0amg9fhm9ps59qtu71xqt" for volume "9u3l0amg9fhm9ps59qtu71xqt": devmapper: Error mounting '/dev/mapper/docker-253:0-34009442-5DCUDNVNWnT9m0zluVxXQ1' on '/opt/serviced/var/volumes/9u3l0amg9fhm9ps59qtu71xqt': input/output error
Oct 30 15:20:05 LDLBN019WMZNS01 serviced: time="2018-10-30T13:20:05Z" level=fatal msg="Could not get volume for tenant" error="devmapper: Error mounting '/dev/mapper/docker-253:0-34009442-5DCUDNVNWnT9m0zluVxXQ1' on '/opt/serviced/var/volumes/9u3l0amg9fhm9ps59qtu71xqt': input/output error" location="daemon.go:515" logger=cli.api tenantid=9u3l0amg9fhm9ps59qtu71xqt​

here is my lsblk:
[root@LDLBN019WMZNS01 ~]# lsblk
NAME                                               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                                                  8:0    0   40G  0 disk
├─sda1                                               8:1    0    1G  0 part /boot
└─sda2                                               8:2    0   39G  0 part
  ├─centos-root                                    253:0    0   35G  0 lvm  /
  └─centos-swap                                    253:1    0    4G  0 lvm  [SWAP]
sdb                                                  8:16   0   40G  0 disk
├─zenoss-docker--pool_tmeta                        253:2    0  816M  0 lvm
│ └─zenoss-docker--pool                            253:4    0 19.6G  0 lvm
├─zenoss-docker--pool_tdata                        253:3    0 19.6G  0 lvm
│ └─zenoss-docker--pool                            253:4    0 19.6G  0 lvm
├─zenoss-serviced--pool_tmeta                      253:5    0   20M  0 lvm
│ └─zenoss-serviced--pool                          253:7    0 16.9G  0 lvm
│   └─docker-253:0-34009442-5DCUDNVNWnT9m0zluVxXQ1 253:15   0  100G  0 dm
└─zenoss-serviced--pool_tdata                      253:6    0 16.9G  0 lvm
  └─zenoss-serviced--pool                          253:7    0 16.9G  0 lvm
    └─docker-253:0-34009442-5DCUDNVNWnT9m0zluVxXQ1 253:15   0  100G  0 dm
sr0                                                 11:0    1 1024M  0 rom​

I'm also attaching the relevant part of /var/log/messages

PS: how can I only subscribe to threads I participate in (like this one) and not subscribe to the entire forum in which it is posted?

------------------------------
Jad
------------------------------

Attachments:

var_log_messages.txt


Subject: RE: serviced fails to start due to devicemapper error
Author: Jad Baz
Posted: 2018-11-07 11:09

Anyone?

------------------------------
Jad
------------------------------

Subject: RE: serviced fails to start due to devicemapper error
Author: Arthur
Posted: 2018-11-08 16:05

Hi

Looks like you have a file-system issue (Dirty inode writeback failed for block device dm-15) and then the follwoing mount fails.

Oct 30 15:20:05 LDLBN019WMZNS01 kernel: Buffer I/O error on dev dm-15, logical block 13107214, lost async page write
Oct 30 15:20:05 LDLBN019WMZNS01 kernel: JBD2: recovery failed
Oct 30 15:20:05 LDLBN019WMZNS01 kernel: EXT4-fs (dm-15): error loading journal
Oct 30 15:20:05 LDLBN019WMZNS01 kernel: VFS: Dirty inode writeback failed for block device dm-15 (err=-5).
Oct 30 15:20:05 LDLBN019WMZNS01 serviced: E1030 13:20:05.438238 05007 devicemapper.go:477] Error mounting device "5DCUDNVNWnT9m0zluVxXQ1" on "/opt/serviced/var/volumes/9u3l0amg9fhm9ps59qtu71xqt" for volume "9u3l0amg9fhm9ps59qtu71xqt": devmapper: Error mounting '/dev/mapper/docker-253:0-34009442-5DCUDNVNWnT9m0zluVxXQ1' on '/opt/serviced/var/volumes/9u3l0amg9fhm9ps59qtu71xqt': input/output error

Next step would by trying to fix the fs with fsck. Looks like you are using EXT4 where XFS is recommended as far I know :-)

https://www.zenoss.com/services-support/documentation/planning-guide


​​

------------------------------
Cheers
Arthur
------------------------------

Subject: RE: serviced fails to start due to devicemapper error
Author: Jad Baz
Posted: 2018-11-09 04:02

Thanks.

I couldn't get this to work so I wiped my server and started from scratch.
I will keep this in my if it happens again

------------------------------
Jad
------------------------------

Subject: RE: serviced fails to start due to devicemapper error
Author: Arthur
Posted: 2018-11-10 15:46

Hi Jad

Yep, that's another possibility to solve it. :-)

Thanks for replay.

------------------------------
Arthur
------------------------------


< Previous
DANGLING REFERENCE (POSKeyError) 		  Next
Windows Event Log summary truncated 		>