Zenoss ZenTech Community

Serviced backup Broken due to input/output error

Subject:

Serviced backup Broken due to input/output error

Author:

Tim Caiazza

Posted:

2017-07-14 15:55

[root@shnetmonitor5: ~] serviced backup /systems/backup/zenoss/
devmapper: Error mounting '/dev/mapper/docker-253:0-268966914-jWCvh3RKTOlbHyCAwnHh' on '/tmp/serviced-export-volume-868712882': input/output error

Jul 14 15:50:29 shnetmonitor5 serviced[12250]: time="2017-07-14T19:50:29Z" level=info msg="Backp estimate done" elapsed=621.85703ms estimate=&{424605756948480 115276832741 4 TB 115 GB /systems/backup/zenoss/ true} location="filesystem.go:171" logger=dao.elasticsearch
Jul 14 15:50:30 shnetmonitor5 serviced[12250]: time="2017-07-14T19:50:30Z" level=info msg="Started backup" excludes=[] location="dfs.go:75" logger=facade
Jul 14 15:50:33 shnetmonitor5 serviced[12250]: time="2017-07-14T19:50:33Z" level=info msg="Preparing docker images for backup" backupversion=1 location="backup.go:59" logger=dfs timestamp=2017-07-14 19:50:30.076160669 +0000 UTC total=3
Jul 14 15:50:33 shnetmonitor5 serviced[12250]: time="2017-07-14T19:50:33Z" level=info msg="Prepared Docker image for backup" backupversion=1 image="zenoss/core_5.2:5.2.0_1" location="backup.go:81" logger=dfs numbercomplete=1 timestamp=2017-07-14 19:50:30.076160669 +0000 UTC total=3
Jul 14 15:50:33 shnetmonitor5 serviced[12250]: time="2017-07-14T19:50:33Z" level=info msg="Prepared Docker image for backup" backupversion=1 image="zenoss/hbase:24.0.2" location="backup.go:81" logger=dfs numbercomplete=2 timestamp=2017-07-14 19:50:30.076160669 +0000 UTC total=3
Jul 14 15:50:33 shnetmonitor5 serviced[12250]: time="2017-07-14T19:50:33Z" level=info msg="Prepared Docker image for backup" backupversion=1 image="zenoss/opentsdb:24.0.2" location="backup.go:81" logger=dfs numbercomplete=3 timestamp=2017-07-14 19:50:30.076160669 +0000 UTC total=3
Jul 14 15:50:33 shnetmonitor5 serviced[12250]: time="2017-07-14T19:50:33Z" level=info msg="Preparing snapshots for backup" backupversion=1 location="backup.go:88" logger=dfs timestamp=2017-07-14 19:50:30.076160669 +0000 UTC total=1
Jul 14 15:50:33 shnetmonitor5 serviced[12250]: time="2017-07-14T19:50:33Z" level=info msg="Preparing images for tenant" backupversion=1 location="backup.go:99" logger=dfs tenant=18unh0yppkc165scv1hs426nk timestamp=2017-07-14 19:50:30.076160669 +0000 UTC
Jul 14 15:50:33 shnetmonitor5 serviced[12250]: time="2017-07-14T19:50:33Z" level=info msg="Prepared Docker image for backup" image="localhost:5000/18unh0yppkc165scv1hs426nk/core_5.2:20170714-195033.266" location="backup.go:129" logger=dfs
Jul 14 15:50:33 shnetmonitor5 serviced[12250]: time="2017-07-14T19:50:33Z" level=info msg="Prepared Docker image for backup" image="localhost:5000/18unh0yppkc165scv1hs426nk/opentsdb:20170714-195033.266" location="backup.go:129" logger=dfs
Jul 14 15:50:33 shnetmonitor5 serviced[12250]: time="2017-07-14T19:50:33Z" level=info msg="Prepared Docker image for backup" image="localhost:5000/18unh0yppkc165scv1hs426nk/hbase:20170714-195033.266" location="backup.go:129" logger=dfs
Jul 14 15:50:34 shnetmonitor5 serviced[12250]: time="2017-07-14T19:50:34Z" level=error msg="Could not export snapshot for backup" backupversion=1 error="devmapper: Error mounting '/dev/mapper/docker-253:0-268966914-jWCvh3RKTOlbHyCAwnHh' on '/tmp/serviced-export-volume-868712882': input/output error" location="backup.go:147" logger=dfs snapshot="18unh0yppkc165scv1hs426nk_20170714-195033.266" timestamp=2017-07-14 19:50:30.076160669 +0000 UTC

Subject:	RE: Serviced backup Broken due to input/output error
Author:	Arthur
Posted:	2017-07-18 14:40

Hi
Do you have any snapshots lef which are corrupted?

# serviced snapshot list

------------------------------
Arthur
------------------------------

Subject:	RE: Serviced backup Broken due to input/output error
Author:	Tim Caiazza
Posted:	2017-07-18 22:32

Nope no snapshots

[tim@zenoss5: ~] serviced snapshot list

no snapshots found

[tim@zenoss5 ~]

------------------------------
Tim Caiazza
------------------------------

Subject:	RE: Serviced backup Broken due to input/output error
Author:	Arthur
Posted:	2017-07-19 16:15

Try running

# serviced-storage check

The serviced-storage check command searches for orphaned snapshot devices in the serviced

application data thin pool and removes them, if requested.

Details can be found in the Control Center Reference Guide
https://www.zenoss.com/services-support/documentation

------------------------------
Arthur
------------------------------

Subject:	RE: Serviced backup Broken due to input/output error
Author:	Tim Caiazza
Posted:	2017-07-20 09:52

[root@zenoss5: ~] serviced-storage check
driver not initialized

But on my test box I get the same error and backups work there fine.

[root@zendev]# serviced-storage check
driver not initialized

Thanks for the help Arthur.

------------------------------
Tim Caiazza
------------------------------

Subject:	RE: Serviced backup Broken due to input/output error
Author:	Arthur
Posted:	2017-07-20 14:35

Hi

Identify the serviced application data thin pool, and then remove orphaned snapshot devices

[root@ ~]# ls /dev/mapper | grep serviced
serviced-serviced--pool
serviced-serviced--pool_tdata
serviced-serviced--pool_tmeta

[root@ ~]# serviced-storage -o dm.thinpooldev=/dev/mapper/serviced-serviced--pool check /opt/serviced/var/volumes
No orphaned devices found.

Cheers Arthur

Subject:	RE: Serviced backup Broken due to input/output error
Author:	Tim Caiazza
Posted:	2017-07-21 09:32

[root@zenoss5] serviced-storage -o dm.thinpooldev=/dev/mapper/serviced-serviced--pool check /opt/serviced/var/volumes
WARN[0000] devmapper: Base device already exists and has filesystem xfs on it. User specified filesystem ext4 will be ignored.
No orphaned devices found.

------------------------------
Tim Caiazza
------------------------------

Subject:	RE: Serviced backup Broken due to input/output error
Author:	Arthur
Posted:	2017-07-24 15:09

Hi Tim

I' fishing a bit in the dark right now. Maybe someone who is more familar with docker and devicemapper can jump in. If you like you can post the output of
# lsblk -a
# df -h
to find out where the block device is mounted.

------------------------------
Arthur
------------------------------