Zenoss ZenTech Community

Subject:	RE: serviced backup no such image
Author:	Jane Curry
Posted:	2017-07-11 05:32

Not personally - but I haven't really used / trusted any backup concept of Zenoss 5 other than VM snapshots ;)

What versions are you on?

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

Subject:	RE: serviced backup no such image
Author:	Tim Caiazza
Posted:	2017-07-11 09:20

I'm running core 5.2.1, don't really want to upgrade till I sort out the backup issue.

------------------------------
Tim Caiazza
------------------------------

Subject:	RE: serviced backup no such image
Author:	Eric Ward
Posted:	2017-07-11 10:04

Hi,

We have the same issue. Yesterday we upgraded again to 5.2.6 and 1.3.3 and see the same issue. We are working with Zenoss support. I'll keep you posted.

E

------------------------------
Eric
Sys Admin
Restaurant Technologies
mendota heights MN
------------------------------

Subject:	RE: serviced backup no such image
Author:	Jagadish N
Posted:	2017-07-11 10:29

Tim,

Just some time before i have tested this backup function in RM 5.2.6 , CC- 1.3.3, I didn't find any issues.

If I am not wrong you are using different backup directory path, which doesn't exist. The default backup path is "/opt/serviced/var/backups/"

Here it is...

[root@slzen ~]# serviced backup /opt/serviced/var/backups/
backup-2017-07-11-142136.tgz
[root@slzen ~]#

Cheers
Jagadish Nagasamudram

------------------------------
Jagadish N
------------------------------

Subject:	RE: serviced backup no such image
Author:	Tim Caiazza
Posted:	2017-07-11 17:07

I tried that and no such luck

[tim@shnetmonitor5: ~] serviced backup /opt/serviced/var/backups/
no such image
[tim@shnetmonitor5: ~]

At some point I did have backups working to another directory, which I changed for the post here. I even moved the last successful backup to the /opt/serviced/var/backups/ dir just in case the backup command did something like read the last backup to get the image name. Unfortunately I don't know enough GO to figure out where the image name is supposed to be coming from.

------------------------------
Tim Caiazza
------------------------------

Subject:	RE: serviced backup no such image
Author:	Arthur
Posted:	2017-07-12 16:38

Hi

For me the command is ok.

[root@~]# serviced backup -h
NAME:
backup - Dump all templates and services to a tgz file

USAGE:
command backup [command options] [arguments...]

DESCRIPTION:
serviced backup DIRPATH <----------

OPTIONS:
--exclude '--exclude option --exclude option' Subdirectory of the tenant volume to exclude from backup
--check check space, but do not do backup

--force attempt backup even if space check fails

It seems that the backup is looking for the docker images. See my journalctl output after starting a backup. Maybe you can share or check your systemd journal which hopefully give you a hint.

--------------

[root@~]# journalctl --since "2017-07-12 21:00:00" |grep image

Jul 12 21:57:08 host serviced[2367]: time="2017-07-12T21:57:08+02:00" level=info msg="Loaded templates and their images" elapsed=208.319357ms location="dfs.go:81" logger=facade
Jul 12 21:57:13 host serviced[2367]: time="2017-07-12T21:57:13+02:00" level=info msg="Preparing docker images for backup" backupversion=1 location="backup.go:59" logger=dfs timestamp=2017-07-12 21:57:08.446157888 +0200 CEST total=11
Jul 12 21:57:13 host serviced[2367]: time="2017-07-12T21:57:13+02:00" level=info msg="Prepared Docker image for backup" backupversion=1 image="zenoss/core_5.2:5.2.4_1" location="backup.go:81" logger=dfs numbercomplete=1 timestamp=2017-07-12 21:57:08.446157888 +0200 CEST total=11
Jul 12 21:57:13 host serviced[2367]: time="2017-07-12T21:57:13+02:00" level=info msg="Prepared Docker image for backup" backupversion=1 image="zenoss/hbase:24.0.3" location="backup.go:81" logger=dfs numbercomplete=2 timestamp=2017-07-12 21:57:08.446157888 +0200 CEST total=11
Jul 12 21:57:13 host serviced[2367]: time="2017-07-12T21:57:13+02:00" level=info msg="Prepared Docker image for backup" backupversion=1 image="zenoss/opentsdb:24.0.3" location="backup.go:81" logger=dfs numbercomplete=3 timestamp=2017-07-12 21:57:08.446157888 +0200 CEST total=11
Jul 12 21:57:13 host serviced[2367]: time="2017-07-12T21:57:13+02:00" level=info msg="Prepared Docker image for backup" backupversion=1 image="zenoss/core_5.2:5.2.3_1" location="backup.go:81" logger=dfs numbercomplete=4 timestamp=2017-07-12 21:57:08.446157888 +0200 CEST total=11
Jul 12 21:57:13 host serviced[2367]: time="2017-07-12T21:57:13+02:00" level=info msg="Prepared Docker image for backup" backupversion=1 image="zenoss/core_5.2:5.2.0_1" location="backup.go:81" logger=dfs numbercomplete=5 timestamp=2017-07-12 21:57:08.446157888 +0200 CEST total=11
Jul 12 21:57:13 host serviced[2367]: time="2017-07-12T21:57:13+02:00" level=info msg="Prepared Docker image for backup" backupversion=1 image="zenoss/hbase:24.0.2" location="backup.go:81" logger=dfs numbercomplete=6 timestamp=2017-07-12 21:57:08.446157888 +0200 CEST total=11
Jul 12 21:57:13 host serviced[2367]: time="2017-07-12T21:57:13+02:00" level=info msg="Prepared Docker image for backup" backupversion=1 image="zenoss/opentsdb:24.0.2" location="backup.go:81" logger=dfs numbercomplete=7 timestamp=2017-07-12 21:57:08.446157888 +0200 CEST total=11
Jul 12 21:57:13 host serviced[2367]: time="2017-07-12T21:57:13+02:00" level=info msg="Prepared Docker image for backup" backupversion=1 image="zenoss/core_5.1:5.1.1_1" location="backup.go:81" logger=dfs numbercomplete=8 timestamp=2017-07-12 21:57:08.446157888 +0200 CEST total=11
Jul 12 21:57:13 host serviced[2367]: time="2017-07-12T21:57:13+02:00" level=info msg="Prepared Docker image for backup" backupversion=1 image="zenoss/hbase:v10" location="backup.go:81" logger=dfs numbercomplete=9 timestamp=2017-07-12 21:57:08.446157888 +0200 CEST total=11
Jul 12 21:57:13 host serviced[2367]: time="2017-07-12T21:57:13+02:00" level=info msg="Prepared Docker image for backup" backupversion=1 image="zenoss/opentsdb:v17" location="backup.go:81" logger=dfs numbercomplete=10 timestamp=2017-07-12 21:57:08.446157888 +0200 CEST total=11
Jul 12 21:57:13 host serviced[2367]: time="2017-07-12T21:57:13+02:00" level=info msg="Prepared Docker image for backup" backupversion=1 image="monitoringartist/zenoss5grafana2demo:latest" location="backup.go:81" logger=dfs numbercomplete=11 timestamp=2017-07-12 21:57:08.446157888 +0200 CEST total=11
Jul 12 21:57:13 host serviced[2367]: time="2017-07-12T21:57:13+02:00" level=info msg="Preparing images for tenant" backupversion=1 location="backup.go:99" logger=dfs tenant=e47so56gvai7f2jioknki83v7 timestamp=2017-07-12 21:57:08.446157888 +0200 CEST
Jul 12 21:57:13 host serviced[2367]: I0712 21:57:13.371570 02367 pull.go:88] Looking up image localhost:5000/e47so56gvai7f2jioknki83v7/zenoss5grafana2demo:20170712-195708.722
Jul 12 21:57:13 host serviced[2367]: time="2017-07-12T21:57:13+02:00" level=info msg="Prepared Docker image for backup" image="localhost:5000/e47so56gvai7f2jioknki83v7/zenoss5grafana2demo:20170712-195708.722" location="backup.go:129" logger=dfs
Jul 12 21:57:14 host serviced[2367]: time="2017-07-12T21:57:14+02:00" level=info msg="Preparing images for tenant" backupversion=1 location="backup.go:99" logger=dfs tenant=4qvth4i0ilwrkqym51thyajvw timestamp=2017-07-12 21:57:08.446157888 +0200 CEST
Jul 12 21:57:14 host serviced[2367]: I0712 21:57:14.802685 02367 pull.go:88] Looking up image localhost:5000/4qvth4i0ilwrkqym51thyajvw/hbase:20170712-195712.500
Jul 12 21:57:14 host serviced[2367]: time="2017-07-12T21:57:14+02:00" level=info msg="Prepared Docker image for backup" image="localhost:5000/4qvth4i0ilwrkqym51thyajvw/hbase:20170712-195712.500" location="backup.go:129" logger=dfs
Jul 12 21:57:14 host serviced[2367]: I0712 21:57:14.805235 02367 pull.go:88] Looking up image localhost:5000/4qvth4i0ilwrkqym51thyajvw/opentsdb:20170712-195712.500
Jul 12 21:57:14 host serviced[2367]: time="2017-07-12T21:57:14+02:00" level=info msg="Prepared Docker image for backup" image="localhost:5000/4qvth4i0ilwrkqym51thyajvw/opentsdb:20170712-195712.500" location="backup.go:129" logger=dfs
Jul 12 21:57:14 host serviced[2367]: I0712 21:57:14.807441 02367 pull.go:88] Looking up image localhost:5000/4qvth4i0ilwrkqym51thyajvw/core_5.2:20170712-195712.500
Jul 12 21:57:14 host serviced[2367]: time="2017-07-12T21:57:14+02:00" level=info msg="Prepared Docker image for backup" image="localhost:5000/4qvth4i0ilwrkqym51thyajvw/core_5.2:20170712-195712.500" location="backup.go:129" logger=dfs
Jul 12 21:58:44 host serviced[2367]: time="2017-07-12T21:58:44+02:00" level=info msg="Starting export of images to backup" backupversion=1 images=[zenoss/core_5.2:5.2.4_1 zenoss/hbase:24.0.3 zenoss/opentsdb:24.0.3 zenoss/core_5.2:5.2.3_1 zenoss/core_5.2:5.2.0_1 zenoss/hbase:24.0.2 zenoss/opentsdb:24.0.2 zenoss/core_5.1:5.1.1_1 zenoss/hbase:v10 zenoss/opentsdb:v17 monitoringartist/zenoss5grafana2demo:latest localhost:5000/e47so56gvai7f2jioknki83v7/zenoss5grafana2demo:20170712-195708.722 localhost:5000/4qvth4i0ilwrkqym51thyajvw/hbase:20170712-195712.500 localhost:5000/4qvth4i0ilwrkqym51thyajvw/opentsdb:20170712-195712.500 localhost:5000/4qvth4i0ilwrkqym51thyajvw/core_5.2:20170712-195712.500] location="backup.go:160" logger=dfs timestamp=2017-07-12 21:57:08.446157888 +0200 CEST
Jul 12 21:58:44 host serviced[2367]: time="2017-07-12T21:58:44+02:00" level=info msg="Starting image export" images=[zenoss/core_5.2:5.2.4_1 zenoss/hbase:24.0.3 zenoss/opentsdb:24.0.3 zenoss/core_5.2:5.2.3_1 zenoss/core_5.2:5.2.0_1 zenoss/hbase:24.0.2 zenoss/opentsdb:24.0.2 zenoss/core_5.1:5.1.1_1 zenoss/hbase:v10 zenoss/opentsdb:v17 monitoringartist/zenoss5grafana2demo:latest localhost:5000/e47so56gvai7f2jioknki83v7/zenoss5grafana2demo:20170712-195708.722 localhost:5000/4qvth4i0ilwrkqym51thyajvw/hbase:20170712-195712.500 localhost:5000/4qvth4i0ilwrkqym51thyajvw/opentsdb:20170712-195712.500 localhost:5000/4qvth4i0ilwrkqym51thyajvw/core_5.2:20170712-195712.500] location="docker.go:146" logger=dfs.docker
Jul 12 22:01:41 host serviced[2367]: time="2017-07-12T22:01:41+02:00" level=info msg="Finished image export." location="docker.go:156" logger=dfs.docker
Jul 12 22:01:41 host serviced[2367]: time="2017-07-12T22:01:41+02:00" level=info msg="Exported images to backup" backupversion=1 images=[zenoss/core_5.2:5.2.4_1 zenoss/hbase:24.0.3 zenoss/opentsdb:24.0.3 zenoss/core_5.2:5.2.3_1 zenoss/core_5.2:5.2.0_1 zenoss/hbase:24.0.2 zenoss/opentsdb:24.0.2 zenoss/core_5.1:5.1.1_1 zenoss/hbase:v10 zenoss/opentsdb:v17 monitoringartist/zenoss5grafana2demo:latest localhost:5000/e47so56gvai7f2jioknki83v7/zenoss5grafana2demo:20170712-195708.722 localhost:5000/4qvth4i0ilwrkqym51thyajvw/hbase:20170712-195712.500 localhost:5000/4qvth4i0ilwrkqym51thyajvw/opentsdb:20170712-195712.500 localhost:5000/4qvth4i0ilwrkqym51thyajvw/core_5.2:20170712-195712.500] location="backup.go:172" logger=dfs timestamp=2017-07-12 21:57:08.446157888 +0200 CEST

----------------

------------------------------
Arthur
------------------------------

Subject:	RE: serviced backup no such image
Author:	Eric Ward
Posted:	2017-07-13 14:57

We had to retag an old image for each error we saw:

# Jul 10 16:07:35 zen-pli-ccp01 serviced[32250]: time="2017-07-10T20:07:35Z" level=info msg="Unable to get history for image zenoss/resmgr_5.1:5.1.5_1" error="no such image" location="docker.go:109"

docker tag localhost:5000/ce64qy6gqp73fgjmguu8ay4a4/resmgr_5.1:latest zenoss/resmgr_5.1:5.1.5_1

# Jul 13 14:28:54 zen-pli-ccp01 serviced[2613]: time="2017-07-13T18:28:54Z" level=info msg="Unable to get history for image zenoss/hbase:v12.1" error="no such image" location="docker.go:109" logger=dfs.docker

docker tag localhost:5000/ce64qy6gqp73fgjmguu8ay4a4/hbase:20170627-202243.262 zenoss/hbase:v12.1

# Jul 13 14:35:30 zen-pli-ccp01 serviced[2613]: time="2017-07-13T18:35:30Z" level=info msg="Unable to get history for image zenoss/opentsdb:v19.1" error="no such image" location="docker.go:109" logger=dfs.docker

docker tag localhost:5000/ce64qy6gqp73fgjmguu8ay4a4/opentsdb:20170627-202243.262 zenoss/opentsdb:v19.1

------------------------------
Eric
Sys Admin
Restaurant Technologies

------------------------------

Subject:	RE: serviced backup no such image
Author:	Tim Caiazza
Posted:	2017-07-14 15:46

Thanks Eric! I was able to retag all the images and the "no such image" error stopped. I now have a new error but I'll make a separate post about that.

To be clear for those who find this later, here is exactly what I did to re tag the image.

[root@shnetmonitor5: ~] journalctl --since "2017-07-12 21:00:00" |grep image
Error was encountered while opening journal files: Not a directory
Jul 14 13:53:30 shnetmonitor5 dockerd[16863]: time="2017-07-14T13:53:30.547229282-04:00" level=error msg="Handler for GET /images/zenoss/core_5.2:5.2.0_1/history returned error: no such id: zenoss/core_5.2:5.2.0_1"
Jul 14 13:53:30 shnetmonitor5 serviced[12250]: time="2017-07-14T17:53:30Z" level=info msg="Unable to get history for image zenoss/core_5.2:5.2.0_1" error="no such image" location="docker.go:109" logger=dfs.docker
Jul 14 13:53:30 shnetmonitor5 serviced[12250]: time="2017-07-14T17:53:30Z" level=info msg="Could not get size for images." error="no such image" location="dfs.go:184" logger=facade

[root@shnetmonitor5: ~] docker ps | grep core_5.2 | awk '{print $2}' | sort -u
localhost:5000/18unh0yppkc165scv1hs426nk/core_5.2:latest

[root@shnetmonitor5: ~] docker tag localhost:5000/18unh0yppkc165scv1hs426nk/core_5.2:latest zenoss/core_5.2:5.2.0_1

So I looked through the journalctl logs, found the name of the image service was looking for, which you can see is zenoss/core_5.2:5.2.0_1 Then I looked through the docker image names for something like that and found localhost:5000/18unh0yppkc165scv1hs426nk/core_5.2:latest then I tagged that image name with what serviced was looking for. Then I ran the serviced backup command only for it to fail again, this time looking for a hbase image which i fixed the same way. I repeated these steps until I sorted all the image names.

Note: If you have multiple machines that your containers can run on you may have to run the docker ps command on another machine but you can run the tag command again on your control center master machine.

------------------------------
Tim Caiazza
------------------------------

Subject:	RE: serviced backup no such image
Author:	Janusz Kujawa
Posted:	2017-09-12 04:56

Hello,
Have you tried to build your own Application Template. I've done and I had same problem. When I removed my own template then "serviced backup /opt/serviced/var/backups/" start works.

------------------------------
Janusz Kujawa
App engineer
N/A
NC
------------------------------

Subject:	serviced backup no such image
Author:	Tim Caiazza
Posted:	2017-07-10 14:39

serviced backup no such image