TECHZEN Zenoss User Community ARCHIVE  

serviced snapshot commit

Subject: serviced snapshot commit
Author: Joana Sianard
Posted: 2019-03-06 17:05

Hello,
I am installing the toolbox,
when I try a commit, here's the message back !

API error (500): {"message":"max depth exceeded"}

I have a multiple node configuration:
1 control center
2 resources pool

Regard




------------------------------
JOANA SIANARD
SYSADMIN
GLOBECAST
PARIS FRANCE
Zenoss 5.3.3 r419
Control Center ControlCenter 1.4.2
------------------------------


Subject: RE: serviced snapshot commit
Author: Arthur
Posted: 2019-03-07 14:19

Hello
Did you follow the steps described in this article (Permanent or Semi-Permanent)?
https://support.zenoss.com/hc/en-us/articles/207610516-How-to-add-tools-or-scripts-into-a-Resource-Manager-5-x-Docker-Container

------------------------------
Arthur
------------------------------


Subject: RE: serviced snapshot commit
Author: Joana Sianard
Posted: 2019-03-07 17:27

hello
I have the same problem when I install a zenpack or I restore !

INFO:zen.ZenPackCmd:Successfully restored zenpacks!
FATA[0339] Unable to commit container containerid=b8236d11c88d6ec302d0c9b2881f053a0b0fccdf2f05a2c34232f2eafea9cb80 containername=4jpVZOxxuVDkEB9egl44Dw error=API error (500): {"message":"max depth exceeded"}
location=shell.go:264 logger=cli.api

------------------------------
JOANA SIANARD
SYSADMIN
GLOBECAST
PARIS FRANCE
Zenoss 5.3.3 r419
Control Center ControlCenter 1.4.2
------------------------------


Subject: RE: serviced snapshot commit
Author: Arthur
Posted: 2019-03-08 16:35

Can you post the output of the following commands:

# docker info && serviced volume status
# img=$(docker images |grep core |grep latest | awk {'print $1'}) && docker history $img

​​​​

------------------------------
Arthur
------------------------------


Subject: RE: serviced snapshot commit
Author: Joana Sianard
Posted: 2019-03-08 18:05

Hello Arthur,

the files.

Regards,
JOANA

------------------------------
JOANA SIANARD
GLOBECAST
------------------------------

Attachments:

docker-history.txt

docker-info.txt

volume-status.txt



Subject: RE: serviced snapshot commit
Author: Arthur
Posted: 2019-03-09 03:01

Hello Joana

It looks like your image has too many layers (126). As I haven't seen this message so far this is onl my assumption and I can't give you a solution yet. You can google for docker AND "message":"max depth exceeded" for example to find some help how to solve it.

Anyhow I would be interested about your solution/findings. If I have some free time I'll try to reproduce it, but I can't give you any date so far.

------------------------------
Arthur
------------------------------


Subject: RE: serviced snapshot commit
Author: Andrew Kirch
Posted: 2019-03-13 14:35

This happens when your docker container has a lot of layers.  Have you added a lot of things to this before?  (Zenpacks etc).  I would suggest flattening it if so.  Note that you may lose the ability to roll back.  

https://tuhrig.de/flatten-a-docker-container-or-image/



------------------------------
Andrew Kirch
Senior Solutions Engineer
GoVanguard
------------------------------


Subject: RE: serviced snapshot commit
Author: Duane Wylie
Posted: 2021-11-05 17:13

Has anyone found a workable solution to this issue?  I'm running into the same problem on Zenoss Core 6.3.2.

------------------------------
Duane Wylie
------------------------------


Subject: RE: serviced snapshot commit
Author: Michael Rogers
Posted: 2021-11-05 18:55

Duane,

The following steps should take care of things.

0) Note - There is a restart of the Zenoss application in this process under step 10

1) First we need to find out what our Zenoss Core image is by running:

[root@core ~]# ZCORE="$(docker images | grep core_ | awk '{print $1}' | head -1)"
[root@core ~]# echo $ZCORE
localhost:5000/5ipo4ged9ffy4gze3zqkq2tdh/core_6.1

2) Check current image layer count:

[root@core ~]# docker history $ZCORE | wc -l
37

3) We need the sha256 hash of the latest layer from the docker history once we have that we run:

[root@core ~]# IMAGE_ID="$(docker history --no-trunc -q $ZCORE | head -1)"
[root@core ~]# echo $IMAGE_ID
sha256:bcc1c00a7c9fd08c19566fd7e1ccbffe61477801279aac8b0356076716005834

4) Create a backup of the current image:

[root@core ~]# docker tag $ZCORE:latest $ZCORE:latest-backup
[root@core ~]# docker images | grep backup
localhost:5000/5ipo4ged9ffy4gze3zqkq2tdh/core_6.1 latest-backup bcc1c00a7c9f 8 weeks ago 2.22GB

5) Create a new container based on the latest layer:

[root@core ~]# docker run --name "corecontainer" $IMAGE_ID /bin/bash
[root@core ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
09efecb488be bcc1c00a7c9f "/bin/bash" 7 seconds ago Exited (0) 5 seconds ago corecontainer

* If serviced is running you will also see other containers including each isvcs

6) Use this manually started container to flatten the current image by performing an export/import (this will take a few minutes):

[root@core ~]# docker export corecontainer | docker import - flat-corecontainer
sha256:d99d9c93c6e65bfe905079382bbc9f081b58efaf420b678b1a93c9ebbdfb5874

* The sha256 line will show when complete

7) Override the current image with the new, flattened image:

[root@core ~]# serviced docker override $ZCORE:latest flat-corecontainer:latest
[root@core ~]# docker images | grep latest | head -n3
flat-corecontainer latest d99d9c93c6e6 2 minutes ago 1.96GB
localhost:5000/5ipo4ged9ffy4gze3zqkq2tdh/core_6.1 latest d99d9c93c6e6 2 minutes ago 1.96GB
localhost:5000/5ipo4ged9ffy4gze3zqkq2tdh/core_6.1 latest-backup bcc1c00a7c9f 8 weeks ago 2.22GB

8) Verify your image has been flattened:

[root@core ~]# docker history $ZCORE | wc -l
2

9) Asynchronously push all images from the serviced Docker registry index into the Docker registry (may take a minute to complete):

[root@core ~]# serviced docker sync

* Command produces no output

10) Restart Zenoss Core to pull the latest image for each container:

serviced service restart Zenoss.core

* Restarting immediately isn't necessarily required, but plan to perform a restart before performing any further image edits.

If the process fails, your original container will still be backed up with the latest-backup tag from Step 4.  

Let us know if this helps?

------------------------------
Michael J. Rogers
Senior Instructor - Zenoss
Austin TX
------------------------------


Subject: RE: serviced snapshot commit
Author: Duane Wylie
Posted: 2021-11-05 20:25

Thanks, Michael.

I've run through this process without issue through step 9.  (I'll restart a bit later).

On the off chance that this detonates in my face, I imagine the rollback would look something like this?

[root@core ~]# serviced docker override $ZCORE:latest $ZCORE:latest-backup
[root@core ~]# serviced docker sync
serviced service restart Zenoss.core


------------------------------
Duane Wylie
------------------------------


Subject: RE: serviced snapshot commit
Author: Michael Rogers
Posted: 2021-11-05 21:48

Duane,

I should have included the instructions for rollback in my last post, but that looks perfect.

------------------------------
Michael J. Rogers
Senior Instructor - Zenoss
Austin TX
------------------------------


Subject: RE: serviced snapshot commit
Author: Duane Wylie
Posted: 2021-11-05 23:56

After restarting with the flattened image, stats from zenperfsnmp weren't being graphed.  (but stats from zenprocess were.)

I tried reverting to the $ZCORE:latest-backup, but this didn't resolve my issue.

I'm restoring the system from backup (hopefully I haven't completely lost the system).

I was seeing a lot of errors like this in various container logs:

error="Identity token signature cannot be verified"

------------------------------
Duane Wylie
------------------------------


< Previous
Using ValueChangeThreshold type in ZenPack
  Next
Zenpack Development Testing
>