TECHZEN Zenoss User Community ARCHIVE  
FORUMS / Configuration & Administration / DISCUSSION

howto extend size in /dev/mapper/docker- ?

Subject: howto extend size in /dev/mapper/docker- ?
Author: Fredrik
Posted: 2018-12-21 06:08

Hello
typical, main zenoss system breaks down 3h before christmas hollidays

i tried a number of different commands but with no luck
serviced-storage resize -d /opt/serviced/var/volumes \ -o dm.thinpooldev=docker-253:0-44154960-pool c2zb7s2kdi6onzyaan9qg5vdr 50G

this is how df looks

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos_core--master-root 470785024 124272500 346512524 27% /
devtmpfs 20021600 0 20021600 0% /dev
tmpfs 20033516 0 20033516 0% /dev/shm
tmpfs 20033516 1096 20032420 1% /run
tmpfs 20033516 0 20033516 0% /sys/fs/cgroup
/dev/sda1 1038336 278408 759928 27% /boot
/dev/mapper/centos_core--master-home 450462820 32992 450429828 1% /home
tmpfs 4006704 0 4006704 0% /run/user/0
/dev/mapper/docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 103080888 92144932 5676692 95% /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr
overlay 470785024 124272500 346512524 27% /var/lib/docker/overlay/ea5dfcbf97149f48d8b4ce6718d78226c80aa448368470a29dde1e6d44be1f58/merged
overlay 470785024 124272500 346512524 27% /var/lib/docker/overlay/f825f5dacd5f4e3d02d3d5d59c855737d8bd1fe15e4115c882fbac197f8454d9/merged
overlay 470785024 124272500 346512524 27% /var/lib/docker/overlay/636a889a6761257bfb6299f852295bb7075b5baf202b440595eb0a13e69b172c/merged
overlay 470785024 124272500 346512524 27% /var/lib/docker/overlay/fcacda6324191eaab132ef475f053f2ba5e74347c2054d311cb0478d50b818e2/merged
overlay 470785024 124272500 346512524 27% /var/lib/docker/overlay/597863313de7fcc96f55d3d2351fd9c44b9c91208094a95d1ea6193a5f75fd14/merged
shm 65536 0 65536 0% /var/lib/docker/containers/1aa99077701a25784cdf75d160b6fae93889a89650c3c1e7410ba7cafb8bbd4a/shm
shm 65536 0 65536 0% /var/lib/docker/containers/1223c1084f59cbd67926dd453cbc35bf61629768e4b5873fd4d1794cac5262fe/shm
shm 65536 0 65536 0% /var/lib/docker/containers/683700f17d8886319fcdd5efec270fa39a2f919e8efd40a9b36ccbc37d03e070/shm
shm 65536 0 65536 0% /var/lib/docker/containers/43d616ac5e7e6e49fdff592ee15da41c3fc78c047da7caae1adf27098a2c25d5/shm
shm 65536 0 65536 0% /var/lib/docker/containers/3e9e3a57aaceb2a9bb08a04685ccb6e89c3b7ccb466d80d45812018a78850f95/shm
overlay 470785024 124272500 346512524 27% /var/lib/docker/overlay/192058a9c7b123a854bf3e2e117c451e36e19404aca4e6a563e6b72a9b44737b/merged
overlay 470785024 124272500 346512524 27% /var/lib/docker/overlay/ecc5179fc308a221d633946a970519590f6833e3cfe3b89854bdddcacf7cb353/merged
shm 65536 0 65536 0% /var/lib/docker/containers/c58549d867a35e6362f306041dbee31a52ac6f67dc9b472cfa25c0e69645f443/shm
shm 65536 0 65536 0% /var/lib/docker/containers/df9284cd9f0e965d31fe50fdc6aa9e5ca2ae8048dc5780bc72fcb271d610664d/shm


can someone help me with what commands to run in order to get more space in the 95% section.

regards /F

------------------------------
Fredrik
------------------------------

Subject: RE: howto extend size in /dev/mapper/docker- ?
Author: Arthur
Posted: 2018-12-22 10:05

Hi Fredrik

Maybe you need to first add additional storage to the volume group.

https://www.zenoss.com/sites/default/files/zenoss-doc/6386/feature/manage/resize.html

serviced volume status and lsblk -a output would also be of interest.

Cheers

------------------------------
Arthur
------------------------------
Subject: RE: howto extend size in /dev/mapper/docker- ?
Author: Ryan Matte
Posted: 2018-12-26 12:25

/dev/mapper/docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 103080888 92144932 5676692 95% /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

So this means you ran out of storage space on your serviced volume.

So first off, it's possible that space is not actually being used by files and is just being taken up by old data which hasn't been cleared from the volume yet.  The first thing you should try doing as root is this...

fstrim /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

See if that frees up a significant amount of space on the volume.  If it doesn't then move on to the following steps to extend the volume size (make sure you add more storage space to the underlying storage device beforehand)...

The following is an example of the steps required to extend a serviced volume...

First, stop serviced on all of your hosts including the master host.  Then do...

vgdisplay # to locate the physical device associated with that volume group, in the case of this example /dev/sdf.  Replace that with the appropriate device for your instance.

# Also, below where it says +200G, replace that with the amount of storage that you plan to add to it once the additional storage is available on the underlying device.  

pvresize /dev/sdf
vgextend serviced /dev/sdf
lvextend /dev/serviced/serviced-pool -L +200G
fstrim /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

Update the following in /etc/default/serviced on the master with the new total volume size before starting serviced back up.

SERVICED_DM_BASESIZE=300G

Hope this helps.  Cheers.



------------------------------
Ryan Matte
------------------------------

Subject: RE: howto extend size in /dev/mapper/docker- ?
Author: Ryan Matte
Posted: 2018-12-26 13:29

Some useful commands while working on this...

lsblk # to list all the block devices
lvs # to list the actual volume usage (both data and metadata)

​​

------------------------------
Ryan Matte
------------------------------

Subject: RE: howto extend size in /dev/mapper/docker- ?
Author: Fredrik
Posted: 2019-01-02 03:24

Hi and thank you for your response..however im cinda stuck.

vgdisplay
gives me

[root@zenoss ~]# vgdisplay
File descriptor 3 (/root/appliance/adminmenu.options) leaked on vgdisplay invocation. Parent PID 5646: bash
--- Volume group ---
VG Name centos_core-master
System ID
Format lvm2
Metadata Areas 3
Metadata Sequence No 11
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 3
Max PV 0
Cur PV 3
Act PV 3
VG Size <898.99 GiB
PE Size 4.00 MiB
Total PE 230141
Alloc PE / Size 229886 / 897.99 GiB
Free PE / Size 255 / 1020.00 MiB
VG UUID KE3VkC-nSXJ-DHGV-Hgiz-tnp7-eIQu-1vv11T

does that meen it´s "centos_core-master" i should use in my case?


pvresize is that on my newly added disk (sdd)
pvresize /dev/sdd
File descriptor 3 (/root/appliance/adminmenu.options) leaked on pvresize invocation. Parent PID 5646: bash
Failed to find physical volume "/dev/sdd".
0 physical volume(s) resized or updated / 0 physical volume(s) not resized


i dont have /dev/serviced what should i use?

[root@zenoss ~]# ls /dev/
autofs centos_core-master crash fb0 hugepages loop-control network_latency ppp rtc0 sdc sg2 sr0 tty1 tty15 tty20 tty26 tty31 tty37 tty42 tty48 tty53 tty59 tty7 ttyS3 vcs vcs6 vcsa5 zero
block char disk fd hwrng mapper network_throughput ptmx sda sdc3 sg3 stderr tty10 tty16 tty21 tty27 tty32 tty38 tty43 tty49 tty54 tty6 tty8 uhid vcs1 vcsa vcsa6
bsg console dm-0 full initctl mcelog null pts sda1 sdd sg4 stdin tty11 tty17 tty22 tty28 tty33 tty39 tty44 tty5 tty55 tty60 tty9 uinput vcs2 vcsa1 vfio
btrfs-control core dm-1 fuse input mem nvram random sda2 sdd1 shm stdout tty12 tty18 tty23 tty29 tty34 tty4 tty45 tty50 tty56 tty61 ttyS0 urandom vcs3 vcsa2 vga_arbiter
bus cpu dm-2 hidraw0 kmsg mqueue oldmem raw sdb sg0 snapshot tty tty13 tty19 tty24 tty3 tty35 tty40 tty46 tty51 tty57 tty62 ttyS1 usbmon0 vcs4 vcsa3 vhci
cdrom cpu_dma_latency dri hpet log net port rtc sdb1 sg1 snd tty0 tty14 tty2 tty25 tty30 tty36 tty41 tty47 tty52 tty58 tty63 ttyS2 usbmon1 vcs5 vcsa4 vhost-net


much regards /F

------------------------------
Fredrik
------------------------------

Some useful commands while working on this...

lsblk # to list all the block devices
lvs # to list the actual volume usage (both data and metadata)

​​

------------------------------
Ryan Matte
------------------------------

/dev/mapper/docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 103080888 92144932 5676692 95% /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

So this means you ran out of storage space on your serviced volume.

So first off, it's possible that space is not actually being used by files and is just being taken up by old data which hasn't been cleared from the volume yet.  The first thing you should try doing as root is this...

fstrim /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

See if that frees up a significant amount of space on the volume.  If it doesn't then move on to the following steps to extend the volume size (make sure you add more storage space to the underlying storage device beforehand)...

The following is an example of the steps required to extend a serviced volume...

First, stop serviced on all of your hosts including the master host.  Then do...

vgdisplay # to locate the physical device associated with that volume group, in the case of this example /dev/sdf.  Replace that with the appropriate device for your instance.

# Also, below where it says +200G, replace that with the amount of storage that you plan to add to it once the additional storage is available on the underlying device.

pvresize /dev/sdf
vgextend serviced /dev/sdf
lvextend /dev/serviced/serviced-pool -L +200G
fstrim /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

Update the following in /etc/default/serviced on the master with the new total volume size before starting serviced back up.

SERVICED_DM_BASESIZE=300G

Hope this helps.  Cheers.



------------------------------
Ryan Matte

Subject: RE: howto extend size in /dev/mapper/docker- ?
Author: Ryan Matte
Posted: 2019-01-03 10:36

Sorry, use pvdisplay instead or lsblk to determine the device for the serviced pool.  The pvdisplay output should look something like...

[root@zenoss5 rmatte]# pvdisplay
--- Physical volume ---
PV Name /dev/sda1
VG Name docker
PV Size 90.00 GiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 23039
Free PE 2304
Allocated PE 20735
PV UUID C7LCve-mYBU-VYlr-jnLv-NEw9-Jq6Y-qaLIJS

--- Physical volume ---
PV Name /dev/sda2
VG Name serviced
PV Size 100.00 GiB / not usable 3.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 25599
Free PE 2335
Allocated PE 23264
PV UUID 6PAzCU-gwgd-bDjD-uAja-6py1-bAAi-dWmbru

In the example output above the serviced device would be /dev/sda2.  That's the device you would reference when completing the other steps.  Run pvdisplay on your instance to locate the correct device then proceed with the rest of the steps.  If your serviced host is a vm I'd recommend cloning it beforehand in case something goes wrong (though the steps should work fine, I've used them many times).



------------------------------
Ryan Matte
------------------------------

Subject: RE: howto extend size in /dev/mapper/docker- ?
Author: Fredrik
Posted: 2019-01-23 03:16

Hello
there seems to be something different with my installation, they are all named "VG Name               centos_core-master"

which one should i extend?, centos_core-master wouldent be right i guess..or?
it´s hdd1 that´s the new 200g device

pvdisplay
File descriptor 3 (/root/appliance/adminmenu.options) leaked on pvdisplay invocation. Parent PID 19959: bash
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               centos_core-master
  PV Size               <499.00 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              127743
  Free PE               0
  Allocated PE          127743
  PV UUID               GxdaGE-PgLF-szht-Q2ha-VsJp-hG7i-s64db2
  
  --- Physical volume ---
  PV Name               /dev/sdb1
  VG Name               centos_core-master
  PV Size               <100.00 GiB / not usable 2.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              25599
  Free PE               0
  Allocated PE          25599
  PV UUID               9SPUZH-1gFZ-58ff-UGjd-qlf9-G0dW-SuV94a
  
  --- Physical volume ---
  PV Name               /dev/sdc3
  VG Name               centos_core-master
  PV Size               <300.00 GiB / not usable 2.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              76799
  Free PE               255
  Allocated PE          76544
  PV UUID               EPZB4v-Ijrg-PeqG-Nmpl-7ZCd-byck-qfIn3X​

regards /F

------------------------------
Fredrik
------------------------------

Sorry, use pvdisplay instead or lsblk to determine the device for the serviced pool.  The pvdisplay output should look something like...

[root@zenoss5 rmatte]# pvdisplay
--- Physical volume ---
PV Name /dev/sda1
VG Name docker
PV Size 90.00 GiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 23039
Free PE 2304
Allocated PE 20735
PV UUID C7LCve-mYBU-VYlr-jnLv-NEw9-Jq6Y-qaLIJS

--- Physical volume ---
PV Name /dev/sda2
VG Name serviced
PV Size 100.00 GiB / not usable 3.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 25599
Free PE 2335
Allocated PE 23264
PV UUID 6PAzCU-gwgd-bDjD-uAja-6py1-bAAi-dWmbru

In the example output above the serviced device would be /dev/sda2.  That's the device you would reference when completing the other steps.  Run pvdisplay on your instance to locate the correct device then proceed with the rest of the steps.  If your serviced host is a vm I'd recommend cloning it beforehand in case something goes wrong (though the steps should work fine, I've used them many times).



------------------------------
Ryan Matte
------------------------------

Some useful commands while working on this...

lsblk # to list all the block devices
lvs # to list the actual volume usage (both data and metadata)

​​

------------------------------
Ryan Matte

/dev/mapper/docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 103080888 92144932 5676692 95% /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

So this means you ran out of storage space on your serviced volume.

So first off, it's possible that space is not actually being used by files and is just being taken up by old data which hasn't been cleared from the volume yet.  The first thing you should try doing as root is this...

fstrim /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

See if that frees up a significant amount of space on the volume.  If it doesn't then move on to the following steps to extend the volume size (make sure you add more storage space to the underlying storage device beforehand)...

The following is an example of the steps required to extend a serviced volume...

First, stop serviced on all of your hosts including the master host.  Then do...

vgdisplay # to locate the physical device associated with that volume group, in the case of this example /dev/sdf.  Replace that with the appropriate device for your instance.

# Also, below where it says +200G, replace that with the amount of storage that you plan to add to it once the additional storage is available on the underlying device.

pvresize /dev/sdf
vgextend serviced /dev/sdf
lvextend /dev/serviced/serviced-pool -L +200G
fstrim /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

Update the following in /etc/default/serviced on the master with the new total volume size before starting serviced back up.

SERVICED_DM_BASESIZE=300G

Hope this helps.  Cheers.



------------------------------
Ryan Matte

Subject: RE: howto extend size in /dev/mapper/docker- ?
Author: Ryan Matte
Posted: 2019-01-23 14:38

Normally I would expect those volumes to be labeled according to what they are for, not sure why they are all labeled the same in this case.  They have different sizes though so perhaps you can use that as a clue as to which one you need to extend?

------------------------------
Ryan Matte
------------------------------

Subject: RE: howto extend size in /dev/mapper/docker- ?
Author: Fredrik
Posted: 2019-01-24 02:59

bare with me ;)..
It is this one i want to extend
  
  --- Physical volume ---
  PV Name               /dev/sdb1
  VG Name               centos_core-master
  PV Size               <100.00 GiB / not usable 2.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              25599
  Free PE               0
  Allocated PE          25599
  PV UUID               9SPUZH-1gFZ-58ff-UGjd-qlf9-G0dW-SuV94a

but i cant use "centos_core-master" then i´ll just add it go root partition, how do i add it to *docker partition?
  


lvextend /dev/centos_core-master/ -L +200G
/dev/centos_core-master/home  /dev/centos_core-master/root  /dev/centos_core-master/swap 
[root@zenoss5 ~]# lvextend /dev/centos_core-master/ -L +200G

------------------------------
Fredrik
Umea
------------------------------

Subject: RE: howto extend size in /dev/mapper/docker- ?
Author: Ryan Matte
Posted: 2019-01-28 16:12

/dev/sdb1 is the device name of that device.  Look at the PV Name.

------------------------------
Ryan Matte
------------------------------

Subject: RE: howto extend size in /dev/mapper/docker- ?
Author: Arthur
Posted: 2019-01-29 16:29

Hi Fredrik / Ryan

As Zenoss V5/6 normally uses the docker devicemapper storage driver, I don't think we can extend the fs by only extend the logical volume.
Based on your pvdisplay output you have 3 physical volumes in your vg "centos_core-master". sda2, sdb1, sdc3.
sda2 and sdb1 have zero free PEs. sdc3 has 255 free PE's. Wth a PE size of 4MB  you  have around 1GB free space which is not a lot.

If you still have the need to extend your serviced volume I need to know a bit more about your configuration.

Please post the following outputs:

# lsblk -a
# serviced volume status
# docker info
# grep FS /etc/default/serviced
# grep DM_ /etc/default/serviced
# vgs && lvs

Cheers

​​​​​

------------------------------
Arthur
------------------------------

Sorry, use pvdisplay instead or lsblk to determine the device for the serviced pool.  The pvdisplay output should look something like...

[root@zenoss5 rmatte]# pvdisplay
--- Physical volume ---
PV Name /dev/sda1
VG Name docker
PV Size 90.00 GiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 23039
Free PE 2304
Allocated PE 20735
PV UUID C7LCve-mYBU-VYlr-jnLv-NEw9-Jq6Y-qaLIJS

--- Physical volume ---
PV Name /dev/sda2
VG Name serviced
PV Size 100.00 GiB / not usable 3.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 25599
Free PE 2335
Allocated PE 23264
PV UUID 6PAzCU-gwgd-bDjD-uAja-6py1-bAAi-dWmbru

In the example output above the serviced device would be /dev/sda2.  That's the device you would reference when completing the other steps.  Run pvdisplay on your instance to locate the correct device then proceed with the rest of the steps.  If your serviced host is a vm I'd recommend cloning it beforehand in case something goes wrong (though the steps should work fine, I've used them many times).



------------------------------
Ryan Matte

Some useful commands while working on this...

lsblk # to list all the block devices
lvs # to list the actual volume usage (both data and metadata)

​​

------------------------------
Ryan Matte

/dev/mapper/docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 103080888 92144932 5676692 95% /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

So this means you ran out of storage space on your serviced volume.

So first off, it's possible that space is not actually being used by files and is just being taken up by old data which hasn't been cleared from the volume yet.  The first thing you should try doing as root is this...

fstrim /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

See if that frees up a significant amount of space on the volume.  If it doesn't then move on to the following steps to extend the volume size (make sure you add more storage space to the underlying storage device beforehand)...

The following is an example of the steps required to extend a serviced volume...

First, stop serviced on all of your hosts including the master host.  Then do...

vgdisplay # to locate the physical device associated with that volume group, in the case of this example /dev/sdf.  Replace that with the appropriate device for your instance.

# Also, below where it says +200G, replace that with the amount of storage that you plan to add to it once the additional storage is available on the underlying device.

pvresize /dev/sdf
vgextend serviced /dev/sdf
lvextend /dev/serviced/serviced-pool -L +200G
fstrim /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

Update the following in /etc/default/serviced on the master with the new total volume size before starting serviced back up.

SERVICED_DM_BASESIZE=300G

Hope this helps.  Cheers.



------------------------------
Ryan Matte

Subject: RE: howto extend size in /dev/mapper/docker- ?
Author: Fredrik
Posted: 2019-01-30 12:39

Hello Arthur and thankyou for your response

here is the out of the commands you requesten

[root@zenoss5 ~]# lsblk -a                                                                                                                                                                                                                      
NAME                                             MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                                                8:0    0   500G  0 disk
â"œâ"€sda1                                             8:1    0     1G  0 part /boot
â""â"€sda2                                             8:2    0   499G  0 part
  â"œâ"€centos_core--master-root                     253:0    0   449G  0 lvm  /
  â"œâ"€centos_core--master-swap                     253:1    0  19.2G  0 lvm  [SWAP]
  â""â"€centos_core--master-home                     253:2    0 429.8G  0 lvm  /home
sdb                                                8:16   0   100G  0 disk
â""â"€sdb1                                             8:17   0   100G  0 part
  â""â"€centos_core--master-root                     253:0    0   449G  0 lvm  /
sdc                                                8:32   0   300G  0 disk
â""â"€sdc3                                             8:35   0   300G  0 part
  â""â"€centos_core--master-root                     253:0    0   449G  0 lvm  /
sdd                                                8:48   0   200G  0 disk
â""â"€sdd1                                             8:49   0   200G  0 part
sr0                                               11:0    1   4.4G  0 rom 
loop0                                              7:0    0   100G  0 loop
â""â"€docker-253:0-44154960-pool                     253:3    0   100G  0 dm  
  â""â"€docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 253:4    0   100G  0 dm   /exports/serviced_volumes_v2/c2zb7s2kdi6onzyaan9qg5vdr
loop1                                              7:1    0     2G  0 loop
â""â"€docker-253:0-44154960-pool                     253:3    0   100G  0 dm  
  â""â"€docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 253:4    0   100G  0 dm   /exports/serviced_volumes_v2/c2zb7s2kdi6onzyaan9qg5vdr
[root@zenoss5 ~]# serviced volume status
Status for volume /opt/serviced/var/volumes:
Driver:      devicemapper
Driver Type: loop-lvm
Volume Path: /opt/serviced/var/volumes
Thin Pool
---------
Logical Volume:               docker-253:0-44154960-pool
Metadata (total/used/avail):  2 GiB   / 60.91 MiB (3%)  / 1.941 GiB (97%)
Data (total/used/avail):      100 GiB / 95.14 GiB (95%) / 4.855 GiB (4.9%)
c2zb7s2kdi6onzyaan9qg5vdr Application Data
-----------------------------------------
Volume Mount Point:            /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr
Filesystem (total/used/avail): 98.31 GiB / 83.64 GiB (85%) / 9.653 GiB (9.8%)
Virtual device size:           100 GiB

[root@zenoss5 ~]# docker info
Containers: 38
 Running: 38
 Paused: 0
 Stopped: 0
Images: 4
Server Version: 17.09.0-ce
Storage Driver: overlay
 Backing Filesystem: xfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-957.1.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 38.21GiB
Name: zenoss5.net.umea.se
ID: PC7S:LOW2:J6GL:RFAC:OYTM:T7R4:HVGV:6WLV:CCI4:ZXAY:HD7Q:3QWN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[root@zenoss5 ~]# grep FS /etc/default/serviced
# SERVICED_FS_TYPE=devicemapper
# Set to 0 in order to prevent this host from attempting to mount the DFS
# SERVICED_NFS_CLIENT=1
# The timeout for performing a DFS snapshot (in seconds)
# SERVICED_MAX_DFS_TIMEOUT=300
[root@zenoss5 ~]# grep DM_ /etc/default/serviced
# SERVICED_DM_ARGS=
SERVICED_DM_BASESIZE=300G
#SERVICED_DM_LOOPDATASIZE=100G
# SERVICED_DM_LOOPMETADATASIZE=2G
# SERVICED_DM_THINPOOLDEV=
[root@zenoss5 ~]# vgs && lvs
File descriptor 3 (/root/appliance/adminmenu.options) leaked on vgs invocation. Parent PID 3620: bash
  VG                 #PV #LV #SN Attr   VSize VFree 
  centos_core-master   4   3   0 wz--n- 1.07t 200.99g
File descriptor 3 (/root/appliance/adminmenu.options) leaked on lvs invocation. Parent PID 3620: bash
  LV   VG                 Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home centos_core-master -wi-ao---- 429.80g                                                   
  root centos_core-master -wi-ao---- 449.00g                                                   
  swap centos_core-master -wi-ao---- <19.19g  


regards /F

------------------------------
Fredrik
Umea
------------------------------

Subject: RE: howto extend size in /dev/mapper/docker- ?
Author: Arthur
Posted: 2019-01-30 16:29

Hi Fredrik

Let's start from the top. For the docker stuff you are using the overlay storage driver. It's not recommended by Zenoss but it's not the point here.

For the serviced stuff you are using the docker devicemapper storage driver in loopback-lvm mode (loop-lvm mode is not supported) – but ok.

https://docs.docker.com/storage/storagedriver/select-storage-driver/

The devicemapper storage driver uses a thin-pool which has two storage areas, Data and Metadata.

In your case it looks like you have two files mounted (loop0 100GB/Data and loop1 2GBMetadata)

loop0                                                                             7:0   0   100G  0 loop

docker-253:0-44154960-pool                     253:3    0   100G  0 dm  

  docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 253:4    0   100G  0 dm   /exports/serviced_volumes_v2/c2zb7s2kdi6onzyaan9qg5vdr

 

loop1                                                                              7:1    0     2G  0 loop

docker-253:0-44154960-pool                     253:3    0   100G  0 dm  

  docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 253:4    0   100G  0 dm   /exports/serviced_volumes_v2/c2zb7s2kdi6onzyaan9qg5vdr

 

If you look at your serviced volume status output, you can see that your data volume has only 4.855GiB space left. The Metadata is only 3% used which is still good enough for me.

 Volume Path: /opt/serviced/var/volumes
Thin Pool
---------
Logical Volume:               docker-253:0-44154960-pool
Metadata (total/used/avail):  2 GiB   / 60.91 MiB (3%)  / 1.941 GiB (97%)
Data (total/used/avail):      100 GiB / 95.14 GiB (95%) / 4.855 GiB (4.9%)
c2zb7s2kdi6onzyaan9qg5vdr Application Data
-----------------------------------------
Volume Mount Point:            /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr
Filesystem (total/used/avail): 98.31 GiB / 83.64 GiB (85%) / 9.653 GiB (9.8%)
Virtual device size:           100 GiB

So, from my point of view have to extend the loop1 device. Unfortunately I haven't any experience in resizing loop devices. I found some information on the below site under "Resize a loop-lvm thin pool"

https://docs.docker.com/storage/storagedriver/device-mapper-driver/

 

Maybe some other readers do have and are willing to jump in!

 

In the meantime I will check out if I can find some more information.

 

Cheers Arthur

 

P.S I saw that you already added 200GB (sdd)  to the environment.


Subject: RE: howto extend size in /dev/mapper/docker- ?
Author: Arthur
Posted: 2019-02-07 16:38

Hi Fredrik

The metadata and data sparse file building the servcied thin-pool are probably located under /opt/serviced/var/volumes/.devicemapper/devicemapper.

Can you please run the following commands and post the results?

# df -h | grep volumes
# ll -alh /opt/serviced/var/volumes/.devicemapper/devicemapper/
# grep -v ^# /etc/default/serviced | grep .

# dmsetup ls |grep pool | sort
# dmsetup status |grep pool | sort

Cheers


------------------------------
Arthur
------------------------------

Hi Fredrik

Let's start from the top. For the docker stuff you are using the overlay storage driver. It's not recommended by Zenoss but it's not the point here.

For the serviced stuff you are using the docker devicemapper storage driver in loopback-lvm mode (loop-lvm mode is not supported) – but ok.

https://docs.docker.com/storage/storagedriver/select-storage-driver/

The devicemapper storage driver uses a thin-pool which has two storage areas, Data and Metadata.

In your case it looks like you have two files mounted (loop0 100GB/Data and loop1 2GBMetadata)

loop0                                                                             7:0   0   100G  0 loop

docker-253:0-44154960-pool                     253:3    0   100G  0 dm  

  docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 253:4    0   100G  0 dm   /exports/serviced_volumes_v2/c2zb7s2kdi6onzyaan9qg5vdr

 

loop1                                                                              7:1    0     2G  0 loop

docker-253:0-44154960-pool                     253:3    0   100G  0 dm  

  docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 253:4    0   100G  0 dm   /exports/serviced_volumes_v2/c2zb7s2kdi6onzyaan9qg5vdr

 

If you look at your serviced volume status output, you can see that your data volume has only 4.855GiB space left. The Metadata is only 3% used which is still good enough for me.

 Volume Path: /opt/serviced/var/volumes
Thin Pool
---------
Logical Volume:               docker-253:0-44154960-pool
Metadata (total/used/avail):  2 GiB   / 60.91 MiB (3%)  / 1.941 GiB (97%)
Data (total/used/avail):      100 GiB / 95.14 GiB (95%) / 4.855 GiB (4.9%)
c2zb7s2kdi6onzyaan9qg5vdr Application Data
-----------------------------------------
Volume Mount Point:            /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr
Filesystem (total/used/avail): 98.31 GiB / 83.64 GiB (85%) / 9.653 GiB (9.8%)
Virtual device size:           100 GiB

So, from my point of view have to extend the loop1 device. Unfortunately I haven't any experience in resizing loop devices. I found some information on the below site under "Resize a loop-lvm thin pool"

https://docs.docker.com/storage/storagedriver/device-mapper-driver/

 

Maybe some other readers do have and are willing to jump in!

 

In the meantime I will check out if I can find some more information.

 

Cheers Arthur

 

P.S I saw that you already added 200GB (sdd)  to the environment.

Sorry, use pvdisplay instead or lsblk to determine the device for the serviced pool.  The pvdisplay output should look something like...

[root@zenoss5 rmatte]# pvdisplay
--- Physical volume ---
PV Name /dev/sda1
VG Name docker
PV Size 90.00 GiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 23039
Free PE 2304
Allocated PE 20735
PV UUID C7LCve-mYBU-VYlr-jnLv-NEw9-Jq6Y-qaLIJS

--- Physical volume ---
PV Name /dev/sda2
VG Name serviced
PV Size 100.00 GiB / not usable 3.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 25599
Free PE 2335
Allocated PE 23264
PV UUID 6PAzCU-gwgd-bDjD-uAja-6py1-bAAi-dWmbru

In the example output above the serviced device would be /dev/sda2.  That's the device you would reference when completing the other steps.  Run pvdisplay on your instance to locate the correct device then proceed with the rest of the steps.  If your serviced host is a vm I'd recommend cloning it beforehand in case something goes wrong (though the steps should work fine, I've used them many times).



------------------------------
Ryan Matte

Some useful commands while working on this...

lsblk # to list all the block devices
lvs # to list the actual volume usage (both data and metadata)

​​

------------------------------
Ryan Matte

/dev/mapper/docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 103080888 92144932 5676692 95% /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

So this means you ran out of storage space on your serviced volume.

So first off, it's possible that space is not actually being used by files and is just being taken up by old data which hasn't been cleared from the volume yet.  The first thing you should try doing as root is this...

fstrim /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

See if that frees up a significant amount of space on the volume.  If it doesn't then move on to the following steps to extend the volume size (make sure you add more storage space to the underlying storage device beforehand)...

The following is an example of the steps required to extend a serviced volume...

First, stop serviced on all of your hosts including the master host.  Then do...

vgdisplay # to locate the physical device associated with that volume group, in the case of this example /dev/sdf.  Replace that with the appropriate device for your instance.

# Also, below where it says +200G, replace that with the amount of storage that you plan to add to it once the additional storage is available on the underlying device.

pvresize /dev/sdf
vgextend serviced /dev/sdf
lvextend /dev/serviced/serviced-pool -L +200G
fstrim /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

Update the following in /etc/default/serviced on the master with the new total volume size before starting serviced back up.

SERVICED_DM_BASESIZE=300G

Hope this helps.  Cheers.



------------------------------
Ryan Matte

Subject: RE: howto extend size in /dev/mapper/docker- ?
Author: Arthur
Posted: 2019-02-27 15:44

Hi Fredrik

Attached a cheet sheet how to increase a dm lvm loop thin-pool.

Cheers

------------------------------
Arthur
------------------------------


< Previous
Over Ridden Templates 		  Next
Zenoss 6.2.1, Zope stops answering on its own, unprovoked 		>