.
instead. It's been a while since I've laid a hand on either of them, so I can't recommend one over the other.
Michael J. Rogers
Subject: |
RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events |
Author: |
Sam Urai |
Posted: |
2022-01-21 12:55 |
Here is the deadlock information provided by
mysql> show engine innodb status\G
*************************** 1. row ***************************
Type: InnoDB
Name:
Status:
=====================================
220119 10:10:36 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 38 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 7878348 1_second, 7878344 sleeps, 787831 10_second, 42 background, 42 flush
srv_master_thread log flush and writes: 7893321
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 29621057, signal count 44703141
Mutex spin waits 251001014, rounds 448583868, OS waits 2082513
RW-shared spins 14501169, rounds 154491848, OS waits 3185354
RW-excl spins 36891245, rounds 770873766, OS waits 24055859
Spin rounds per wait: 1.79 mutex, 10.65 RW-shared, 20.90 RW-excl
------------------------
LATEST DETECTED DEADLOCK
------------------------
220119 8:52:33
*** (1) TRANSACTION:
TRANSACTION 35583B36, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 6 lock struct(s), heap size 1248, 2 row lock(s)
MySQL thread id 922890, OS thread handle 0x7fd30c07e700, query id 210213559 localhost 127.0.0.1 zenoss statistics
SELECT event_count,first_seen,last_seen,details_json,status_id,uuid FROM event_summary WHERE fingerprint_hash=x'A784 D0E9A2E2B0C19116665D777B252FDEA93E9F' FOR UPDATE
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 1117028 n bits 72 index `PRIMARY` of table `zenoss_zep`.`event_summary` trx id 35583 B36 lock_mode X locks rec but not gap waiting
Record lock, heap no 4 PHYSICAL RECORD: n_fields 40; compact format; info bits 0
*** (2) TRANSACTION:
TRANSACTION 35583B35, ACTIVE 0 sec updating or deleting
mysql tables in use 1, locked 1
9 lock struct(s), heap size 1248, 11 row lock(s), undo log entries 11
MySQL thread id 922930, OS thread handle 0x7fd2f0895700, query id 210213568 localhost 127.0.0.1 zenoss Updating
UPDATE event_summary SET status_id=4,status_change=1642611153281,update_time=1642611153281,current_user_uuid=null,cu rrent_user_name=null,cleared_by_event_uuid=x'5254007B971A89FF11EC794832755829',fingerprint_hash=x'9E0AFDD08B913BAF0D 5B8C0FB3AAC023488CE954',audit_json='{"timestamp":1642611153281,"new_status":4}' WHERE uuid=x'5254007B971A89FF11EC794 83274BBE8'
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 0 page no 1117028 n bits 72 index `PRIMARY` of table `zenoss_zep`.`event_summary` trx id 35583 B35 lock_mode X locks rec but not gap
Record lock, heap no 4 PHYSICAL RECORD: n_fields 40; compact format; info bits 0
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 4360710 n bits 320 index `fingerprint_hash` of table `zenoss_zep`.`event_summary` tr x id 35583B35 lock_mode X locks rec but not gap waiting
Record lock, heap no 250 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
0: len 20; hex a784d0e9a2e2b0c19116665d777b252fdea93e9f; asc f]w{%/ > ;;
1: len 16; hex 5254007b971a89ff11ec79483274bbe8; asc RT { yH2t ;;
*** WE ROLL BACK TRANSACTION (1)
------------------------------
Sam
------------------------------
Subject: |
RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events |
Author: |
Michael Rogers |
Posted: |
2022-01-24 17:22 |
Sam,
Did you have an opportunity to test either the mysqltuner.pl or Percona toolkit scripts?
Also, my initial response had a typo in the link for
mysqltuner.pl. It should have been
http://
and not
https://
. I have since corrected it here.
------------------------------
Michael J. Rogers
Senior Instructor - Zenoss
Austin TX
------------------------------
Subject: |
RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events |
Author: |
Sam Urai |
Posted: |
2022-01-27 18:58 |
I haven't had a chance yet but I will run the toolkit scripts to get some insights. I suppose a performance tuning may be required.
Thanks
------------------------------
Sam
------------------------------
Subject: |
RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events |
Author: |
Sam Urai |
Posted: |
2022-02-02 19:36 |
I've set this flag in zeneventserver.conf and hope that it will help.
zep.database.optimize_minutes = 0
------------------------------
Sam
------------------------------
Subject: |
RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events |
Author: |
Michael Rogers |
Posted: |
2022-02-03 12:02 |
According to the comments in zeneventserver.conf, an
optimize_minutes
value of 0 or less will disable optimization entirely. I'm concerned that could lead to trouble down the line.
If you have the Percona toolkit installed, you can use the
zep.database.optimize_external_tool_path
option to point at the executable.
Did some particular resource recommend disabling the database optimization feature?------------------------------
Michael J. Rogers
Senior Instructor - Zenoss
Austin TX
------------------------------
Subject: |
RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events |
Author: |
Sam Urai |
Posted: |
2022-02-03 16:04 |
Yes. Here is the Zenoss tuning wiki page I came across - http://wiki.zenoss.org/Zenoss_tuning
Optimize flag
By default Zeneventserver is configured to issue an 'optimize table' against several key tables. Under certain circumstances this can cause the system to stop processing events during the lock. Optimization of these tables, in theory, renders only minor improvement in performance since the tables in question are frequently partitioned off.
In the $ZENHOME/etc/zeneventserver.conf, uncomment 'zep.database.optimize_minutes' and set it equal to 0.
Thanks
------------------------------
Sam
------------------------------
Subject: |
RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events |
Author: |
Michael Rogers |
Posted: |
2022-02-03 16:25 |
Sam,
Good find, and thanks for clarifying!
If you opt to turn table optimization back on, we do have a KB article about using the Percona toolkit:
https://support.zenoss.com/hc/en-us/articles/203103735-How-To-Use-The-Percona-Toolkit-To-Speed-Up-Event-Database-Optimization------------------------------
Michael J. Rogers
Senior Instructor - Zenoss
Austin TX
------------------------------
Subject: |
RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events |
Author: |
Sam Urai |
Posted: |
2022-02-03 20:08 |
Thanks Michael. The article is good reference and I've it bookmarked
I did conclude that it is the optimization causing the deadlock based on the timeframes of its run and the events stuck in the queues during that period.
Thanks
------------------------------
Sam
------------------------------