TECHZEN Zenoss User Community ARCHIVE  

Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexi ...

Subject: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events
Author: Sam Urai
Posted: 2022-01-12 20:13

2022-01-12T13:31:20.926 [INDEXER_EVENT_SUMMARY] ERROR org.springframework.transaction.interceptor.T
ransactionInterceptor - Application exception overridden by rollback exception
org.springframework.dao.DeadlockLoserDataAccessException: PreparedStatementCallback; SQL [INSERT IN
TO event_trigger_signal_spool (created,event_summary_uuid,event_trigger_subscription_uuid,flush_tim
e,event_count,sent_signal,uuid) VALUES(?,?,?,?,?,?,?)]; Deadlock found when trying to get lock; try
 restarting transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollba
ckException: Deadlock found when trying to get lock; try restarting transaction
        at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLError
CodeSQLExceptionTranslator.java:265) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]

This error is followed by bunch of other jdbc errors. It seems due to the deadlock, a transaction failure happens and then it gets rolled back at lease that's what I see in the errors following it.    However the major impact is that lots of events gets stacked up in the RabbitMqueue until the deadlock goes away.

Is there something needed to tune mysql or something to avoid this deadlock?

Please advise,
Thanks





------------------------------
Sam
------------------------------


Subject: RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events
Author: Michael Rogers
Posted: 2022-01-12 21:00

Sam,

If you check the archived Zenoss 4.2.5 Installation Manual, step 2.5 directs you to the MySQL Tuner script at mysqltuner.pl.

Later versions point toward the Percona Toolkit instead.  It's been a while since I've laid a hand on either of them, so I can't recommend one over the other.

------------------------------
Michael J. Rogers
Senior Instructor - Zenoss
Austin TX
------------------------------


Subject: RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events
Author: Sam Urai
Posted: 2022-01-21 12:55

Here is the deadlock information provided by

mysql> show engine innodb status\G

*************************** 1. row ***************************
  Type: InnoDB
  Name:
Status:
=====================================
220119 10:10:36 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 38 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 7878348 1_second, 7878344 sleeps, 787831 10_second, 42 background, 42 flush
srv_master_thread log flush and writes: 7893321
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 29621057, signal count 44703141
Mutex spin waits 251001014, rounds 448583868, OS waits 2082513
RW-shared spins 14501169, rounds 154491848, OS waits 3185354
RW-excl spins 36891245, rounds 770873766, OS waits 24055859
Spin rounds per wait: 1.79 mutex, 10.65 RW-shared, 20.90 RW-excl
------------------------
LATEST DETECTED DEADLOCK
------------------------
220119  8:52:33
*** (1) TRANSACTION:
TRANSACTION 35583B36, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 6 lock struct(s), heap size 1248, 2 row lock(s)
MySQL thread id 922890, OS thread handle 0x7fd30c07e700, query id 210213559 localhost 127.0.0.1 zenoss statistics
SELECT event_count,first_seen,last_seen,details_json,status_id,uuid FROM event_summary WHERE fingerprint_hash=x'A784                            D0E9A2E2B0C19116665D777B252FDEA93E9F' FOR UPDATE
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 1117028 n bits 72 index `PRIMARY` of table `zenoss_zep`.`event_summary` trx id 35583                            B36 lock_mode X locks rec but not gap waiting
Record lock, heap no 4 PHYSICAL RECORD: n_fields 40; compact format; info bits 0


*** (2) TRANSACTION:
TRANSACTION 35583B35, ACTIVE 0 sec updating or deleting
mysql tables in use 1, locked 1
9 lock struct(s), heap size 1248, 11 row lock(s), undo log entries 11
MySQL thread id 922930, OS thread handle 0x7fd2f0895700, query id 210213568 localhost 127.0.0.1 zenoss Updating
UPDATE event_summary SET status_id=4,status_change=1642611153281,update_time=1642611153281,current_user_uuid=null,cu                            rrent_user_name=null,cleared_by_event_uuid=x'5254007B971A89FF11EC794832755829',fingerprint_hash=x'9E0AFDD08B913BAF0D                            5B8C0FB3AAC023488CE954',audit_json='{"timestamp":1642611153281,"new_status":4}' WHERE uuid=x'5254007B971A89FF11EC794                            83274BBE8'
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 0 page no 1117028 n bits 72 index `PRIMARY` of table `zenoss_zep`.`event_summary` trx id 35583                            B35 lock_mode X locks rec but not gap
Record lock, heap no 4 PHYSICAL RECORD: n_fields 40; compact format; info bits 0


*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 4360710 n bits 320 index `fingerprint_hash` of table `zenoss_zep`.`event_summary` tr                            x id 35583B35 lock_mode X locks rec but not gap waiting
Record lock, heap no 250 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 0: len 20; hex a784d0e9a2e2b0c19116665d777b252fdea93e9f; asc           f]w{%/  > ;;
 1: len 16; hex 5254007b971a89ff11ec79483274bbe8; asc RT {      yH2t  ;;

*** WE ROLL BACK TRANSACTION (1)
​


------------------------------
Sam
------------------------------


Subject: RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events
Author: Michael Rogers
Posted: 2022-01-24 17:22

Sam,

Did you have an opportunity to test either the mysqltuner.pl or Percona toolkit scripts?

Also, my initial response had a typo in the link for mysqltuner.pl.  It should have been http:// and not https://.  I have since corrected it here.

------------------------------
Michael J. Rogers
Senior Instructor - Zenoss
Austin TX
------------------------------


Subject: RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events
Author: Sam Urai
Posted: 2022-01-27 18:58

I haven't had a chance yet but I will run the toolkit scripts to get some insights.  I suppose a performance tuning may be required. 

Thanks

------------------------------
Sam
------------------------------


Subject: RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events
Author: Sam Urai
Posted: 2022-02-02 19:36

I've set this flag in zeneventserver.conf and hope that it will help. 

zep.database.optimize_minutes = 0​


------------------------------
Sam
------------------------------


Subject: RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events
Author: Michael Rogers
Posted: 2022-02-03 12:02

According to the comments in zeneventserver.conf, an optimize_minutes value of 0 or less will disable optimization entirely.  I'm concerned that could lead to trouble down the line.

If you have the Percona toolkit installed, you can use the zep.database.optimize_external_tool_path option to point at the executable.

Did some particular resource recommend disabling the database optimization feature?


------------------------------
Michael J. Rogers
Senior Instructor - Zenoss
Austin TX
------------------------------


Subject: RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events
Author: Sam Urai
Posted: 2022-02-03 16:04

Yes. Here is the Zenoss tuning wiki page I came across  - http://wiki.zenoss.org/Zenoss_tuning 

Optimize flag
By default Zeneventserver is configured to issue an 'optimize table' against several key tables. Under certain circumstances this can cause the system to stop processing events during the lock. Optimization of these tables, in theory, renders only minor improvement in performance since the tables in question are frequently partitioned off.

In the $ZENHOME/etc/zeneventserver.conf, uncomment 'zep.database.optimize_minutes' and set it equal to 0.​

Thanks



------------------------------
Sam
------------------------------


Subject: RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events
Author: Michael Rogers
Posted: 2022-02-03 16:25

Sam,

Good find, and thanks for clarifying!  

If you opt to turn table optimization back on, we do have a KB article about using the Percona toolkit:
https://support.zenoss.com/hc/en-us/articles/203103735-How-To-Use-The-Percona-Toolkit-To-Speed-Up-Event-Database-Optimization


------------------------------
Michael J. Rogers
Senior Instructor - Zenoss
Austin TX
------------------------------


Subject: RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events
Author: Sam Urai
Posted: 2022-02-03 20:08

Thanks Michael.  The article is good reference and I've it bookmarked

I did conclude that it is the optimization causing the deadlock based on the timeframes of its run and the events stuck in the queues during that period.

Thanks

------------------------------
Sam
------------------------------


< Previous
Modeler plugin - zenoss.snmp.SnmpV3EngineIdMap
  Next
Configure Zenoss 4.2.5 Zope UI to use SSL
>