Greatsql MGR集群其中一个节点一直UNREACHABLE无法恢复
本帖最后由 running_db 于 2023-12-5 16:04 编辑1、环境说明
OS：Rocky Linux release 8.8 5.4.259-1.el8.elrepo.x86_64
DB：GreatSQL-8.0.32-24-Linux-glibc2.28-x86_64
相关参数：
loose-group_replication_unreachable_majority_timeout = 30
loose-group_replication_member_expel_timeout = 5
(Mon Dec4 17:50:16 2023)[(none)]>show variables like 'group_replication_unreachable_majority_timeout';
+------------------------------------------------+-------+
| Variable_name | Value |
+------------------------------------------------+-------+
| group_replication_unreachable_majority_timeout | 30 |
+------------------------------------------------+-------+
1 row in set (0.00 sec)
(Mon Dec4 17:50:29 2023)[(none)]>show variables like 'group_replication_member_expel_timeout';
+----------------------------------------+-------+
| Variable_name | Value |
+----------------------------------------+-------+
| group_replication_member_expel_timeout | 5 |
+----------------------------------------+-------+
2、问题描述
192.168.6.188 PRIMARY
192.168.6.189 SECONDARY
192.168.6.190 SECONDARY
在批量导入数据过程中，发现三节点mgr中其中一个SECONDARY节点190处于UNREACHABLE状态，且，重启190数据库和重启操作系统，手动执行start group_replication都无法加入集群。
3、具体报错
PRIMARY：一直重复一下错误Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17016823274129381. Please stop the old node or wait for it to leave the group.'
2023-12-04T17:32:08.439396+08:00 0 Plugin group_replication reported: ' set fd:52 connected'
2023-12-04T17:32:08.439578+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:52'
2023-12-04T17:32:08.440629+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17016823274129381. Please stop the old node or wait for it to leave the group.'
2023-12-04T17:32:08.441617+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:52 in close_connection'
2023-12-04T17:32:08.539157+08:00 0 Plugin group_replication reported: ' set fd:65 connected'
2023-12-04T17:32:08.539322+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:65'
2023-12-04T17:32:08.540069+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17016823274129381. Please stop the old node or wait for it to leave the group.'
2023-12-04T17:32:08.540974+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:65 in close_connection'
2023-12-04T17:32:08.638952+08:00 0 Plugin group_replication reported: ' set fd:52 connected'
2023-12-04T17:32:08.639208+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:52'
2023-12-04T17:32:08.640082+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17016823274129381. Please stop the old node or wait for it to leave the group.'
2023-12-04T17:32:08.641199+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:52 in close_connection'
190 SECONDARY
大量重复：
2023-12-04T17:44:16.818907+08:00 0 Plugin group_replication reported: ' TCP_NODELAY already set'
2023-12-04T17:44:16.819005+08:00 0 Plugin group_replication reported: ' Sucessfully connected to peer 192.168.6.189:33081. Sending a request to be added to the group'
2023-12-04T17:44:16.819043+08:00 0 Plugin group_replication reported: ' Sending add_node request to a peer XCom node'
2023-12-04T17:44:16.844098+08:00 0 Plugin group_replication reported: ' xcom_send_client_app_data sets CON_PROTO for fd:49'
2023-12-04T17:44:16.844845+08:00 0 Plugin group_replication reported: ' Sending a request to a remote XCom failed. Please check the remote node log for more details.'
2023-12-04T17:44:16.844930+08:00 0 Plugin group_replication reported: ' Failed to send add_node request to a peer XCom node.'其余日志：
2023-12-04T17:44:16.844930+08:00 0 Plugin group_replication reported: ' Failed to send add_node request to a peer XCom node.'
2023-12-04T17:44:16.845118+08:00 0 Plugin group_replication reported: ' Error connecting to all peers. Member join failed. Local port: 33081'
2023-12-04T17:44:16.846761+08:00 0 Plugin group_replication reported: ' 1701683056.846689 pid 992 xcom_id 7b487969 state xcom_fsm_start action x_fsm_terminate'
2023-12-04T17:44:16.846922+08:00 0 Plugin group_replication reported: ' 1701683056.846903 pid 992 xcom_id 7b487969 state xcom_fsm_start action x_fsm_exit'
2023-12-04T17:44:16.846977+08:00 0 Plugin group_replication reported: ' Exiting xcom thread'
2023-12-04T17:44:17.503680+08:00 0 Plugin group_replication reported: ' The member was unable to join the group. Local port: 33081'
当前状态：
+---------------------------+--------------------------------------+---------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+---------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | a55de711-812e-11ee-b3fc-5254005b7a28 | 192.168.6.190 | 3308 | UNREACHABLE| SECONDARY | 8.0.32 | XCom |
| group_replication_applier | a5609358-812e-11ee-abe4-525400b01dcf | 192.168.6.188 | 3308 | ONLINE | PRIMARY | 8.0.32 | XCom |
| group_replication_applier | a5623edf-812e-11ee-92a9-525400980c24 | 192.168.6.189 | 3308 | ONLINE | SECONDARY | 8.0.32 | XCom |
+---------------------------+--------------------------------------+---------------+-------------+--------------+-------------+----------------+----------------------------+
补充：
190节点于Dec4 16:53:31发生oom，自动重启
/var/log/messeage
Dec4 16:53:31 greatsql03 kernel: Tasks state (memory values in pages):
Dec4 16:53:31 greatsql03 kernel: uidtgid total_vm rss pgtables_bytes swapents oom_score_adj name
Dec4 16:53:31 greatsql03 kernel: [ 489] 0 489 26700 1196 258048 0 0 systemd-journal
Dec4 16:53:31 greatsql03 kernel: [ 515] 0 515 24476 522 208896 0 -1000 systemd-udevd
Dec4 16:53:31 greatsql03 kernel: [ 545] 0 545 14276 154 126976 0 -1000 auditd
Dec4 16:53:31 greatsql03 kernel: [ 591] 0 591 31252 161 151552 0 0 irqbalance
Dec4 16:53:31 greatsql03 kernel: [ 594] 81 594 14126 204 147456 0 -900 dbus-daemon
Dec4 16:53:31 greatsql03 kernel: [ 595] 0 595 147629 605 393216 0 0 NetworkManager
Dec4 16:53:31 greatsql03 kernel: [ 597] 0 597 20395 227 192512 0 0 systemd-logind
Dec4 16:53:31 greatsql03 kernel: [ 601] 996 601 35018 177 180224 0 0 chronyd
Dec4 16:53:31 greatsql03 kernel: [ 625] 0 625 173448 4113 438272 0 0 tuned
Dec4 16:53:31 greatsql03 kernel: [ 627] 0 627 19162 226 188416 0 -1000 sshd
Dec4 16:53:31 greatsql03 kernel: [ 631] 0 631 58906 213 102400 0 0 crond
Dec4 16:53:31 greatsql03 kernel: [ 635] 0 635 54902 28 65536 0 0 agetty
Dec4 16:53:31 greatsql03 kernel: [ 659] 0 659 64761 1084 233472 0 0 rsyslogd
Dec4 16:53:31 greatsql03 kernel: [ 843] 998 843 436316 1265 348160 0 0 polkitd
Dec4 16:53:31 greatsql03 kernel: [ 1030] 60110301486663 930214 10153984 0 0 mysqld
Dec4 16:53:31 greatsql03 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/greatsql.service,task=mysqld,pid=1030,uid=601
Dec4 16:53:31 greatsql03 kernel: Out of memory: Killed process 1030 (mysqld) total-vm:5946652kB, anon-rss:3720856kB, file-rss:0kB, shmem-rss:0kB, UID:601 pgtables:9916kB oom_score_adj:0
Dec4 16:53:31 greatsql03 systemd: greatsql.service: Main process exited, code=killed, status=9/KILL
Dec4 16:53:31 greatsql03 systemd: greatsql.service: Failed with result 'signal'.
Dec4 16:53:31 greatsql03 systemd: greatsql.service: Service RestartSec=100ms expired, scheduling restart.
Dec4 16:53:31 greatsql03 systemd: greatsql.service: Scheduled restart job, restart counter is at 1.
Dec4 16:53:31 greatsql03 systemd: Stopped GreatSQL Server.
Dec4 16:53:31 greatsql03 systemd: Starting GreatSQL Server...
Dec4 16:53:42 greatsql03 systemd: Started GreatSQL Server.
Dec4 17:00:01 greatsql03 systemd: Starting system activity accounting tool...
188当时的日志：
2023-12-04T16:27:06.008289+08:00 285892 DDL log post ddl : begin for thread id : 285892
2023-12-04T16:27:06.008375+08:00 285892 DDL log post ddl : end for thread id : 285892
2023-12-04T16:29:42.099037+08:00 285908 Plugin group_replication reported: 'MGR request time:628458us, server id:188, thread_id:285908'
2023-12-04T16:32:08.618436+08:00 285908 Plugin group_replication reported: 'MGR request time:1228858us, server id:188, thread_id:285908'
2023-12-04T16:34:31.903747+08:00 285908 Plugin group_replication reported: 'MGR request time:924387us, server id:188, thread_id:285908'
2023-12-04T16:35:40.021236+08:00 285868 'wait_timeout' period of 600 seconds was exceeded for `root`@`localhost`. The idle time since last command was too long.
2023-12-04T16:35:40.021484+08:00 285868 Aborted connection 285868 to db: 'unconnected' user: 'root' host: 'localhost' (The client was disconnected by the server because of inactivity.).
2023-12-04T16:36:55.493502+08:00 285908 Plugin group_replication reported: 'MGR request time:1099862us, server id:188, thread_id:285908'
2023-12-04T16:39:18.889251+08:00 285908 Plugin group_replication reported: 'MGR request time:1026873us, server id:188, thread_id:285908'
2023-12-04T16:41:42.790207+08:00 285908 Plugin group_replication reported: 'MGR request time:1074727us, server id:188, thread_id:285908'
2023-12-04T16:44:03.741350+08:00 285908 Plugin group_replication reported: 'MGR request time:1402106us, server id:188, thread_id:285908'
2023-12-04T16:46:23.746283+08:00 285908 Plugin group_replication reported: 'MGR request time:1014195us, server id:188, thread_id:285908'
2023-12-04T16:48:46.386828+08:00 285908 Plugin group_replication reported: 'MGR request time:1056717us, server id:188, thread_id:285908'
2023-12-04T16:49:48.111022+08:00 285979 'wait_timeout' period of 600 seconds was exceeded for `root`@`localhost`. The idle time since last command was too long.
2023-12-04T16:49:48.111302+08:00 285979 Aborted connection 285979 to db: 'unconnected' user: 'root' host: 'localhost' (The client was disconnected by the server because of inactivity.).
2023-12-04T16:51:09.286181+08:00 285908 Plugin group_replication reported: 'MGR request time:1114375us, server id:188, thread_id:285908'
2023-12-04T16:53:30.862585+08:00 285908 Plugin group_replication reported: 'MGR request time:1008435us, server id:188, thread_id:285908'
2023-12-04T16:53:31.651559+08:00 0 Plugin group_replication reported: ' Failure reading from fd=62 n=18446744073709551615 from 192.168.6.190:33081'
2023-12-04T16:53:31.651693+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-04T16:53:31.652006+08:00 0 Plugin group_replication reported: ' fast_skip_allowed_for_kill is set here'
2023-12-04T16:53:31.652147+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:67 in close_connection'
2023-12-04T16:53:31.717465+08:00 0 Plugin group_replication reported: ' Sender task disconnected from 192.168.6.190:33081'
2023-12-04T16:53:31.717555+08:00 0 Plugin group_replication reported: ' Connecting to 192.168.6.190:33081'
2023-12-04T16:53:31.717987+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:31.817881+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:31.928331+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:32.049835+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:32.183331+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:32.329817+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:32.490846+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:32.668289+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:32.802803+08:00 0 Plugin group_replication reported: ' A configuration change was detected. Sending a Global View Message to all nodes. My node identifier is 0 and my address is 192.168.6.188:33081'
2023-12-04T16:53:32.802941+08:00 0 Plugin group_replication reported: ' call send_my_view in detector'
2023-12-04T16:53:32.802982+08:00 0 Plugin group_replication reported: ' send_my_view calls xcom_send'
2023-12-04T16:53:32.803009+08:00 0 Plugin group_replication reported: ' notify is set true in check_local_node_set'
2023-12-04T16:53:32.803030+08:00 0 Plugin group_replication reported: ' call deliver_view_msg in detector'
2023-12-04T16:53:32.803204+08:00 0 Plugin group_replication reported: ' xcom_receive_local_view is called'
2023-12-04T16:53:32.803333+08:00 0 Plugin group_replication reported: 'on_suspicions is activated'
2023-12-04T16:53:32.803453+08:00 0 Plugin group_replication reported: 'Member with address 192.168.6.190:3308 has become unreachable.'
2023-12-04T16:53:32.803612+08:00 0 Plugin group_replication reported: 'on_suspicions is called over'
2023-12-04T16:53:32.803652+08:00 0 Plugin group_replication reported: ' xcom_receive_local_view return true'
2023-12-04T16:53:32.803710+08:00 0 Plugin greatdb_ha reported: 'Error: try to bind vip failed'
2023-12-04T16:53:32.805957+08:00 0 Plugin group_replication reported: ' ::xcom_receive_global_view() is called'
2023-12-04T16:53:32.862833+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:33.077281+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:33.312944+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:33.572701+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:33.858344+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:34.172197+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:34.517920+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:34.898032+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
2023-12-04T16:53:35.316105+08:00 0 Plugin group_replication reported: ' Connection to 192.168.6.190:33081 failed'
可能是因为192.168.6.190节点之前是组复制的成员,但是没有正常下线而离开了组。导致它的一些状态信息没有得到清理,还残留在其他成员节点上。
现在它重新加入组时,其他节点检测到它有一个旧的 incarnation 17016823274129381,所以拒绝让它加入。
试着清理一下组复制信息看看有没有效果
RESET SLAVE;
在190异常退出后想重新加入，在primary节点上日志发现：Old incarnation found关键字，表示在原来的mgr里，190节点还没被驱逐出去。
试着这样做：
1. 暂停190节点反复重试加入mgr。
2. 观察188 primary节点上最新产生的日志，看看mgr状态是怎样的。
3. 提供188 primary节点上的事务状态。
show engine innodb status\G
4. 提供在188和189两个节点上看到的mgr状态。
参考这个方法提供数据：https://greatsql.cn/docs/8032/user-manual/8-mgr/4-mgr-status-monitor.html#_2-mgr%E4%BA%8B%E5%8A%A1%E7%8A%B6%E6%80%81%E7%9B%91%E6%8E%A7 本帖最后由 running_db 于 2023-12-5 11:43 编辑
yejr 发表于 2023-12-5 11:05
在190异常退出后想重新加入，在primary节点上日志发现：Old incarnation found关键字，表示在原来的mgr里， ...
1. stop group_replication;select sleep(1);start group_replication;操作了6次
2. 这6次中，188master日志如下,不管几次操作都是如下类似重复日志：
2023-12-05T11:22:22.771569+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:22.868889+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:22:22.869056+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:22:22.870079+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465417096765. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:22.871105+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:22:22.968579+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:22.968725+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:22.969724+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465417096765. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:22.970809+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:23.068229+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:22:23.068375+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:22:23.069379+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465417096765. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:23.070655+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:22:23.167730+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:23.167877+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:23.168850+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465417096765. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:23.169991+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:23.267696+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:22:23.267843+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:22:23.268747+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465417096765. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:23.269624+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:22:23.367403+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:23.367580+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:23.368629+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465417096765. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:23.369691+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:23.466530+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:22:23.466676+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:22:23.467677+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465417096765. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:23.468775+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:22:23.566365+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:23.566532+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:23.567516+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465417096765. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:23.568428+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:23.665596+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:22:23.665744+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:22:23.666590+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465417096765. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:23.667606+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:22:30.338833+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:30.339096+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:30.339974+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465492827673. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:30.340819+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:30.437804+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:22:30.437953+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:22:30.438884+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465492827673. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:30.439826+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:22:30.537219+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:30.537375+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:30.538076+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465492827673. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:30.538930+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:30.636592+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:22:30.636737+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:22:30.637592+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465492827673. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:30.638393+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:22:30.735858+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:30.736034+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:30.737086+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465492827673. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:30.738342+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:30.835882+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:22:30.836104+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:22:30.837235+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465492827673. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:30.838368+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:22:30.935916+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:30.936067+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:30.937143+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465492827673. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:30.938473+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:31.035307+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:22:31.035494+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:22:31.036528+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465492827673. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:31.037624+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:22:31.135336+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:31.135537+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:31.136521+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465492827673. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:31.137656+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:31.235049+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:22:31.235224+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:22:31.236470+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465492827673. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:31.237586+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:22:37.906772+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:37.906980+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:37.908408+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465568264640. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:37.909514+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:38.006803+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:22:38.006950+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:22:38.007983+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465568264640. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:38.009093+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:22:38.106385+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:38.106566+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:38.107667+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465568264640. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:38.108724+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:38.206078+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:22:38.206260+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:22:38.207309+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465568264640. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:38.208697+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:22:38.305149+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:38.305313+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:38.306234+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465568264640. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:38.307235+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:38.404683+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:22:38.404835+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:22:38.405731+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465568264640. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:38.406858+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:22:38.504064+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:38.504276+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:38.505091+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465568264640. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:38.506062+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:38.603796+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:22:38.603947+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:22:38.604973+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465568264640. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:38.606180+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:22:38.702867+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:38.703012+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:38.703965+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465568264640. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:38.704962+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:38.802072+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:22:38.802250+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:22:38.803346+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465568264640. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:38.804235+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:22:45.677648+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:45.677841+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:45.678808+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465646129897. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:45.679719+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:45.777390+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:22:45.777559+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:22:45.778481+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465646129897. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:45.779489+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:22:45.876820+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:45.876998+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:45.877869+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465646129897. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:45.878945+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:45.976231+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:22:45.976397+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:22:45.977414+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465646129897. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:45.978573+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:22:46.075675+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:46.075836+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:46.076665+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465646129897. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:46.077532+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:46.175147+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:22:46.175305+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:22:46.176339+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465646129897. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:46.177579+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:22:46.274257+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:46.274416+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:46.275354+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465646129897. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:46.276402+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:46.374422+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:22:46.374647+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:22:46.375634+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465646129897. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:46.376672+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:22:46.479270+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:46.479497+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:46.480862+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465646129897. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:46.481762+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:46.578432+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:22:46.578637+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:22:46.579584+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465646129897. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:46.580479+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:22:53.258667+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:53.258841+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:53.259819+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465722004489. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:53.260682+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:53.358251+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:22:53.358411+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:22:53.359371+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465722004489. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:53.360528+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:22:53.457576+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:53.457730+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:53.458564+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465722004489. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:53.459625+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:53.556739+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:22:53.556891+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:22:53.557993+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465722004489. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:53.559276+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:22:53.656086+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:53.656275+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:53.657280+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465722004489. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:53.658682+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:53.755652+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:22:53.755802+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:22:53.756645+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465722004489. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:53.757774+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:22:53.855555+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:53.855703+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:53.856672+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465722004489. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:53.857863+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:53.954803+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:22:53.954951+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:22:53.956024+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465722004489. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:53.957187+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:22:54.058273+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:22:54.058460+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:22:54.059580+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465722004489. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:54.060840+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:22:54.158841+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:22:54.159051+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:22:54.160225+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465722004489. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:22:54.161378+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:23:13.476254+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:13.476543+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:13.477625+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465923127005. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:13.478808+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:13.575307+08:00 0 Plugin group_replication reported: ' set fd:52 connected'
2023-12-05T11:23:13.575495+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:52'
2023-12-05T11:23:13.576278+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465923127005. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:13.577100+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:52 in close_connection'
2023-12-05T11:23:13.674626+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:13.674760+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:13.675508+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465923127005. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:13.676540+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:13.773966+08:00 0 Plugin group_replication reported: ' set fd:52 connected'
2023-12-05T11:23:13.774103+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:52'
2023-12-05T11:23:13.775235+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465923127005. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:13.776343+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:52 in close_connection'
2023-12-05T11:23:13.873540+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:13.873718+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:13.874666+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465923127005. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:13.875763+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:13.972787+08:00 0 Plugin group_replication reported: ' set fd:52 connected'
2023-12-05T11:23:13.972931+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:52'
2023-12-05T11:23:13.973739+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465923127005. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:13.974681+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:52 in close_connection'
2023-12-05T11:23:14.072871+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:14.073025+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:14.074163+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465923127005. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:14.075217+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:14.172165+08:00 0 Plugin group_replication reported: ' set fd:52 connected'
2023-12-05T11:23:14.172362+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:52'
2023-12-05T11:23:14.173258+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465923127005. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:14.174365+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:52 in close_connection'
2023-12-05T11:23:14.271268+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:14.271412+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:14.272577+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465923127005. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:14.273789+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:14.370784+08:00 0 Plugin group_replication reported: ' set fd:52 connected'
2023-12-05T11:23:14.370943+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:52'
2023-12-05T11:23:14.371995+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465923127005. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:14.373158+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:52 in close_connection'
2023-12-05T11:23:20.943969+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:20.944245+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:20.945105+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465999651787. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:20.945865+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:21.043787+08:00 0 Plugin group_replication reported: ' set fd:52 connected'
2023-12-05T11:23:21.043963+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:52'
2023-12-05T11:23:21.044757+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465999651787. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:21.045775+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:52 in close_connection'
2023-12-05T11:23:21.143391+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:21.143594+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:21.144628+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465999651787. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:21.145579+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:21.243570+08:00 0 Plugin group_replication reported: ' set fd:52 connected'
2023-12-05T11:23:21.243754+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:52'
2023-12-05T11:23:21.244760+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465999651787. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:21.246017+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:52 in close_connection'
2023-12-05T11:23:21.342993+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:21.343212+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:21.344081+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465999651787. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:21.345054+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:21.442231+08:00 0 Plugin group_replication reported: ' set fd:52 connected'
2023-12-05T11:23:21.442462+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:52'
2023-12-05T11:23:21.443370+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465999651787. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:21.444398+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:52 in close_connection'
2023-12-05T11:23:21.541486+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:21.541641+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:21.542509+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465999651787. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:21.543504+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:21.641056+08:00 0 Plugin group_replication reported: ' set fd:48 connected'
2023-12-05T11:23:21.641232+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:48'
2023-12-05T11:23:21.642303+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465999651787. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:21.643483+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:48 in close_connection'
2023-12-05T11:23:21.740341+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:21.740542+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:21.741547+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465999651787. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:21.742603+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:21.839591+08:00 0 Plugin group_replication reported: ' set fd:48 connected'
2023-12-05T11:23:21.839734+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:48'
2023-12-05T11:23:21.840638+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017465999651787. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:21.841747+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:48 in close_connection'
2023-12-05T11:23:28.514003+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:28.514250+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:28.515228+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466074625778. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:28.516276+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:28.613288+08:00 0 Plugin group_replication reported: ' set fd:52 connected'
2023-12-05T11:23:28.613482+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:52'
2023-12-05T11:23:28.614430+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466074625778. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:28.615532+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:52 in close_connection'
2023-12-05T11:23:28.713284+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:28.713455+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:28.714357+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466074625778. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:28.715625+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:28.813343+08:00 0 Plugin group_replication reported: ' set fd:52 connected'
2023-12-05T11:23:28.813522+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:52'
2023-12-05T11:23:28.814425+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466074625778. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:28.815713+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:52 in close_connection'
2023-12-05T11:23:28.912809+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:28.912980+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:28.914160+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466074625778. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:28.915373+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:29.012703+08:00 0 Plugin group_replication reported: ' set fd:52 connected'
2023-12-05T11:23:29.012849+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:52'
2023-12-05T11:23:29.013577+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466074625778. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:29.014477+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:52 in close_connection'
2023-12-05T11:23:29.111824+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:29.112039+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:29.113227+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466074625778. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:29.114532+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:29.210925+08:00 0 Plugin group_replication reported: ' set fd:52 connected'
2023-12-05T11:23:29.211179+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:52'
2023-12-05T11:23:29.212072+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466074625778. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:29.213318+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:52 in close_connection'
2023-12-05T11:23:29.310294+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:29.310477+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:29.311615+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466074625778. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:29.312547+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:29.410089+08:00 0 Plugin group_replication reported: ' set fd:52 connected'
2023-12-05T11:23:29.410284+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:52'
2023-12-05T11:23:29.411230+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466074625778. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:29.412250+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:52 in close_connection'
2023-12-05T11:23:35.987001+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:35.987216+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:35.987966+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466150214388. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:35.988985+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:36.086237+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:36.086415+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:36.087528+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466150214388. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:36.088851+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:36.185607+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:36.185762+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:36.186696+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466150214388. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:36.187663+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:36.285704+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:36.285856+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:36.286753+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466150214388. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:36.287685+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:36.385167+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:36.385307+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:36.386283+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466150214388. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:36.387321+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:36.484224+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:36.484399+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:36.485498+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466150214388. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:36.486570+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:36.583618+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:36.583784+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:36.584850+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466150214388. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:36.585762+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:36.683385+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:36.683549+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:36.684628+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466150214388. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:36.685573+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:36.783497+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:36.783653+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:36.787678+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466150214388. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:36.788749+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:36.883315+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:36.883493+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:36.884386+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466150214388. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:36.885467+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:43.557848+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:43.558046+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:43.558857+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466225999935. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:43.559793+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:43.657203+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:43.657337+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:43.658151+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466225999935. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:43.659220+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:43.756248+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:43.756402+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:43.757345+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466225999935. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:43.758518+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:43.855731+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:43.855887+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:43.856913+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466225999935. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:43.858065+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:43.955423+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:43.955656+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:43.956577+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466225999935. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:43.957523+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:44.058030+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:44.058246+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:44.059199+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466225999935. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:44.060239+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:44.157766+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:44.157947+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:44.158743+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466225999935. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:44.160055+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:44.256846+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:44.257001+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:44.257954+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466225999935. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:44.258929+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:44.355986+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:44.356198+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:44.356948+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466225999935. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:44.357827+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:44.455211+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:44.455387+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:44.456461+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466225999935. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:44.457606+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:51.129890+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:51.130143+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:51.131053+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466300766585. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:51.132232+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:51.229793+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:51.229979+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:51.231073+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466300766585. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:51.232506+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:51.328968+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:51.329217+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:51.330170+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466300766585. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:51.331255+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:51.428325+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:51.428517+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:51.429591+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466300766585. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:51.430831+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:51.527822+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:51.527987+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:51.529013+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466300766585. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:51.529961+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:51.627545+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:51.627714+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:51.628821+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466300766585. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:51.629966+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:51.726832+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:51.726990+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:51.728146+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466300766585. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:51.729265+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:51.826653+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:51.826832+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:51.827914+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466300766585. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:51.829036+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:51.926295+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:51.926493+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:51.927544+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466300766585. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:51.928720+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:52.025407+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:52.025631+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:52.026583+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466300766585. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:52.027590+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:59.096492+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:59.096684+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:59.097894+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466376926735. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:59.098950+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:59.195911+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:59.196044+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:59.196980+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466376926735. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:59.198233+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:59.295642+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:59.295815+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:59.296636+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466376926735. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:59.297751+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:59.395590+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:59.395772+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:59.396741+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466376926735. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:59.397836+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:59.494968+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:59.495177+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:59.496024+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466376926735. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:59.497066+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:59.594920+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:59.595067+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:59.596279+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466376926735. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:59.597562+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:59.694772+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:59.694904+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:59.695801+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466376926735. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:59.696749+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:59.793829+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:59.793964+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:59.794916+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466376926735. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:59.795868+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:23:59.892993+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:23:59.893179+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:23:59.894144+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466376926735. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:59.895083+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:23:59.992737+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T11:23:59.992917+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T11:23:59.993937+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466376926735. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:23:59.994746+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T11:24:06.670031+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:24:06.670278+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:24:06.671044+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466457198100. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:24:06.672191+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:24:06.769731+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:24:06.769905+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:24:06.770763+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466457198100. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:24:06.771702+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:24:06.869599+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:24:06.869800+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:24:06.870859+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466457198100. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:24:06.872009+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:24:06.969421+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:24:06.969595+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:24:06.970520+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466457198100. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:24:06.971846+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:24:07.068623+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:24:07.068767+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:24:07.069692+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466457198100. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:24:07.070646+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:24:07.167971+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:24:07.168162+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:24:07.169164+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466457198100. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:24:07.170410+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:24:07.267635+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:24:07.267791+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:24:07.268778+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466457198100. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:24:07.270100+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:24:07.367638+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:24:07.367792+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:24:07.368902+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466457198100. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:24:07.369964+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
2023-12-05T11:24:07.467723+08:00 0 Plugin group_replication reported: ' set fd:43 connected'
2023-12-05T11:24:07.467882+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:43'
2023-12-05T11:24:07.468775+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466457198100. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:24:07.469730+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:43 in close_connection'
2023-12-05T11:24:07.567859+08:00 0 Plugin group_replication reported: ' set fd:64 connected'
2023-12-05T11:24:07.568005+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:64'
2023-12-05T11:24:07.569068+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017466457198100. Please stop the old node or wait for it to leave the group.'
2023-12-05T11:24:07.570039+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:64 in close_connection'
3. 188执行show engine innodb status\G
(Tue Dec5 11:25:31 2023)[(none)]>show engine innodb status\G
*************************** 1. row ***************************
Type: InnoDB
Name:
Status:
=====================================
2023-12-05 11:25:33 139837876147968 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 7 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 146634 srv_active, 0 srv_shutdown, 1047831 srv_idle
srv_master_thread log flush and writes: 0
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 81876
OS WAIT ARRAY INFO: signal count 207328
RW-shared spins 0, rounds 0, OS waits 0
RW-excl spins 0, rounds 0, OS waits 0
RW-sx spins 0, rounds 0, OS waits 0
Spin rounds per wait: 0.00 RW-shared, 0.00 RW-excl, 0.00 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 41735491
Purge done for trx's n:o < 41735490 undo n:o < 0 state: running but idle
History list length 0
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 421318168155752, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168156600, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168159144, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168163384, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168162536, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168161688, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168160840, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168159992, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168157448, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168154904, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168154056, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
I/O thread 10 state: waiting for completed aio requests (write thread)
I/O thread 11 state: waiting for completed aio requests (write thread)
I/O thread 12 state: waiting for completed aio requests (write thread)
I/O thread 13 state: waiting for completed aio requests (write thread)
I/O thread 14 state: waiting for completed aio requests (write thread)
I/O thread 15 state: waiting for completed aio requests (write thread)
Pending normal aio reads: , aio writes: ,
ibuf aio reads:, log i/o's:
Pending flushes (fsync) log: 0; buffer pool: 18
2277 OS file reads, 2926538 OS file writes, 885733 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.41 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 276707, node heap has 0 buffer(s)
Hash table size 276707, node heap has 0 buffer(s)
Hash table size 276707, node heap has 0 buffer(s)
Hash table size 276707, node heap has 0 buffer(s)
Hash table size 276707, node heap has 0 buffer(s)
Hash table size 276707, node heap has 0 buffer(s)
Hash table size 276707, node heap has 0 buffer(s)
Hash table size 276707, node heap has 0 buffer(s)
0.00 hash searches/s, 3.00 non-hash searches/s
---
LOG
---
Log sequence number 25345870386
Log buffer assigned up to 25345870386
Log buffer completed up to 25345870386
Log written up to 25345870386
Log flushed up to 25345870386
Added dirty pages up to 25345870386
Pages flushed up to 25345870386
Last checkpoint at 25345870386
Log minimum file id is 119
Log maximum file id is 125
Modified age no less than 25345870386
Checkpoint age 0
Max checkpoint age 5422668288
2262305 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 0
Dictionary memory allocated 733538
Buffer pool size 65528
Buffer pool size, bytes 1073610752
Free buffers 32000
Database pages 33528
Old database pages 12216
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 36127, not young 38986
0.54 youngs/s, 0.00 non-youngs/s
Pages read 1536, created 229813, written 545499
0.00 reads/s, 0.27 creates/s, 0.41 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 36 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 33528, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 45, not young 129
0.00 youngs/s, 0.00 non-youngs/s
Pages read 193, created 5618, written 7926
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
---BUFFER POOL 1
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1019, not young 230
0.00 youngs/s, 0.00 non-youngs/s
Pages read 165, created 6556, written 9143
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
---BUFFER POOL 2
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 113, not young 345
0.00 youngs/s, 0.00 non-youngs/s
Pages read 196, created 6877, written 8871
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
---BUFFER POOL 3
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 7349, not young 14868
0.54 youngs/s, 0.00 non-youngs/s
Pages read 289, created 52332, written 132747
0.00 reads/s, 0.27 creates/s, 0.41 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 57 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
---BUFFER POOL 4
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 7683, not young 177
0.00 youngs/s, 0.00 non-youngs/s
Pages read 93, created 52549, written 130217
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
---BUFFER POOL 5
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 9584, not young 1297
0.00 youngs/s, 0.00 non-youngs/s
Pages read 27, created 48200, written 117488
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
---BUFFER POOL 6
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 10050, not young 7780
0.00 youngs/s, 0.00 non-youngs/s
Pages read 139, created 52121, written 128902
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
---BUFFER POOL 7
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 284, not young 14160
0.00 youngs/s, 0.00 non-youngs/s
Pages read 434, created 5560, written 10205
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
0 RW transactions active inside InnoDB
Process ID=1062, Main thread ID=139841544886016 , state=sleeping
Number of rows inserted 7347575, updated 115920, deleted 0, read 2372983
1.57 inserts/s, 0.00 updates/s, 0.00 deletes/s, 1.57 reads/s
Number of system rows inserted 9229, updated 2019, deleted 8977, read 116739649
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 2.86 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
1 row in set (0.00 sec)
(Tue Dec5 11:25:33 2023)[(none)]>
4、mgr状态
188执行
(Tue Dec5 11:25:33 2023)[(none)]>select * from performance_schema.replication_group_members;SELECT * FROM performance_schema.replication_group_member_stats\G;show master status\G;SELECT MEMBER_ID AS id, COUNT_TRANSACTIONS_IN_QUEUE AS trx_tobe_certified, COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE AS relaylog_tobe_applied, COUNT_TRANSACTIONS_CHECKED AS trx_chkd, COUNT_TRANSACTIONS_REMOTE_APPLIED AS trx_done, COUNT_TRANSACTIONS_LOCAL_PROPOSED AS proposed FROM performance_schema.replication_group_member_stats;SELECT RECEIVED_TRANSACTION_SET FROM performance_schema.replication_connection_status WHEREchannel_name = 'group_replication_applier' UNION ALL SELECT variable_value FROM performance_schema.global_variables WHEREvariable_name = 'gtid_executed'\G;select now();SELECT MEMBER_ID,COUNT_TRANSACTIONS_ROWS_VALIDATING FROM performance_schema.replication_group_member_stats;
+---------------------------+--------------------------------------+---------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+---------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | a55de711-812e-11ee-b3fc-5254005b7a28 | 192.168.6.190 | 3308 | UNREACHABLE| SECONDARY | 8.0.32 | XCom |
| group_replication_applier | a5609358-812e-11ee-abe4-525400b01dcf | 192.168.6.188 | 3308 | ONLINE | PRIMARY | 8.0.32 | XCom |
| group_replication_applier | a5623edf-812e-11ee-92a9-525400980c24 | 192.168.6.189 | 3308 | ONLINE | SECONDARY | 8.0.32 | XCom |
+---------------------------+--------------------------------------+---------------+-------------+--------------+-------------+----------------+----------------------------+
3 rows in set (0.00 sec)
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 17005521402960008:4
MEMBER_ID: a55de711-812e-11ee-b3fc-5254005b7a28
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 115994
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19543900
LAST_CONFLICT_FREE_TRANSACTION: f774dd39-814f-11ee-aae7-525400b01dcf:19543900
COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE: 0
COUNT_TRANSACTIONS_REMOTE_APPLIED: 116005
COUNT_TRANSACTIONS_LOCAL_PROPOSED: 0
COUNT_TRANSACTIONS_LOCAL_ROLLBACK: 0
*************************** 2. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 17005521402960008:4
MEMBER_ID: a5609358-812e-11ee-abe4-525400b01dcf
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 116195
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19543900
LAST_CONFLICT_FREE_TRANSACTION: f774dd39-814f-11ee-aae7-525400b01dcf:19544101
COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE: 0
COUNT_TRANSACTIONS_REMOTE_APPLIED: 4
COUNT_TRANSACTIONS_LOCAL_PROPOSED: 116195
COUNT_TRANSACTIONS_LOCAL_ROLLBACK: 0
*************************** 3. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 17005521402960008:4
MEMBER_ID: a5623edf-812e-11ee-92a9-525400980c24
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 269
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19543900
LAST_CONFLICT_FREE_TRANSACTION: f774dd39-814f-11ee-aae7-525400b01dcf:19544101
COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE: 0
COUNT_TRANSACTIONS_REMOTE_APPLIED: 269
COUNT_TRANSACTIONS_LOCAL_PROPOSED: 0
COUNT_TRANSACTIONS_LOCAL_ROLLBACK: 0
3 rows in set (0.00 sec)
ERROR:
No query specified
*************************** 1. row ***************************
File: binlog.000692
Position: 246047058
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19544101
1 row in set (0.00 sec)
ERROR:
No query specified
+--------------------------------------+--------------------+-----------------------+----------+----------+----------+
| id | trx_tobe_certified | relaylog_tobe_applied | trx_chkd | trx_done | proposed |
+--------------------------------------+--------------------+-----------------------+----------+----------+----------+
| a55de711-812e-11ee-b3fc-5254005b7a28 | 0 | 0 | 115994 | 116005 | 0 |
| a5609358-812e-11ee-abe4-525400b01dcf | 0 | 0 | 116195 | 4 | 116195 |
| a5623edf-812e-11ee-92a9-525400980c24 | 0 | 0 | 269 | 269 | 0 |
+--------------------------------------+--------------------+-----------------------+----------+----------+----------+
3 rows in set (0.00 sec)
*************************** 1. row ***************************
RECEIVED_TRANSACTION_SET: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19543900
*************************** 2. row ***************************
RECEIVED_TRANSACTION_SET: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19544101
2 rows in set (0.00 sec)
ERROR:
No query specified
+---------------------+
| now() |
+---------------------+
| 2023-12-05 11:27:05 |
+---------------------+
1 row in set (0.00 sec)
+--------------------------------------+------------------------------------+
| MEMBER_ID | COUNT_TRANSACTIONS_ROWS_VALIDATING |
+--------------------------------------+------------------------------------+
| a55de711-812e-11ee-b3fc-5254005b7a28 | 0 |
| a5609358-812e-11ee-abe4-525400b01dcf | 0 |
| a5623edf-812e-11ee-92a9-525400980c24 | 0 |
+--------------------------------------+------------------------------------+
3 rows in set (0.00 sec)
(Tue Dec5 11:27:05 2023)[(none)]>补充188执行的mysqlsh检测结果：
MySQL192.168.6.188:3308 sslPy > c=dba.get_cluster()
MySQL192.168.6.188:3308 sslPy > c.status()
{
"clusterName": "test_mgr01",
"defaultReplicaSet": {
"name": "default",
"primary": "192.168.6.188:3308",
"ssl": "REQUIRED",
"status": "OK_NO_TOLERANCE",
"statusText": "Cluster is NOT tolerant to any failures. 1 member is not active.",
"topology": {
"192.168.6.188:3308": {
"address": "192.168.6.188:3308",
"memberRole": "PRIMARY",
"mode": "R/W",
"readReplicas": {},
"replicationLag": null,
"role": "HA",
"status": "ONLINE",
"version": "8.0.32"
},
"192.168.6.189:3308": {
"address": "192.168.6.189:3308",
"memberRole": "SECONDARY",
"mode": "R/O",
"readReplicas": {},
"replicationLag": null,
"role": "HA",
"status": "ONLINE",
"version": "8.0.32"
},
"192.168.6.190:3308": {
"address": "192.168.6.190:3308",
"instanceErrors": [
"NOTE: group_replication is stopped."
],
"memberRole": "SECONDARY",
"memberState": "OFFLINE",
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "UNREACHABLE",
"version": "8.0.32"
}
},
"topologyMode": "Single-Primary"
},
"groupInformationSourceMember": "192.168.6.188:3308"
}
MySQL192.168.6.188:3308 sslPy >
189执行
(Tue Dec5 10:30:34 2023)>select * from performance_schema.replication_group_members;SELECT * FROM performance_schema.replication_group_member_stats\G;show master status\G;SELECT MEMBER_ID AS id, COUNT_TRANSACTIONS_IN_QUEUE AS trx_tobe_certified, COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE AS relaylog_tobe_applied, COUNT_TRANSACTIONS_CHECKED AS trx_chkd, COUNT_TRANSACTIONS_REMOTE_APPLIED AS trx_done, COUNT_TRANSACTIONS_LOCAL_PROPOSED AS proposed FROM performance_schema.replication_group_member_stats;SELECT RECEIVED_TRANSACTION_SET FROM performance_schema.replication_connection_status WHEREchannel_name = 'group_replication_applier' UNION ALL SELECT variable_value FROM performance_schema.global_variables WHEREvariable_name = 'gtid_executed'\G;select now();SELECT MEMBER_ID,COUNT_TRANSACTIONS_ROWS_VALIDATING FROM performance_schema.replication_group_member_stats;
ERROR 4031 (HY000): The client was disconnected by the server because of inactivity. See wait_timeout and interactive_timeout for configuring this behavior.
No connection. Trying to reconnect...
Connection id: 38
Current database: tpcc10
+---------------------------+--------------------------------------+---------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+---------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | a55de711-812e-11ee-b3fc-5254005b7a28 | 192.168.6.190 | 3308 | UNREACHABLE| SECONDARY | 8.0.32 | XCom |
| group_replication_applier | a5609358-812e-11ee-abe4-525400b01dcf | 192.168.6.188 | 3308 | ONLINE | PRIMARY | 8.0.32 | XCom |
| group_replication_applier | a5623edf-812e-11ee-92a9-525400980c24 | 192.168.6.189 | 3308 | ONLINE | SECONDARY | 8.0.32 | XCom |
+---------------------------+--------------------------------------+---------------+-------------+--------------+-------------+----------------+----------------------------+
3 rows in set (0.00 sec)
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 17005521402960008:4
MEMBER_ID: a55de711-812e-11ee-b3fc-5254005b7a28
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 115994
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19543900
LAST_CONFLICT_FREE_TRANSACTION: f774dd39-814f-11ee-aae7-525400b01dcf:19543900
COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE: 0
COUNT_TRANSACTIONS_REMOTE_APPLIED: 116005
COUNT_TRANSACTIONS_LOCAL_PROPOSED: 0
COUNT_TRANSACTIONS_LOCAL_ROLLBACK: 0
*************************** 2. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 17005521402960008:4
MEMBER_ID: a5609358-812e-11ee-abe4-525400b01dcf
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 116195
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19543900
LAST_CONFLICT_FREE_TRANSACTION: f774dd39-814f-11ee-aae7-525400b01dcf:19544101
COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE: 0
COUNT_TRANSACTIONS_REMOTE_APPLIED: 4
COUNT_TRANSACTIONS_LOCAL_PROPOSED: 116195
COUNT_TRANSACTIONS_LOCAL_ROLLBACK: 0
*************************** 3. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 17005521402960008:4
MEMBER_ID: a5623edf-812e-11ee-92a9-525400980c24
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 269
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19543900
LAST_CONFLICT_FREE_TRANSACTION: f774dd39-814f-11ee-aae7-525400b01dcf:19544101
COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE: 0
COUNT_TRANSACTIONS_REMOTE_APPLIED: 269
COUNT_TRANSACTIONS_LOCAL_PROPOSED: 0
COUNT_TRANSACTIONS_LOCAL_ROLLBACK: 0
3 rows in set (0.00 sec)
ERROR:
No query specified
*************************** 1. row ***************************
File: binlog.000026
Position: 151728036
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19544101
1 row in set (0.01 sec)
ERROR:
No query specified
+--------------------------------------+--------------------+-----------------------+----------+----------+----------+
| id | trx_tobe_certified | relaylog_tobe_applied | trx_chkd | trx_done | proposed |
+--------------------------------------+--------------------+-----------------------+----------+----------+----------+
| a55de711-812e-11ee-b3fc-5254005b7a28 | 0 | 0 | 115994 | 116005 | 0 |
| a5609358-812e-11ee-abe4-525400b01dcf | 0 | 0 | 116195 | 4 | 116195 |
| a5623edf-812e-11ee-92a9-525400980c24 | 0 | 0 | 269 | 269 | 0 |
+--------------------------------------+--------------------+-----------------------+----------+----------+----------+
3 rows in set (0.00 sec)
*************************** 1. row ***************************
RECEIVED_TRANSACTION_SET: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19544101
*************************** 2. row ***************************
RECEIVED_TRANSACTION_SET: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19544101
2 rows in set (0.01 sec)
ERROR:
No query specified
+---------------------+
| now() |
+---------------------+
| 2023-12-05 11:27:45 |
+---------------------+
1 row in set (0.00 sec)
+--------------------------------------+------------------------------------+
| MEMBER_ID | COUNT_TRANSACTIONS_ROWS_VALIDATING |
+--------------------------------------+------------------------------------+
| a55de711-812e-11ee-b3fc-5254005b7a28 | 0 |
| a5609358-812e-11ee-abe4-525400b01dcf | 0 |
| a5623edf-812e-11ee-92a9-525400980c24 | 0 |
+--------------------------------------+------------------------------------+
3 rows in set (0.00 sec)
(Tue Dec5 11:27:45 2023)>
running_db 发表于 2023-12-5 11:27
1. stop group_replication;select sleep(1);start group_replication;操作了6次
2. 这6次中，188master日 ...
注意我说的是“1. 暂停190节点反复重试加入mgr。”，也就是停止尝试加入mgr这个动作，不是让你反复重试加入mgr。 yejr 发表于 2023-12-5 12:51
注意我说的是“1. 暂停190节点反复重试加入mgr。”，也就是停止尝试加入mgr这个动作，不是让你反复重试加 ...
1. 停止后，master日志没有任何变化，依旧是上次拒绝190加入的日志。到是189有新日志，两条。
188：
# tail -f error.log
2023-12-05T12:11:47.695006+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017495059959455. Please stop the old node or wait for it to leave the group.'
2023-12-05T12:11:47.696147+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:63 in close_connection'
2023-12-05T12:11:47.793762+08:00 0 Plugin group_replication reported: ' set fd:62 connected'
2023-12-05T12:11:47.793916+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:62'
2023-12-05T12:11:47.794826+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017495059959455. Please stop the old node or wait for it to leave the group.'
2023-12-05T12:11:47.796004+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:62 in close_connection'
2023-12-05T12:11:47.893200+08:00 0 Plugin group_replication reported: ' set fd:63 connected'
2023-12-05T12:11:47.893350+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:63'
2023-12-05T12:11:47.894470+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017495059959455. Please stop the old node or wait for it to leave the group.'
2023-12-05T12:11:47.895709+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:63 in close_connection'
189：
# tail -f /data/greatsql/8.0.32/log/error.log
2023-12-05T12:11:47.799393+08:00 0 Plugin group_replication reported: ' set fd:42 connected'
2023-12-05T12:11:47.799614+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:42'
2023-12-05T12:11:47.800686+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017495059959455. Please stop the old node or wait for it to leave the group.'
2023-12-05T12:11:47.801667+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:42 in close_connection'
2023-12-05T12:11:47.899419+08:00 0 Plugin group_replication reported: ' set fd:45 connected'
2023-12-05T12:11:47.899694+08:00 0 Plugin group_replication reported: ' buffered_read_msg sets CON_PROTO for fd:45'
2023-12-05T12:11:47.900776+08:00 0 Plugin group_replication reported: ' Old incarnation found while trying to add node 192.168.6.190:33081 17017495059959455. Please stop the old node or wait for it to leave the group.'
2023-12-05T12:11:47.901968+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:45 in close_connection'
2023-12-05T13:44:41.027189+08:00 0 Plugin group_replication reported: ' set fd:35 connected'
2023-12-05T13:44:41.027717+08:00 0 Plugin group_replication reported: ' set CON_NULL for fd:35 in close_connection'
2. 188执行show engine innodb status\G
(Tue Dec5 13:48:52 2023)[(none)]>show engine innodb status\G
*************************** 1. row ***************************
Type: InnoDB
Name:
Status:
=====================================
2023-12-05 13:48:53 139837876680448 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 7 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 147638 srv_active, 0 srv_shutdown, 1055424 srv_idle
srv_master_thread log flush and writes: 0
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 81940
OS WAIT ARRAY INFO: signal count 207394
RW-shared spins 0, rounds 0, OS waits 0
RW-excl spins 0, rounds 0, OS waits 0
RW-sx spins 0, rounds 0, OS waits 0
Spin rounds per wait: 0.00 RW-shared, 0.00 RW-excl, 0.00 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 41735491
Purge done for trx's n:o < 41735490 undo n:o < 0 state: running but idle
History list length 0
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 421318168159144, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168155752, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168158296, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168156600, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168163384, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168162536, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168161688, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168160840, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168159992, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168157448, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168154904, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421318168154056, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
I/O thread 10 state: waiting for completed aio requests (write thread)
I/O thread 11 state: waiting for completed aio requests (write thread)
I/O thread 12 state: waiting for completed aio requests (write thread)
I/O thread 13 state: waiting for completed aio requests (write thread)
I/O thread 14 state: waiting for completed aio requests (write thread)
I/O thread 15 state: waiting for completed aio requests (write thread)
Pending normal aio reads: , aio writes: ,
ibuf aio reads:, log i/o's:
Pending flushes (fsync) log: 0; buffer pool: 18
2300 OS file reads, 2930377 OS file writes, 886147 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 1.11 writes/s, 0.14 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 276707, node heap has 0 buffer(s)
Hash table size 276707, node heap has 0 buffer(s)
Hash table size 276707, node heap has 0 buffer(s)
Hash table size 276707, node heap has 0 buffer(s)
Hash table size 276707, node heap has 0 buffer(s)
Hash table size 276707, node heap has 0 buffer(s)
Hash table size 276707, node heap has 0 buffer(s)
Hash table size 276707, node heap has 0 buffer(s)
0.00 hash searches/s, 782.75 non-hash searches/s
---
LOG
---
Log sequence number 25345870386
Log buffer assigned up to 25345870386
Log buffer completed up to 25345870386
Log written up to 25345870386
Log flushed up to 25345870386
Added dirty pages up to 25345870386
Pages flushed up to 25345870386
Last checkpoint at 25345870386
Log minimum file id is 119
Log maximum file id is 125
Modified age no less than 25345870386
Checkpoint age 0
Max checkpoint age 5422668288
2262305 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 0
Dictionary memory allocated 733538
Buffer pool size 65528
Buffer pool size, bytes 1073610752
Free buffers 32000
Database pages 33528
Old database pages 12216
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 40135, not young 39298
1.11 youngs/s, 0.00 non-youngs/s
Pages read 1559, created 231057, written 548924
0.00 reads/s, 0.42 creates/s, 0.97 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 33528, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 45, not young 143
0.00 youngs/s, 0.00 non-youngs/s
Pages read 198, created 5618, written 7926
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
---BUFFER POOL 1
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1019, not young 230
0.00 youngs/s, 0.00 non-youngs/s
Pages read 165, created 6556, written 9143
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
---BUFFER POOL 2
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 113, not young 345
0.00 youngs/s, 0.00 non-youngs/s
Pages read 196, created 6877, written 8871
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
---BUFFER POOL 3
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 8586, not young 14895
1.11 youngs/s, 0.00 non-youngs/s
Pages read 289, created 52707, written 133801
0.00 reads/s, 0.28 creates/s, 0.83 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 8 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
---BUFFER POOL 4
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 8031, not young 177
0.00 youngs/s, 0.00 non-youngs/s
Pages read 93, created 52642, written 130509
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
---BUFFER POOL 5
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 11017, not young 1351
0.00 youngs/s, 0.00 non-youngs/s
Pages read 27, created 48644, written 118717
0.00 reads/s, 0.14 creates/s, 0.14 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
---BUFFER POOL 6
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 11038, not young 7859
0.00 youngs/s, 0.00 non-youngs/s
Pages read 143, created 52453, written 129752
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
---BUFFER POOL 7
Buffer pool size 8191
Buffer pool size, bytes 134201344
Free buffers 4000
Database pages 4191
Old database pages 1527
Modified db pages0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 286, not young 14298
0.00 youngs/s, 0.00 non-youngs/s
Pages read 448, created 5560, written 10205
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 4191, unzip_LRU len: 0
I/O sum:cur, unzip sum:cur
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
0 RW transactions active inside InnoDB
Process ID=1062, Main thread ID=139841544886016 , state=sleeping
Number of rows inserted 7363728, updated 115920, deleted 0, read 2389239
8.86 inserts/s, 0.00 updates/s, 0.00 deletes/s, 8.86 reads/s
Number of system rows inserted 9229, updated 2019, deleted 8977, read 117606390
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 828.31 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
1 row in set (0.00 sec)
(Tue Dec5 13:48:53 2023)[(none)]>
3. mgr状态检测
188：
(Tue Dec5 13:49:31 2023)[(none)]>select * from performance_schema.replication_group_members;SELECT * FROM performance_schema.replication_group_member_stats\G;show master status\G;SELECT MEMBER_ID AS id, COUNT_TRANSACTIONS_IN_QUEUE AS trx_tobe_certified, COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE AS relaylog_tobe_applied, COUNT_TRANSACTIONS_CHECKED AS trx_chkd, COUNT_TRANSACTIONS_REMOTE_APPLIED AS trx_done, COUNT_TRANSACTIONS_LOCAL_PROPOSED AS proposed FROM performance_schema.replication_group_member_stats;SELECT RECEIVED_TRANSACTION_SET FROM performance_schema.replication_connection_status WHEREchannel_name = 'group_replication_applier' UNION ALL SELECT variable_value FROM performance_schema.global_variables WHEREvariable_name = 'gtid_executed'\G;select now();SELECT MEMBER_ID,COUNT_TRANSACTIONS_ROWS_VALIDATING FROM performance_schema.replication_group_member_stats;
+---------------------------+--------------------------------------+---------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+---------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | a55de711-812e-11ee-b3fc-5254005b7a28 | 192.168.6.190 | 3308 | UNREACHABLE| SECONDARY | 8.0.32 | XCom |
| group_replication_applier | a5609358-812e-11ee-abe4-525400b01dcf | 192.168.6.188 | 3308 | ONLINE | PRIMARY | 8.0.32 | XCom |
| group_replication_applier | a5623edf-812e-11ee-92a9-525400980c24 | 192.168.6.189 | 3308 | ONLINE | SECONDARY | 8.0.32 | XCom |
+---------------------------+--------------------------------------+---------------+-------------+--------------+-------------+----------------+----------------------------+
3 rows in set (0.00 sec)
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 17005521402960008:4
MEMBER_ID: a55de711-812e-11ee-b3fc-5254005b7a28
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 115994
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19543900
LAST_CONFLICT_FREE_TRANSACTION: f774dd39-814f-11ee-aae7-525400b01dcf:19543900
COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE: 0
COUNT_TRANSACTIONS_REMOTE_APPLIED: 116005
COUNT_TRANSACTIONS_LOCAL_PROPOSED: 0
COUNT_TRANSACTIONS_LOCAL_ROLLBACK: 0
*************************** 2. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 17005521402960008:4
MEMBER_ID: a5609358-812e-11ee-abe4-525400b01dcf
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 116195
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19543900
LAST_CONFLICT_FREE_TRANSACTION: f774dd39-814f-11ee-aae7-525400b01dcf:19544101
COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE: 0
COUNT_TRANSACTIONS_REMOTE_APPLIED: 4
COUNT_TRANSACTIONS_LOCAL_PROPOSED: 116195
COUNT_TRANSACTIONS_LOCAL_ROLLBACK: 0
*************************** 3. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 17005521402960008:4
MEMBER_ID: a5623edf-812e-11ee-92a9-525400980c24
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 269
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19543900
LAST_CONFLICT_FREE_TRANSACTION: f774dd39-814f-11ee-aae7-525400b01dcf:19544101
COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE: 0
COUNT_TRANSACTIONS_REMOTE_APPLIED: 269
COUNT_TRANSACTIONS_LOCAL_PROPOSED: 0
COUNT_TRANSACTIONS_LOCAL_ROLLBACK: 0
3 rows in set (0.00 sec)
ERROR:
No query specified
*************************** 1. row ***************************
File: binlog.000692
Position: 246047058
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19544101
1 row in set (0.00 sec)
ERROR:
No query specified
+--------------------------------------+--------------------+-----------------------+----------+----------+----------+
| id | trx_tobe_certified | relaylog_tobe_applied | trx_chkd | trx_done | proposed |
+--------------------------------------+--------------------+-----------------------+----------+----------+----------+
| a55de711-812e-11ee-b3fc-5254005b7a28 | 0 | 0 | 115994 | 116005 | 0 |
| a5609358-812e-11ee-abe4-525400b01dcf | 0 | 0 | 116195 | 4 | 116195 |
| a5623edf-812e-11ee-92a9-525400980c24 | 0 | 0 | 269 | 269 | 0 |
+--------------------------------------+--------------------+-----------------------+----------+----------+----------+
3 rows in set (0.00 sec)
*************************** 1. row ***************************
RECEIVED_TRANSACTION_SET: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19543900
*************************** 2. row ***************************
RECEIVED_TRANSACTION_SET: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19544101
2 rows in set (0.01 sec)
ERROR:
No query specified
+---------------------+
| now() |
+---------------------+
| 2023-12-05 13:49:32 |
+---------------------+
1 row in set (0.00 sec)
+--------------------------------------+------------------------------------+
| MEMBER_ID | COUNT_TRANSACTIONS_ROWS_VALIDATING |
+--------------------------------------+------------------------------------+
| a55de711-812e-11ee-b3fc-5254005b7a28 | 0 |
| a5609358-812e-11ee-abe4-525400b01dcf | 0 |
| a5623edf-812e-11ee-92a9-525400980c24 | 0 |
+--------------------------------------+------------------------------------+
3 rows in set (0.00 sec)
(Tue Dec5 13:49:32 2023)[(none)]>
188使用mysqlsh检测：
MySQL192.168.6.188:3308 sslPy > c=dba.get_cluster()
MySQL192.168.6.188:3308 sslPy > c.status()
{
"clusterName": "test_mgr01",
"defaultReplicaSet": {
"name": "default",
"primary": "192.168.6.188:3308",
"ssl": "REQUIRED",
"status": "OK_NO_TOLERANCE",
"statusText": "Cluster is NOT tolerant to any failures. 1 member is not active.",
"topology": {
"192.168.6.188:3308": {
"address": "192.168.6.188:3308",
"memberRole": "PRIMARY",
"mode": "R/W",
"readReplicas": {},
"replicationLag": null,
"role": "HA",
"status": "ONLINE",
"version": "8.0.32"
},
"192.168.6.189:3308": {
"address": "192.168.6.189:3308",
"memberRole": "SECONDARY",
"mode": "R/O",
"readReplicas": {},
"replicationLag": null,
"role": "HA",
"status": "ONLINE",
"version": "8.0.32"
},
"192.168.6.190:3308": {
"address": "192.168.6.190:3308",
"memberRole": "SECONDARY",
"memberState": "(MISSING)",
"mode": "n/a",
"readReplicas": {},
"role": "HA",
"shellConnectError": "MySQL Error 2003 (HY000): Can't connect to MySQL server on '192.168.6.190:3308' (111)",
"status": "UNREACHABLE",
"version": "8.0.32"
}
},
"topologyMode": "Single-Primary"
},
"groupInformationSourceMember": "192.168.6.188:3308"
}
MySQL192.168.6.188:3308 sslPy >
189：
(Tue Dec5 13:51:08 2023)>select * from performance_schema.replication_group_members;SELECT * FROM performance_schema.replication_group_member_stats\G;show master status\G;SELECT MEMBER_ID AS id, COUNT_TRANSACTIONS_IN_QUEUE AS trx_tobe_certified, COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE AS relaylog_tobe_applied, COUNT_TRANSACTIONS_CHECKED AS trx_chkd, COUNT_TRANSACTIONS_REMOTE_APPLIED AS trx_done, COUNT_TRANSACTIONS_LOCAL_PROPOSED AS proposed FROM performance_schema.replication_group_member_stats;SELECT RECEIVED_TRANSACTION_SET FROM performance_schema.replication_connection_status WHEREchannel_name = 'group_replication_applier' UNION ALL SELECT variable_value FROM performance_schema.global_variables WHEREvariable_name = 'gtid_executed'\G;select now();SELECT MEMBER_ID,COUNT_TRANSACTIONS_ROWS_VALIDATING FROM performance_schema.replication_group_member_stats;
+---------------------------+--------------------------------------+---------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+---------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | a55de711-812e-11ee-b3fc-5254005b7a28 | 192.168.6.190 | 3308 | UNREACHABLE| SECONDARY | 8.0.32 | XCom |
| group_replication_applier | a5609358-812e-11ee-abe4-525400b01dcf | 192.168.6.188 | 3308 | ONLINE | PRIMARY | 8.0.32 | XCom |
| group_replication_applier | a5623edf-812e-11ee-92a9-525400980c24 | 192.168.6.189 | 3308 | ONLINE | SECONDARY | 8.0.32 | XCom |
+---------------------------+--------------------------------------+---------------+-------------+--------------+-------------+----------------+----------------------------+
3 rows in set (0.00 sec)
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 17005521402960008:4
MEMBER_ID: a55de711-812e-11ee-b3fc-5254005b7a28
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 115994
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19543900
LAST_CONFLICT_FREE_TRANSACTION: f774dd39-814f-11ee-aae7-525400b01dcf:19543900
COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE: 0
COUNT_TRANSACTIONS_REMOTE_APPLIED: 116005
COUNT_TRANSACTIONS_LOCAL_PROPOSED: 0
COUNT_TRANSACTIONS_LOCAL_ROLLBACK: 0
*************************** 2. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 17005521402960008:4
MEMBER_ID: a5609358-812e-11ee-abe4-525400b01dcf
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 116195
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19543900
LAST_CONFLICT_FREE_TRANSACTION: f774dd39-814f-11ee-aae7-525400b01dcf:19544101
COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE: 0
COUNT_TRANSACTIONS_REMOTE_APPLIED: 4
COUNT_TRANSACTIONS_LOCAL_PROPOSED: 116195
COUNT_TRANSACTIONS_LOCAL_ROLLBACK: 0
*************************** 3. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 17005521402960008:4
MEMBER_ID: a5623edf-812e-11ee-92a9-525400980c24
COUNT_TRANSACTIONS_IN_QUEUE: 0
COUNT_TRANSACTIONS_CHECKED: 269
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19543900
LAST_CONFLICT_FREE_TRANSACTION: f774dd39-814f-11ee-aae7-525400b01dcf:19544101
COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE: 0
COUNT_TRANSACTIONS_REMOTE_APPLIED: 269
COUNT_TRANSACTIONS_LOCAL_PROPOSED: 0
COUNT_TRANSACTIONS_LOCAL_ROLLBACK: 0
3 rows in set (0.00 sec)
ERROR:
No query specified
*************************** 1. row ***************************
File: binlog.000026
Position: 151728036
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19544101
1 row in set (0.00 sec)
ERROR:
No query specified
+--------------------------------------+--------------------+-----------------------+----------+----------+----------+
| id | trx_tobe_certified | relaylog_tobe_applied | trx_chkd | trx_done | proposed |
+--------------------------------------+--------------------+-----------------------+----------+----------+----------+
| a55de711-812e-11ee-b3fc-5254005b7a28 | 0 | 0 | 115994 | 116005 | 0 |
| a5609358-812e-11ee-abe4-525400b01dcf | 0 | 0 | 116195 | 4 | 116195 |
| a5623edf-812e-11ee-92a9-525400980c24 | 0 | 0 | 269 | 269 | 0 |
+--------------------------------------+--------------------+-----------------------+----------+----------+----------+
3 rows in set (0.00 sec)
*************************** 1. row ***************************
RECEIVED_TRANSACTION_SET: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19544101
*************************** 2. row ***************************
RECEIVED_TRANSACTION_SET: a5609358-812e-11ee-abe4-525400b01dcf:1,
f774dd39-814f-11ee-aae7-525400b01dcf:1-19544101
2 rows in set (0.01 sec)
ERROR:
No query specified
+---------------------+
| now() |
+---------------------+
| 2023-12-05 13:51:08 |
+---------------------+
1 row in set (0.00 sec)
+--------------------------------------+------------------------------------+
| MEMBER_ID | COUNT_TRANSACTIONS_ROWS_VALIDATING |
+--------------------------------------+------------------------------------+
| a55de711-812e-11ee-b3fc-5254005b7a28 | 0 |
| a5609358-812e-11ee-abe4-525400b01dcf | 0 |
| a5623edf-812e-11ee-92a9-525400980c24 | 0 |
+--------------------------------------+------------------------------------+
3 rows in set (0.00 sec)
(Tue Dec5 13:51:08 2023)>
running_db 发表于 2023-12-5 13:51
1. 停止后，master日志没有任何变化，依旧是上次拒绝190加入的日志。到是189有新日志，两条。
188：
经查，是因为190节点被OOM kill了，在这之前有大事务尚未结束，并且设置了 group_replication_unreachable_majority_timeout = 30（默认值为0），导致在MGR中，190节点还没被驱逐出去前，该实例已重新拉起并尝试再次加入MGR，所以存在旧视图冲突告警。
建议：
1. 故障节点再次加入MGR前，要先判断该节点已被正常驱逐出MGR。
2. 调整group_replication_unreachable_majority_timeout = 0（相应的my.cnf模板也已调整 https://gitee.com/GreatSQL/GreatSQL-Doc/blob/master/docs/my.cnf-example-greatsql-8.0.32-24）。
3. 采用MySQL Shell工具来管理MGR，也利用Shell负责处理将故障节点重新加回MGR的工作。
4. 尽量不要在MGR中执行大事务，遇到大事务要拆分成多个小事务执行。 yejr 发表于 2023-12-5 16:30
经查，是因为190节点被OOM kill了，在这之前有大事务尚未结束，并且设置了 group_replication_unreachabl ...
叶老师，请问当前状态如何恢复到正常呢，不能设置这个参数group_replication_force_members 。用mysqlsh的remove_instance？顺便问下，这个remove_instance底层调用的mysql本身那边什么操作，函数？sql？ running_db 发表于 2023-12-7 09:29
叶老师，请问当前状态如何恢复到正常呢，不能设置这个参数group_replication_force_members 。用mysqlsh ...
问题1：
可以试试几个方法（任选一种）：
1. 停掉189这个secondary节点。
2. 在188、189间做一次切换primary节点。
3. 用remove_instance()方法踢掉一个节点后再加入。
问题2：
mysql shell创建连接时，加上启用log参数，详见：https://greatsql.cn/docs/8032/user-manual/8-mgr/2-mgr-install-deploy.html#_4-%E4%BD%BF%E7%94%A8mysql-shell%E7%9A%84%E7%AA%8D%E9%97%A8
页:
[1]