mabai 发表于 2024-4-20 09:59:59

greatesql-8.0.32-25 压测的时候出现性能抖动

本帖最后由 mabai 于 2024-4-24 15:28 编辑

[ 898s ] thds: 50 tps: 14721.41 qps: 294379.29 (r/w/o: 206059.81/58878.66/29440.83) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 899s ] thds: 50 tps: 14801.96 qps: 296080.20 (r/w/o: 207278.43/59194.85/29606.92) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 900s ] thds: 50 tps: 14524.56 qps: 290521.30 (r/w/o: 203333.90/58138.27/29049.13) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 901s ] thds: 50 tps: 14852.87 qps: 297158.46 (r/w/o: 208018.22/59434.49/29705.74) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 902s ] thds: 50 tps: 14404.93 qps: 288246.69 (r/w/o: 201730.08/57706.73/28809.87) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 903s ] thds: 50 tps: 9370.41 qps: 187209.22 (r/w/o: 131185.76/37282.64/18740.82) lat (ms,95%): 9.56 err/s: 0.00 reconn/s: 0.00
[ 904s ] thds: 50 tps: 2177.06 qps: 43359.21 (r/w/o: 30232.84/8774.24/4352.12) lat (ms,95%): 6.79 err/s: 0.00 reconn/s: 0.00
[ 905s ] thds: 50 tps: 14917.85 qps: 298545.97 (r/w/o: 208995.87/59713.39/29836.70) lat (ms,95%): 3.96 err/s: 0.00 reconn/s: 0.00
[ 906s ] thds: 50 tps: 14464.73 qps: 289283.53 (r/w/o: 202528.17/57824.90/28930.45) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 907s ] thds: 50 tps: 14383.19 qps: 287646.71 (r/w/o: 201343.60/57536.74/28766.37) lat (ms,95%): 4.25 err/s: 0.00 reconn/s: 0.00
[ 908s ] thds: 50 tps: 14598.07 qps: 291963.49 (r/w/o: 204346.04/58421.30/29196.15) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 909s ] thds: 50 tps: 14743.70 qps: 294886.01 (r/w/o: 206423.81/58978.80/29483.40) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 910s ] thds: 50 tps: 14765.87 qps: 295203.45 (r/w/o: 206642.21/59027.49/29533.74) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 911s ] thds: 50 tps: 14652.31 qps: 293098.21 (r/w/o: 205186.35/58607.24/29304.62) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 912s ] thds: 50 tps: 14802.52 qps: 296112.36 (r/w/o: 207274.25/59231.07/29607.04) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 913s ] thds: 50 tps: 10693.45 qps: 213824.06 (r/w/o: 149697.35/42739.81/21386.91) lat (ms,95%): 4.41 err/s: 0.00 reconn/s: 0.00
[ 914s ] thds: 50 tps: 14731.29 qps: 294650.88 (r/w/o: 206222.11/58966.18/29462.59) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 915s ] thds: 50 tps: 14537.74 qps: 290752.83 (r/w/o: 203534.38/58142.97/29075.48) lat (ms,95%): 4.25 err/s: 0.00 reconn/s: 0.00
[ 916s ] thds: 50 tps: 14701.42 qps: 294000.31 (r/w/o: 205807.82/58789.66/29402.83) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 917s ] thds: 50 tps: 14729.38 qps: 294557.51 (r/w/o: 206190.25/58911.50/29455.75) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 918s ] thds: 50 tps: 14620.97 qps: 292487.39 (r/w/o: 204751.57/58491.88/29243.94) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 919s ] thds: 50 tps: 14771.42 qps: 295592.45 (r/w/o: 206880.92/59167.68/29543.85) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 920s ] thds: 50 tps: 14046.03 qps: 280772.49 (r/w/o: 196566.35/56114.10/28092.05) lat (ms,95%): 4.33 err/s: 0.00 reconn/s: 0.00
[ 921s ] thds: 50 tps: 14544.75 qps: 290973.06 (r/w/o: 203685.54/58199.01/29088.51) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 922s ] thds: 50 tps: 6237.73 qps: 124651.53 (r/w/o: 87266.17/24908.90/12476.45) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 923s ] thds: 50 tps: 14649.28 qps: 292871.66 (r/w/o: 204996.96/58577.13/29297.56) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 924s ] thds: 50 tps: 14627.25 qps: 292686.07 (r/w/o: 204874.55/58556.01/29255.51) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 925s ] thds: 50 tps: 14487.80 qps: 289743.99 (r/w/o: 202832.19/57936.20/28975.60) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 926s ] thds: 50 tps: 14789.01 qps: 295764.14 (r/w/o: 207024.10/59166.03/29574.01) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 927s ] thds: 50 tps: 14678.46 qps: 293550.17 (r/w/o: 205481.42/58707.83/29360.92) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 928s ] thds: 50 tps: 14573.18 qps: 291380.59 (r/w/o: 203967.52/58266.72/29146.36) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 929s ] thds: 50 tps: 14675.84 qps: 293422.69 (r/w/o: 205397.68/58677.34/29347.67) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 930s ] thds: 50 tps: 14372.35 qps: 287663.08 (r/w/o: 201377.96/57536.42/28748.71) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 931s ] thds: 50 tps: 14152.70 qps: 282993.03 (r/w/o: 198109.82/56577.81/28305.40) lat (ms,95%): 4.25 err/s: 0.00 reconn/s: 0.00
[ 932s ] thds: 50 tps: 14749.56 qps: 295028.16 (r/w/o: 206507.81/59021.23/29499.12) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 933s ] thds: 50 tps: 14681.77 qps: 293536.42 (r/w/o: 205438.80/58738.08/29359.54) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 934s ] thds: 50 tps: 14695.83 qps: 293866.51 (r/w/o: 205721.55/58749.31/29395.65) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 935s ] thds: 50 tps: 14764.97 qps: 295534.49 (r/w/o: 206873.65/59130.91/29529.94) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 936s ] thds: 50 tps: 14661.06 qps: 293150.11 (r/w/o: 205211.78/58616.22/29322.11) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 937s ] thds: 50 tps: 4229.83 qps: 84641.68 (r/w/o: 59325.67/16856.34/8459.67) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 938s ] thds: 50 tps: 10122.77 qps: 202321.38 (r/w/o: 141528.77/40547.08/20245.54) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 939s ] thds: 50 tps: 14660.81 qps: 293285.19 (r/w/o: 205305.33/58662.24/29317.62) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 940s ] thds: 50 tps: 14522.14 qps: 290431.88 (r/w/o: 203316.01/58067.58/29048.29) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 941s ] thds: 50 tps: 13405.74 qps: 268183.71 (r/w/o: 187738.30/53633.94/26811.47) lat (ms,95%): 4.25 err/s: 0.00 reconn/s: 0.00
[ 942s ] thds: 50 tps: 14665.92 qps: 293210.41 (r/w/o: 205265.89/58613.69/29330.84) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 943s ] thds: 50 tps: 14702.09 qps: 294144.81 (r/w/o: 205872.26/58867.37/29405.18) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 944s ] thds: 50 tps: 13074.78 qps: 261490.62 (r/w/o: 183132.93/52208.13/26149.56) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 945s ] thds: 50 tps: 10213.15 qps: 204284.08 (r/w/o: 142923.17/40934.60/20426.31) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 946s ] thds: 50 tps: 14610.51 qps: 292159.24 (r/w/o: 204509.17/58429.05/29221.02) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 947s ] thds: 50 tps: 14636.01 qps: 292677.13 (r/w/o: 204850.08/58555.03/29272.01) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 948s ] thds: 50 tps: 14826.44 qps: 296544.74 (r/w/o: 207607.12/59284.75/29652.87) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 949s ] thds: 50 tps: 14723.97 qps: 294446.30 (r/w/o: 206104.51/58893.86/29447.93) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 950s ] thds: 50 tps: 14683.84 qps: 293802.80 (r/w/o: 205658.76/58776.36/29367.68) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 951s ] thds: 50 tps: 14702.13 qps: 293878.64 (r/w/o: 205694.84/58781.53/29402.27) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 952s ] thds: 50 tps: 14376.82 qps: 287618.47 (r/w/o: 201375.52/57487.30/28755.65) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 953s ] thds: 50 tps: 12441.57 qps: 248682.49 (r/w/o: 174044.04/49755.30/24883.15) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 954s ] thds: 50 tps: 14929.71 qps: 298710.12 (r/w/o: 209096.88/59753.82/29859.41) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 955s ] thds: 50 tps: 14644.26 qps: 292724.19 (r/w/o: 204928.63/58507.04/29288.52) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 956s ] thds: 50 tps: 14671.32 qps: 293779.49 (r/w/o: 205592.52/58845.33/29341.64) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 957s ] thds: 50 tps: 14860.58 qps: 297059.56 (r/w/o: 207987.08/59352.33/29720.15) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 958s ] thds: 50 tps: 14653.61 qps: 292988.08 (r/w/o: 205067.45/58611.42/29309.21) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 959s ] thds: 50 tps: 14556.99 qps: 291221.88 (r/w/o: 203881.91/58225.98/29113.99) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 960s ] thds: 50 tps: 14595.77 qps: 291859.35 (r/w/o: 204294.74/58373.07/29191.53) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 961s ] thds: 50 tps: 14490.97 qps: 289804.33 (r/w/o: 202854.53/57967.87/28981.93) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 962s ] thds: 50 tps: 14641.92 qps: 293031.39 (r/w/o: 205117.87/58629.67/29283.84) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 963s ] thds: 50 tps: 14711.76 qps: 294151.24 (r/w/o: 205914.67/58813.04/29423.53) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 964s ] thds: 50 tps: 14848.00 qps: 296871.09 (r/w/o: 207820.06/59355.02/29696.01) lat (ms,95%): 3.96 err/s: 0.00 reconn/s: 0.00
[ 965s ] thds: 50 tps: 14618.19 qps: 292364.71 (r/w/o: 204666.59/58461.74/29236.37) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 966s ] thds: 50 tps: 14549.18 qps: 290933.53 (r/w/o: 203639.47/58195.71/29098.35) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 967s ] thds: 50 tps: 11580.88 qps: 231766.65 (r/w/o: 162316.33/46288.55/23161.77) lat (ms,95%): 4.33 err/s: 0.00 reconn/s: 0.00
[ 968s ] thds: 50 tps: 2730.45 qps: 54532.95 (r/w/o: 38103.22/10968.82/5460.90) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 969s ] thds: 50 tps: 14768.87 qps: 295342.50 (r/w/o: 206726.25/59081.50/29534.75) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 970s ] thds: 50 tps: 14682.16 qps: 293808.29 (r/w/o: 205656.30/58784.66/29367.33) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 971s ] thds: 50 tps: 14583.57 qps: 291527.42 (r/w/o: 204078.99/58285.29/29163.14) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 972s ] thds: 50 tps: 13586.43 qps: 272000.70 (r/w/o: 190356.08/54467.75/27176.87) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 973s ] thds: 50 tps: 14571.92 qps: 291058.46 (r/w/o: 203770.92/58147.69/29139.85) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 974s ] thds: 50 tps: 14589.92 qps: 291843.45 (r/w/o: 204318.91/58342.69/29181.85) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 975s ] thds: 50 tps: 14668.26 qps: 293513.21 (r/w/o: 205440.64/58734.05/29338.52) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 976s ] thds: 50 tps: 13880.92 qps: 277608.34 (r/w/o: 194423.83/55422.68/27761.83) lat (ms,95%): 4.33 err/s: 0.00 reconn/s: 0.00
[ 977s ] thds: 50 tps: 430.99 qps: 8575.79 (r/w/o: 5898.85/1814.96/861.98) lat (ms,95%): 1013.60 err/s: 0.00 reconn/s: 0.00
[ 978s ] thds: 50 tps: 14174.95 qps: 283486.97 (r/w/o: 198442.28/56696.79/28347.90) lat (ms,95%): 4.33 err/s: 0.00 reconn/s: 0.00
[ 979s ] thds: 50 tps: 14623.61 qps: 292439.26 (r/w/o: 204736.58/58457.45/29245.23) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 980s ] thds: 50 tps: 14693.13 qps: 293890.52 (r/w/o: 205717.76/58785.50/29387.25) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 981s ] thds: 50 tps: 14709.88 qps: 294232.63 (r/w/o: 205937.35/58872.52/29422.76) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 982s ] thds: 50 tps: 14876.17 qps: 297562.44 (r/w/o: 208322.41/59487.68/29752.34) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 983s ] thds: 50 tps: 14729.29 qps: 294687.74 (r/w/o: 206246.02/58983.15/29458.57) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 984s ] thds: 50 tps: 14853.63 qps: 296947.55 (r/w/o: 207886.79/59353.50/29707.26) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 985s ] thds: 50 tps: 8596.14 qps: 171950.71 (r/w/o: 120370.90/34387.54/17192.27) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 986s ] thds: 50 tps: 14510.96 qps: 290214.18 (r/w/o: 203131.43/58060.84/29021.92) lat (ms,95%): 4.25 err/s: 0.00 reconn/s: 0.00
[ 987s ] thds: 50 tps: 14871.74 qps: 297253.79 (r/w/o: 208091.35/59419.96/29742.48) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 988s ] thds: 50 tps: 14749.95 qps: 295182.05 (r/w/o: 206604.34/59076.81/29500.91) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 989s ] thds: 50 tps: 14521.42 qps: 290352.38 (r/w/o: 203262.86/58046.68/29042.84) lat (ms,95%): 4.25 err/s: 0.00 reconn/s: 0.00
[ 990s ] thds: 50 tps: 14531.09 qps: 290497.86 (r/w/o: 203352.30/58088.37/29057.19) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 991s ] thds: 50 tps: 14736.65 qps: 294898.07 (r/w/o: 206440.15/58979.61/29478.31) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 992s ] thds: 50 tps: 14577.47 qps: 291568.48 (r/w/o: 204071.63/58341.90/29154.95) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 993s ] thds: 50 tps: 12146.80 qps: 242942.96 (r/w/o: 170097.17/48552.20/24293.60) lat (ms,95%): 4.33 err/s: 0.00 reconn/s: 0.00
[ 994s ] thds: 50 tps: 14599.83 qps: 292026.55 (r/w/o: 204402.58/58424.32/29199.65) lat (ms,95%): 4.18 err/s: 0.00 reconn/s: 0.00
[ 995s ] thds: 50 tps: 13302.01 qps: 265977.16 (r/w/o: 186169.12/53204.03/26604.02) lat (ms,95%): 4.33 err/s: 0.00 reconn/s: 0.00
[ 996s ] thds: 50 tps: 14498.22 qps: 289884.35 (r/w/o: 202944.04/57943.87/28996.43) lat (ms,95%): 4.25 err/s: 0.00 reconn/s: 0.00
[ 997s ] thds: 50 tps: 14671.89 qps: 293475.90 (r/w/o: 205422.53/58709.58/29343.79) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 998s ] thds: 50 tps: 14655.16 qps: 293156.18 (r/w/o: 205214.23/58631.64/29310.32) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 999s ] thds: 50 tps: 14710.10 qps: 294199.07 (r/w/o: 205908.45/58870.41/29420.21) lat (ms,95%): 4.10 err/s: 0.00 reconn/s: 0.00
[ 1000s ] thds: 50 tps: 14814.68 qps: 296296.67 (r/w/o: 207443.56/59223.74/29629.37) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 1001s ] thds: 50 tps: 14765.29 qps: 295298.74 (r/w/o: 206695.02/59073.15/29530.57) lat (ms,95%): 4.03 err/s: 0.00 reconn/s: 0.00
[ 1002s ] thds: 50 tps: 1512.50 qps: 30292.02 (r/w/o: 21279.89/5987.12/3025.01) lat (ms,95%): 4.33 err/s: 0.00 reconn/s: 0.00


mgr压测时候 出现性能抖动,仅截取了部分输出

sysbench oltp_read_write --db-driver=mysql --mysql-host=ip --mysql-port=3306 --mysql-db=lizi --mysql-user=username--mysql-password='password' --table_size=5000000 --tables=20 --threads=50 --report-interval=1 --rand-type=uniform --time=1800 run


yejr 发表于 2024-4-20 14:48:42

各节点的机器配置以及GreatSQL参数选项设置和 https://greatsql.cn/thread-719-1-1.html 这里的是一样的吗

请提供压测期间 vmstat -S m 1 100 负载观测数据,并最好同时也提供 `perf record -g -a -p `pidof mysqld` -- sleep 30` 的结果

mabai 发表于 2024-4-20 17:12:02

本帖最后由 mabai 于 2024-4-20 17:35 编辑

yejr 发表于 2024-4-20 14:48
各节点的机器配置以及GreatSQL参数选项设置和 https://greatsql.cn/thread-719-1-1.html 这里的是一样的吗
...
1、配置是一样的
下图是我用128threads压测也会出现性能抖动
2、# vmstat -S m 1 100
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
rb   swpd   free   buffcache   si   so    bi    bo   in   cs us sy id wa st
720      0    775    14735287    0    0    11   216    0    000 9900
92      0    741    14735325    0    0   0 715012 1163153 1574324 53 13 3310
1400      0    691    14735372    0    0   0 559612 1048428 1583195 54 13 3210
673      0    791    14735276    0    0   0 465880 1748325 1572358 53 13 3310
962      0    778    14735290    0    0   0 713448 1196782 1583365 53 13 3310
11      0    746    14735333    0    0   0 652640 559750 817756 256 6800
981      0    710    14735344    0    0   0 557960 1288859 1163974 40 10 5000
581      0    668    14735389    0    0   0 622576 1438116 1601763 53 13 3310
402      0    786    14735280    0    0   0 772512 1120681 1529299 51 13 3510
1012      0    754    14735310    0    0   0 548688 1061983 1572827 53 13 3310
752      0    693    14735353    0    0   0 665912 1558388 1557561 53 14 3310
231      0    662    14735396    0    0   0 569368 1364374 1559056 53 13 3310
931      0    787    14735280    0    0   0 567712 1018041 1583737 53 13 3310
823      0    746    14735315    0    0   0 469112 1248035 1572642 54 13 3210
01      0    732    14735330    0    0   0 578896 143767 20907351 9400
391      0    690    14735360    0    0   0 434064 1876286 1409136 48 12 4000
873      0    692    14735371    0    0   4 501172 1077859 1554489 53 13 3310
493      0    731    14735339    0    0   0 669648 1054170 1568069 53 13 3410
951      0    669    14735380    0    0   0 604096 1641296 1531009 53 14 3310
941      0    793    14735273    0    0   0 591040 1863655 1582923 55 14 3110
470      0    748    14735317    0    0   0 580564 1068211 1604894 54 13 3210
562      0    699    14735365    0    01452 622184 955615 1573770 53 13 3310
681      0    664    14735400    0    0   0 598504 1137856 1105693 369 5510
580      0    795    14735268    0    0   0 635912 1510711 1511411 51 13 3610
371      0    770    14735297    0    0   0 518276 1066001 1584010 54 13 3310
1204      0    710    14735342    0    0   0 548240 1347706 1574906 54 13 3310
833      0    675    14735384    0    0   0 476404 1618553 1458447 50 13 3610
880      0    675    14735393    0    0   0 679020 1025746 1568568 53 13 3310
792      0    665    14735395    0    0   0 584368 1080931 1556333 53 13 3310
741      0    782    14735287    0    0   0 544864 1569982 1537875 54 13 3210
891      0    737    14735313    0    0   0 605056 1755044 1450788 48 12 3910
841      0    713    14735353    0    0   0 653144 817048 1434936 48 12 3910
1170      0    671    14735391    0    0   0 493160 791000 1387222 48 12 4010
444      0    790    14735276    0    0   0 620420 1596863 1566986 54 13 3310
931      0    747    14735300    0    0   0 652272 1892585 1567470 52 13 3510
851      0    711    14735345    0    0   0 532624 876952 1586089 55 13 3210
1392      0    676    14735391    0    0   0 624952 863893 1579126 53 13 3410
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
rb   swpd   free   buffcache   si   so    bi    bo   in   cs us sy id wa st
911      0    780    14735278    0    0   0 649268 2042676 1523060 52 13 3510
1282      0    744    14735315    0    0   0 542448 1553823 1571215 54 13 3310
480      0    693    14735360    0    0   0 621920 956136 1578382 54 13 3210
461      0    662    14735408    0    0   0 646980 948703 1604729 54 13 3310
1231      0    761    14735286    0    0    24 464200 2182348 1566673 53 13 3310
1222      0    725    14735328    0    0   0 640136 1353359 1549134 52 13 3410
881      0    692    14735375    0    0   0 680488 886753 1589849 53 13 3310
672      0    675    14735399    0    0   0 618292 1123181 1603447 54 14 3210
521      0    791    14735279    0    0   0 560628 1922023 1590201 53 13 3310
1301      0    737    14735318    0    0   0 597612 1153777 1541282 52 13 3410
822      0    696    14735369    0    05628 497092 989353 1587271 55 13 3210
1001      0    677    14735383    0    0 15180 668252 1440969 1561901 52 13 3410
310      0    683    14735376    0    0   4 660340 1414563 1578704 54 13 3210
512      0    677    14735374    0    0   0 471728 1045645 1547791 54 13 3300
431      0    758    14735286    0    0   0 633456 1782028 1558480 54 13 3210
331      0    714    14735319    0    0   136 612740 1699390 1596925 53 13 3410
802      0    658    14735369    0    0    16 632464 920137 1561751 53 13 3310
420      0    788    14735241    0    0   0 614196 1284378 1442117 50 12 3810
481      0    735    14735282    0    0   0 576812 1539625 1560846 54 13 3310
431      0    696    14735330    0    0   0 575120 1371183 1572977 53 13 3410
663      0    790    14735231    0    0   0 600420 1054690 1588388 54 13 3210
732      0    769    14735272    0    0   0 685348 1055853 1573765 53 13 3310
810      0    679    14735351    0    0 26780 571592 1620269 1548692 54 13 3210
812      0    682    14735325    0    0 49284 548084 1154553 1515620 52 13 3510
191      0    752    14735218    0    0 42424 667316 1053365 1522227 52 13 3510
911      0    704    14735258    0    0   0 635132 1107580 1553225 53 13 3410
1241      0    677    14735295    0    0   0 595356 1360504 1574405 54 13 3310
302      0    778    14735187    0    0   0 571188 1465606 1586327 53 13 3310
351      0    724    14735240    0    03092 592896 1097240 1558549 53 13 3310
901      0    756    14735286    0    07696 565176 1638904 1529611 53 13 3310
1413      0    729    14735327    0    0   0 645052 1572734 1564107 52 13 3410
492      0    694    14735375    0    0   0 670072 889150 1602703 54 13 3310
1111      0    654    14735425    0    0   0 569840 887132 1582557 54 13 3310
1011      0    768    14735284    0    0   0 629668 1682984 1553687 53 13 3310
863      0    733    14735207    0    02968 520356 1424013 1461928 50 13 3710
731      0    721    14735220    0    0   0 559156 1047099 1463301 50 13 3610
441      0    742    14735188    0    0   0 517592 1639002 1371438 49 14 3610
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
rb   swpd   free   buffcache   si   so    bi    bo   in   cs us sy id wa st
1120      0    699    14735225    0    0   0 536396 2274231 1377064 51 15 3310
1061      0    752    14735163    0    0   0 679208 1689570 1391721 50 14 3510
132      0    699    14735215    0    0   0 617560 1350362 1435495 53 15 3210
321      0    686    14735230    0    0   0 532952 1364995 1393994 51 15 3400
503      0    730    14735173    0    0   0 643084 2617060 1371318 49 15 3610
591      0    703    14735213    0    0   0 589112 1358351 1410389 51 15 3410
691      0    672    14735244    0    0   0 577260 1351544 1406940 52 15 3310
681      0    748    14735168    0    0   0 630880 1724297 1354355 50 15 3510
961      0    668    14735246    0    0   0 570372 2133563 1403582 51 15 3410
511      0    744    14735158    0    0   0 475228 1544143 1408012 52 15 3300
1031      0    673    14735239    0    0   0 666588 1347400 1351905 50 14 3510
802      0    746    14735145    0    0   0 585592 2025874 1358319 51 15 3310
1091      0    692    14735196    0    0   208 524944 1917954 1474021 52 14 3310
910      0    672    14735239    0    0   0 742304 875289 1543120 54 13 3310
300      0    788    14735120    0    0   0 609216 984049 1542069 54 13 3210
923      0    715    14735173    0    02956 597484 1869531 1448750 51 13 3510
810      0    817    14735214    0    0   0 655928 1723838 1551041 53 13 3410
1150      0    780    14735257    0    0   0 604704 871486 1586289 54 13 3310
811      0    743    14735303    0    0   0 451604 922923 1584548 54 13 3210
542      0    688    14735352    0    0   0 698900 1683786 1568547 52 13 3410
482      0    790    14735252    0    0   180 606744 1080399 1596809 54 13 3210
1030      0    768    14735279    0    0   0 622212 1116134 1585562 53 13 3310
00      0    734    14735322    0    0   0 705432 707699 1001746 328 5910
890      0    685    14735342    0    0   0 563808 2096801 1187997 40 10 4900
411      0    664    14735384    0    0   0 550732 1037903 1588232 54 13 3310
882      0    785    14735262    0    0   0 704060 1051758 1575831 53 13 3310



3perf.data这个用perf report 看是空白的

yejr 发表于 2024-4-20 19:42:18

mabai 发表于 2024-4-20 17:12
1、配置是一样的
下图是我用128threads压测也会出现性能抖动
2、# vmstat -S m 1...

那运行perf top,持续几分钟后截图也行

yejr 发表于 2024-4-20 19:47:00

mabai 发表于 2024-4-20 17:12
1、配置是一样的
下图是我用128threads压测也会出现性能抖动
2、# vmstat -S m 1...

从这个结果来看,CPU负载过分地高了,应该是某些地方存在瓶颈

试着把sysbench压测客户端和MGR的primary节点放在一起,走本地unix socket方式连接压测再观察

mabai 发表于 2024-4-20 23:09:44

yejr 发表于 2024-4-20 19:47
从这个结果来看,CPU负载过分地高了,应该是某些地方存在瓶颈

试着把sysbench压测客户端和MGR的primary ...

我是放在一起的,但是采用的是连接VIP的模式,我把numa打开了

mabai 发表于 2024-4-20 23:41:15

本帖最后由 mabai 于 2024-4-20 23:43 编辑

yejr 发表于 2024-4-20 19:47
从这个结果来看,CPU负载过分地高了,应该是某些地方存在瓶颈

试着把sysbench压测客户端和MGR的primary ...
这是压测了400多秒时候的截图

yejr 发表于 2024-4-22 08:57:52

mabai 发表于 2024-4-20 23:41
这是压测了400多秒时候的截图

请再做几个尝试
1. 关闭NUMA;
2. sysbench改用unix socket方式连接,不走tcp
3. 观察错误日志,看看是否有类似 " Plugin group_replication reported: 'MGR request time" 这样的事件

mabai 发表于 2024-4-22 10:49:47

yejr 发表于 2024-4-22 08:57
请再做几个尝试
1. 关闭NUMA;
2. sysbench改用unix socket方式连接,不走tcp


1、已关闭numa
2、新增了--mysql-socket参数
3、 没有出现 类似的时间

性能基本无抖动。

想额外请教一个问题,压测过程当中发现延迟较为严重,在20分钟以上

yejr 发表于 2024-4-22 11:16:48

mabai 发表于 2024-4-22 10:49
1、已关闭numa
2、新增了--mysql-socket参数
3、 没有出现 类似的时间


请问你怎么确认延迟20分钟的呢,通过观测哪些指标
页: [1] 2
查看完整版本: greatesql-8.0.32-25 压测的时候出现性能抖动