部署方式

3节点部署etcd，通过https访问

etcd 内存 8G

特殊配置

修改etcd存储为100G

ETCD_QUOTA_BACKEND_BYTES=107374182400

调整bbolt Freelist为map
ETCD_EXPERIMENTAL_BACKEND_BBOLT_FREELIST_TYPE=map

验证情况总结

Etcd写入性能（put)

并发1w，etcd会出现大量异常

并发5k，比较稳定

验证过程

写入验证

并发1 创建1

benchmark --cacert /etc/ssl/etcd/ssl/ca.pem 
  --cert /etc/ssl/etcd/ssl/xx.pem 
  --key /etc/ssl/etcd/ssl/xx.pem 
  --endpoints https://xx:2379,https://xx:2379,https://xx:2379 
  put  willow --clients 1  --conns 1 --total 1 
  --key-size 128 --val-size 4096 

Summary:
  Total:    0.0087 secs.
  Slowest:    0.0086 secs.
  Fastest:    0.0086 secs.
  Average:    0.0086 secs.
  Stddev:    0.0000 secs.
  Requests/sec:    115.0226

Response time histogram:
  0.0086 [1]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.0086 [0]    |
  0.0086 [0]    |
  0.0086 [0]    |
  0.0086 [0]    |
  0.0086 [0]    |
  0.0086 [0]    |
  0.0086 [0]    |
  0.0086 [0]    |
  0.0086 [0]    |
  0.0086 [0]    |

Latency distribution:

并发200 创建100000

benchmark --cacert /etc/ssl/etcd/ssl/ca.pem 
  --cert /etc/ssl/etcd/ssl/xx.pem 
  --key /etc/ssl/etcd/ssl/xx.pem 
  --endpoints https://xx:2379,https://xx:2379,https://xx:2379 
  put  willow --clients 200  --conns 200 --total 100000 
  --key-size 128 --val-size 4096 

Summary:
  Total:    9.8616 secs.
  Slowest:    0.7639 secs.
  Fastest:    0.0017 secs.
  Average:    0.0195 secs.
  Stddev:    0.0375 secs.
  Requests/sec:    10140.3505

Response time histogram:
  0.0017 [1]    |
  0.0779 [95541]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.1541 [2975]    |∎
  0.2304 [713]    |
  0.3066 [231]    |
  0.3828 [456]    |
  0.4590 [58]    |
  0.5353 [10]    |
  0.6115 [12]    |
  0.6877 [2]    |
  0.7639 [1]    |

Latency distribution:
  10% in 0.0048 secs.
  25% in 0.0058 secs.
  50% in 0.0089 secs.
  75% in 0.0141 secs.
  90% in 0.0455 secs.
  95% in 0.0720 secs.
  99% in 0.2188 secs.
  99.9% in 0.3654 secs.

并发2000 创建100000

benchmark --cacert /etc/ssl/etcd/ssl/ca.pem 
  --cert /etc/ssl/etcd/ssl/xx.pem 
  --key /etc/ssl/etcd/ssl/xx.pem 
  --endpoints https://xx:2379,https://xx:2379,https://xx:2379 
  put  willow --clients 2000  --conns 2000 --total 100000 
  --key-size 128 --val-size 4096 
  
Summary:
  Total:    6.5828 secs.
  Slowest:    0.9898 secs.
  Fastest:    0.0028 secs.
  Average:    0.1279 secs.
  Stddev:    0.1123 secs.
  Requests/sec:    15190.9975

Response time histogram:
  0.0028 [1]    |
  0.1015 [48845]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.2002 [36525]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.2989 [6503]    |∎∎∎∎∎
  0.3976 [3651]    |∎∎
  0.4963 [3094]    |∎∎
  0.5950 [603]    |
  0.6937 [188]    |
  0.7924 [179]    |
  0.8911 [374]    |
  0.9898 [37]    |

Latency distribution:
  10% in 0.0353 secs.
  25% in 0.0565 secs.
  50% in 0.1038 secs.
  75% in 0.1447 secs.
  90% in 0.2721 secs.
  95% in 0.3602 secs.
  99% in 0.5357 secs.
  99.9% in 0.8188 secs.

并发5000 创建100000

  benchmark --cacert /etc/ssl/etcd/ssl/ca.pem 
  --cert /etc/ssl/etcd/ssl/xx.pem 
  --key /etc/ssl/etcd/ssl/xx.pem 
  --endpoints https://xx:2379,https://xx:2379,https://xx:2379 
  put  willow --clients 5000  --conns 5000 --total 100000 
  --key-size 128 --val-size 4096 

Summary:
  Total:    16.5595 secs.
  Slowest:    2.0423 secs.
  Fastest:    0.0024 secs.
  Average:    0.8085 secs.
  Stddev:    0.3494 secs.
  Requests/sec:    6038.8424

Response time histogram:
  0.0024 [1]    |
  0.2064 [3450]    |∎∎∎∎∎∎∎
  0.4104 [11455]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.6144 [19345]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.8184 [16299]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.0224 [19242]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.2264 [16561]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.4303 [9065]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.6343 [4008]    |∎∎∎∎∎∎∎∎
  1.8383 [557]    |∎
  2.0423 [17]    |

Latency distribution:
  10% in 0.3470 secs.
  25% in 0.5554 secs.
  50% in 0.8146 secs.
  75% in 1.0454 secs.
  90% in 1.2669 secs.
  95% in 1.4002 secs.
  99% in 1.5371 secs.
  99.9% in 1.7958 secs.

Summary:
  Total:    13.6374 secs.
  Slowest:    2.1364 secs.
  Fastest:    0.0028 secs.
  Average:    0.6559 secs.
  Stddev:    0.2851 secs.
  Requests/sec:    7332.7822

Response time histogram:
  0.0028 [1]    |
  0.2162 [3561]    |∎∎∎∎∎
  0.4295 [23349]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.6429 [26266]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.8563 [21458]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.0696 [18364]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.2830 [5575]    |∎∎∎∎∎∎∎∎
  1.4963 [1228]    |∎
  1.7097 [160]    |
  1.9231 [34]    |
  2.1364 [4]    |

Latency distribution:
  10% in 0.3158 secs.
  25% in 0.4117 secs.
  50% in 0.6177 secs.
  75% in 0.8607 secs.
  90% in 1.0413 secs.
  95% in 1.1171 secs.
  99% in 1.3022 secs.
  99.9% in 1.6058 secs.

4G

Summary:
  Total:    15.2797 secs.
  Slowest:    1.9684 secs.
  Fastest:    0.0030 secs.
  Average:    0.7531 secs.
  Stddev:    0.2727 secs.
  Requests/sec:    6544.6360

Response time histogram:
  0.0030 [1]    |
  0.1995 [95]    |
  0.3961 [8302]    |∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.5926 [25425]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.7892 [24643]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.9857 [20072]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.1822 [14046]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.3788 [5664]    |∎∎∎∎∎∎∎∎
  1.5753 [1166]    |∎
  1.7719 [531]    |
  1.9684 [55]    |

Latency distribution:
  10% in 0.4152 secs.
  25% in 0.5563 secs.
  50% in 0.7296 secs.
  75% in 0.9191 secs.
  90% in 1.1439 secs.
  95% in 1.2452 secs.
  99% in 1.4874 secs.
  99.9% in 1.6231 secs.

并发1W，创建10w

  
benchmark --cacert /etc/ssl/etcd/ssl/ca.pem 
  --cert /etc/ssl/etcd/ssl/xx.pem 
  --key /etc/ssl/etcd/ssl/xx.pem 
  --endpoints https://xx:2379,https://xx:2379,https://xx:2379 
  put  willow --clients 10000  --conns 10000 --total 100000 
  --key-size 128 --val-size 4096 
Summary:
  Total:    83.2836 secs.
  Slowest:    1.4706 secs.
  Fastest:    0.0027 secs.
  Average:    0.4283 secs.
  Stddev:    0.1829 secs.
  Requests/sec:    512.0213

Response time histogram:
  0.0027 [1]    |
  0.1495 [2076]    |????
  0.2963 [5480]    |????????????
  0.4431 [17421]    |????????????????????????????????????????
  0.5899 [11274]    |?????????????????????????
  0.7367 [3992]    |?????????
  0.8835 [1140]    |??
  1.0303 [809]    |?
  1.1771 [327]    |
  1.3238 [117]    |
  1.4706 [6]    |

Latency distribution:
  10% in 0.2144 secs.
  25% in 0.3343 secs.
  50% in 0.4054 secs.
  75% in 0.5124 secs.
  90% in 0.6353 secs.
  95% in 0.7556 secs.
  99% in 1.0307 secs.
  99.9% in 1.2323 secs.

Error distribution:
  [16124]    etcdserver: too many requests
  [37899]    etcdserver: mvcc: database space exceeded
  [3334]    etcdserver: request timed out
  
Summary:
  Total:    12.9188 secs.
  Slowest:    3.2987 secs.
  Fastest:    0.0050 secs.
  Average:    1.3142 secs.
  Stddev:    0.4723 secs.
  Requests/sec:    6991.7237

Response time histogram:
  0.0050 [1]    |
  0.3344 [2062]    |∎∎∎
  0.6637 [5474]    |∎∎∎∎∎∎∎∎
  0.9931 [13790]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.3225 [25723]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.6518 [24558]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.9812 [9700]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  2.3106 [6521]    |∎∎∎∎∎∎∎∎∎∎
  2.6399 [2278]    |∎∎∎
  2.9693 [208]    |
  3.2987 [10]    |

Latency distribution:
  10% in 0.7319 secs.
  25% in 1.0262 secs.
  50% in 1.3035 secs.
  75% in 1.5639 secs.
  90% in 1.9809 secs.
  95% in 2.1758 secs.
  99% in 2.5001 secs.
  99.9% in 2.8421 secs.

Error distribution:
  [9675]    etcdserver: too many requests

并发2w，创建10w

benchmark --cacert /etc/ssl/etcd/ssl/ca.pem 
  --cert /etc/ssl/etcd/ssl/xx.pem 
  --key /etc/ssl/etcd/ssl/xx.pem 
  --endpoints https://xx:2379,https://xx:2379,https://xx:2379 
  put  willow --clients 20000  --conns 20000 --total 100000 
  --key-size 128 --val-size 4096 

Summary:
  Total:    15.8323 secs.
  Slowest:    3.7089 secs.
  Fastest:    0.0034 secs.
  Average:    1.2176 secs.
  Stddev:    0.5662 secs.
  Requests/sec:    5060.8509

Response time histogram:
  0.0034 [1]    |
  0.3739 [1549]    |∎
  0.7445 [6600]    |∎∎∎∎∎∎∎
  1.1150 [28983]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.4856 [33436]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.8561 [3259]    |∎∎∎
  2.2267 [1564]    |∎
  2.5972 [463]    |
  2.9678 [1658]    |∎
  3.3383 [436]    |
  3.7089 [2176]    |∎∎

Latency distribution:
  10% in 0.7233 secs.
  25% in 1.0223 secs.
  50% in 1.1231 secs.
  75% in 1.2799 secs.
  90% in 1.4967 secs.
  95% in 2.8303 secs.
  99% in 3.5279 secs.
  99.9% in 3.6458 secs.

Error distribution:
  [18126]    etcdserver: too many requests
  [1749]    etcdserver: request timed out