kubernetes+ prometheus自动伸缩的设计与实现(四)

之前还有一些环境部署上面的事情没有交代，这里补充完毕

apiserver参数修改

1
2
3
4
1--runtime-config=autoscaling/v2alpha1=true  

2

3"--requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-client-ca-file=/var/run/kubernetes/request-header-ca.crt --requestheader-allowed-names=system:auth-proxy --proxy-client-cert-file=/var/run/kubernetes/client-auth-proxy.crt --proxy-client-key-file=/var/run/kubernetes/client-auth-proxy.key",

4

controller-manager参数修改

1
2
1--horizontal-pod-autoscaler-use-rest-clients --horizontal-pod-autoscaler-sync-period=60s

2

创建secert

1
2
1kubectl create  secret generic  cm-adapter-serving-certs --from-file=serving.crt=apiserver.crt --from-file=serving.key=apiserver.key -n custom-metrics

2

创建转化器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
1kind: ServiceAccount

2apiVersion: v1

3metadata:

4  name: custom-metrics-apiserver

5  namespace: custom-metrics

6---

7apiVersion: rbac.authorization.k8s.io/v1beta1

8kind: ClusterRoleBinding

9metadata:

10  name: custom-metrics:system:auth-delegator

11roleRef:

12  apiGroup: rbac.authorization.k8s.io

13  kind: ClusterRole

14  name: system:auth-delegator

15subjects:

16- kind: ServiceAccount

17  name: custom-metrics-apiserver

18  namespace: custom-metrics

19---

20apiVersion: rbac.authorization.k8s.io/v1beta1

21kind: RoleBinding

22metadata:

23  name: custom-metrics-auth-reader

24  namespace: kube-system

25roleRef:

26  apiGroup: rbac.authorization.k8s.io

27  kind: Role

28  name: extension-apiserver-authentication-reader

29subjects:

30- kind: ServiceAccount

31  name: custom-metrics-apiserver

32  namespace: custom-metrics

33---

34apiVersion: rbac.authorization.k8s.io/v1beta1

35kind: ClusterRole

36metadata:

37  name: custom-metrics-resource-reader

38rules:

39- apiGroups:

40  - ""

41  resources:

42  - configmaps

43  - namespaces

44  - pods

45  - services

46  verbs:

47  - get

48  - list

49---

50apiVersion: rbac.authorization.k8s.io/v1beta1

51kind: ClusterRoleBinding

52metadata:

53  name: custom-metrics-resource-reader

54roleRef:

55  apiGroup: rbac.authorization.k8s.io

56  kind: ClusterRole

57  name: custom-metrics-resource-reader

58subjects:

59- kind: ServiceAccount

60  name: custom-metrics-apiserver

61  namespace: custom-metrics

62---

63apiVersion: extensions/v1beta1

64kind: Deployment

65metadata:

66  labels:

67    app: custom-metrics-apiserver

68  name: custom-metrics-apiserver

69  namespace: custom-metrics

70spec:

71  replicas: 1

72  selector:

73    matchLabels:

74      app: custom-metrics-apiserver

75  template:

76    metadata:

77      labels:

78        app: custom-metrics-apiserver

79      name: custom-metrics-apiserver

80    spec:

81      serviceAccountName: custom-metrics-apiserver

82      containers:

83      - name: custom-metrics-apiserver

84        image: harbor.enncloud.cn/paas/k8s-prom

85        args:

86        - /adapter

87        - --secure-port=6443

88        - --tls-cert-file=/var/run/serving-cert/serving.crt

89        - --tls-private-key-file=/var/run/serving-cert/serving.key

90        - --logtostderr=true

91        - --prometheus-url=http://prometheus-monitor.kube-system:9090/

92        - --requestheader-client-ca-file=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

93        - --metrics-relist-interval=30s

94        - --rate-interval=30s

95        - --v=10

96        ports:

97        - containerPort: 6443

98        volumeMounts:

99        - mountPath: /var/run/serving-cert

100          name: volume-serving-cert

101          readOnly: true

102      volumes:

103      - name: volume-serving-cert

104        secret:

105          secretName: cm-adapter-serving-certs

106---

107apiVersion: v1

108kind: Service

109metadata:

110  name: api

111  namespace: custom-metrics

112spec:

113  ports:

114  - port: 443

115    targetPort: 6443

116  selector:

117    app: custom-metrics-apiserver

118---

119apiVersion: apiregistration.k8s.io/v1beta1

120kind: APIService

121metadata:

122  name: v1alpha1.custom-metrics.metrics.k8s.io

123spec:

124  insecureSkipTLSVerify: true

125  group: custom-metrics.metrics.k8s.io

126  groupPriorityMinimum: 1000

127  versionPriority: 15

128  service:

129    name: api

130    namespace: custom-metrics

131  version: v1alpha1

132---

133# Make a ClusterRole so that the HPA controller is able to read the custom metrics this adapter provides

134apiVersion: rbac.authorization.k8s.io/v1beta1

135kind: ClusterRole

136metadata:

137  name: custom-metrics-server-resources

138rules:

139- apiGroups:

140  - custom-metrics.metrics.k8s.io

141  resources: ["*"]

142  verbs: ["*"]

143---

144apiVersion: rbac.authorization.k8s.io/v1beta1

145kind: ClusterRoleBinding

146metadata:

147  name: hpa-controller-custom-metrics

148roleRef:

149  apiGroup: rbac.authorization.k8s.io

150  kind: ClusterRole

151  name: custom-metrics-server-resources

152subjects:

153- kind: ServiceAccount

154  name: horizontal-pod-autoscaler

155  namespace: kube-system

156

这里数部署文档，通过上面yaml就可以创建出，

校验一下：

1
2
1kubectl get --raw=apis/custom-metrics.metrics.k8s.io/v1alpha1

2

建立一个内存伸缩的规则

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
1kind: HorizontalPodAutoscaler

2apiVersion: autoscaling/v2alpha1

3metadata:

4  name: testhpa

5  namespace: wangxxg

6spec:

7  scaleTargetRef:

8    apiVersion: apps/v1beta1

9    kind: Deployment

10    name: tomcat

11  minReplicas: 1

12  maxReplicas: 3

13  metrics:

14  - type: Pods

15    pods:

16      metricName: memory_usage_bytes

17      targetAverageValue: 5685461333m

18

如果是CPU则是，cpu_usage。