0%

云环境搭建k3s集群实现gitlab的cicd

gitlab搭建在自家的nas上,k3s部署于阿里云的ecs,自己提交代码后,通过gitlab runner自动执行pipeline。
cicd自动执行2个操作:

  1. nas代码自动备份到github
  2. 博客自动发布到github pages

部署环境

  • master:1C2G,CentOS Linux 7 (Core)
  • agent1:2C8G,CentOS Linux 8
  • agent2:1C2G,Alibaba Cloud Linux 3 (Soaring Falcon)

别人的好机器只能当worker,自己的烂机器当master,-_-

端口开通

阿里云ecs安全策略开通以下接口。

协议 端口 描述
TCP 6443 K3s agent 节点 Kubernetes API Server
UDP 8472 K3s server 和 agent 节点 仅对 Flannel VXLAN 需要
TCP 10250 K3s server 和 agent 节点 Kubelet metrics
TCP 2379-2380 K3s server 节点 只有嵌入式 etcd 高可用才需要 

master

master_ip=ip1

安装master

1
2
3
4
curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | \
INSTALL_K3S_MIRROR=cn \
INSTALL_K3S_EXEC="--tls-san ${master_ip}" \
sh -s -

修改启动配置

sudo vim /etc/systemd/system/multi-user.target.wants/k3s.service

1
ExecStart=/usr/local/bin/k3s server  '--tls-san' '${master_ip}'  write-kubeconfig-mode 664  --node-ip  ${master_ip} --node-external-ip ${master_ip}

apiserver地址

https://${master_ip}:6443

master操作

  • 查看join_tokencat /var/lib/rancher/k3s/server/node-token
  • 查看kubeconfig:cat /etc/rancher/k3s/k3s.yaml

agent

agent_ip=ip2

安装agent

1
2
3
curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | \
INSTALL_K3S_MIRROR=cn \
sh -s -

修改启动配置

sudo vim /etc/systemd/system/multi-user.target.wants/k3s.service

1
/usr/local/bin/k3s  agent --server https://${master_ip}:6443 --node-name ${agent_ip} --with-node-id --node-ip  ${agent_ip} --node-external-ip ${agent_ip} --token  ${join_token}

重启

systemctl daemon-reload
systemctl restart k3s

dashboard

  • bear token生成:k3s kubectl -n kubernetes-dashboard describe secret admin-user-token | grep '^token'
  • 本地端口代理:kubectl port-forward -n kubernetes-dashboard service/kubernetes-dashboard 8080:443
  • 浏览器访问
    • firefox访问:https://localhost:8080
    • chrome访问:https://localhost:8080,在打开后键盘输入:thisisunsafe

gitlab连接k3s集群

api地址

https://${master_ip}:6443

certification

k3s kubectl config view —raw -o=jsonpath=’{.clusters[0].cluster.certificate-authority-data}’ | base64 —decode

创建账户

kubectl apply -f gitlab_admin.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: v1
kind: ServiceAccount
metadata:
name: gitlab-admin
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: gitlab-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: gitlab-admin
namespace: kube-system

查看token

1
2
3
4
SECRET=$(kubectl -n kube-system get secret | grep gitlab-admin | awk '{print $1}')
echo $SECRET
TOKEN=$(kubectl -n kube-system get secret $SECRET -o jsonpath='{.data.token}' | base64 --decode)
echo $TOKEN

本地代理

kubectl port-forward -n kubernetes-dashboard service/kubernetes-dashboard 8080:443

网络调试

调试工具

1
2
3
4
5
6
7
8
9
10
11
12
13
14
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: busybox:1.28.4
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
restartPolicy: Always

测试dns:kubectl exec -it busybox -- nslookup gitlab-org.gitlab.io

用好dignslookup排查网络连接问题

coredns配置域名

gitlab-runner内部报错,不能访问外网的gitlab服务器

1
WARNING: Checking for jobs... failed runner=6Kxs-qJj status=couldn't execute POST against http://www.geosmart.top:40000/api/v4/jobs/request: Post http://www.geosmart.top:40000/api/v4/jobs/request: dial tcp: lookup www.geosmart.top on 10.43.0.10:53: no such host

将root dns服务器由forward /etc/resolv.conf改为外网的forward . 114.114.114.114就可以了

针对不能连接的域名,在kubectl -n kube-system edit cm coredns新增host配置即可

1
2
3
4
5
6
hosts /etc/coredns/NodeHosts {
115.196.121.1 www.mydomain.top
ttl 60
reload 15s
fallthrough
}

k3s卸载

/usr/local/bin/k3s-uninstall.sh

gitlab.cicd

cicd

gitlab-ci.yaml示例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
image: node:10.19.0

cache:
paths:
- hexo/node_modules/

before_script:
- cd hexo
- npm install hexo-cli -g
- test -e package.json && npm install

stages:
- deploy

deploy:
stage: deploy
script:
- hexo g -d
only:
- master

github推送问题

gnutls_handshake() failed: The TLS connection was non-properly terminated.
因为source有git修改记录,public网站不需要修改记录
所以改为ssh认证,每次push -f强制替换网站

参考