DevOps with Gitlab-CI && Kubernetes • An de Arms

将代码推送到GitLab之后，现在开始配置GitLab CICD 流水线配置

1、流水线文件解读#

GitLab流水线类似Github Action，都是通过在项目根目录下的特定文件来定义流水线步骤和规则，GitLab采用的流水线配置文件为.gitlab-ci.yml，以center-course服务的流水线规则演示，其他项目的流水线规则大同小异。

stages:
  - 项目打包
  - 镜像构建
  - 归档产物
  - 镜像推送
  - 集群部署

variables:
  # 从CI/CD配置中读取环境变量
  PROJECT_NAME: $PROJECT_NAME
  IMAGE_VERSION: $CI_PIPELINE_IID
  DOCKER_REGISTRY: $DOCKER_REGISTRY
  MY_PROFILE: $SPRING_PROFILE_ACTIVE
  MY_SERVER: $INFRASTRUCTURE_HOST

Package With Maven:
  stage: 项目打包
  script:
    - pwd
    - mvn clean package -Dmaven.test.skip=true
  cache:
    key: binaries-cache
    paths:
      - hiseas-center-course-service/target/*.jar

Build Docker Image:
  stage: 镜像构建
  script:
    - pwd
    - ls -al
    - docker build  --build-arg DOCKER_REGISTRY=$DOCKER_REGISTRY  --build-arg PROFILE=$MY_PROFILE   --build-arg SERVER_URL=$MY_SERVER  -t $DOCKER_REGISTRY/$PROJECT_NAME:v$IMAGE_VERSION .
  cache:
    key: binaries-cache
    paths:
      - hiseas-center-course-service/target/*.jar

Archive Tag Artifact:
  stage: 归档产物
  rules:
    - if: '$CI_COMMIT_TAG'
  script:
    - echo "Archiving Artifacts"
  artifacts:
    name: $PROJECT_NAME-$CI_COMMIT_TAG
    paths:
      - target/*.jar

Push Docker Image:
  stage: 镜像推送
  script:
    - echo "Pushing Docker image to $DOCKER_REGISTRY"
    - docker push $DOCKER_REGISTRY/$PROJECT_NAME:v$IMAGE_VERSION

Deploy Kubernetes:
  stage: 集群部署
  script:
    - echo "Deploying Kubernetes"
    - cd /usr/local/app/hiseas
    # 第一次执行
    - kubectl apply -f $PROJECT_NAME-$MY_PROFILE.yaml
    - kubectl set image deployment/$PROJECT_NAME-$MY_PROFILE $PROJECT_NAME-$MY_PROFILE=$DOCKER_REGISTRY/$PROJECT_NAME:v$IMAGE_VERSION -n hiseas-dev

yaml

PROJECT_NAME、DOCKER_REGISTRY、SPRING_PROFILE_ACTIVE、INFRASTRUCTURE_HOST这四个变量是在GitLab项目的CICD配置相中定义的变量，由于我们的服务在打包成Docker镜像时指定了2个参数变量，这2个变量在项目的配置变量中提供，其中在镜像构建阶段中的--build-arg PROFILE=$MY_PROFILE --build-arg SERVER_URL=$MY_SERVER 参数就是在传递Dockerfile中需要的环境变量。

DOCKER_REGISTRY：10.211.55.11:5566/hiseas-cloud

INFRASTRUCTURE_HOST：10.211.55.10

PROJECT_NAME：hiseas-center-course

SPRING_PROFILE_ACTIVE：dev
DOCKER_REGISTRY：指定Docker镜像地址，也就是Harbor地址，这里注意，由于GitLab流水线是调用GitLab Runner来执行的，所以在GitLab Runner的服务器上需要配置私有镜像仓库地址
```
{
  "insecure-registries" : ["http://10.211.55.11:5566"]
}
```
json
缓存构建物
```
cache:
  key: binaries-cache
  paths:
    - hiseas-center-course-service/target/*.jar
```
yml
该配置的含义，将本次构建的产物缓存起来，用于后续构建使用

归档产物

Archive Tag Artifact:
  stage: 归档产物
  rules:
    - if: '$CI_COMMIT_TAG'
  script:
    - echo "Archiving Artifacts"
  artifacts:
    name: $PROJECT_NAME-$CI_COMMIT_TAG
    paths:
      - target/*.jar

yml

如果代码提交了一个tag，那么进行产物的归档，具体要归档的文件是 target目录下的所有jar文件

Build Number

CI_PIPELINE_IID是流水线构建的一个整数数字，该数字唯一且递增，可以用作我们的镜像版本号
集群部署

首先在Kubernetes集群的/usr/local/app/hiseas目录下准备好所有服务的Deployment配置文件

2、Harbor#

GitLab 项目CICD变量中的DOCKER_REGISTRY环境变量的值配置的值为10.211.55.11:5566/hiseas-cloud，因此首先需要再Harbor中建立指定的项目

由于在Kubernetes部署阶段需要从Harbor拉取镜像，所以需要配置一个Docker Registry 秘钥

kubectl create secret docker-registry harbor-pass \
  --docker-server=10.211.55.11:5566 \
  --docker-username=admin \
  --docker-password=759023074Fx

shell

harbor-pass这个秘钥将在Deployment配置文件中使用

3、部署文件#

以center-course服务的Kubernetes Deployment文件举例说明

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: hiseas-dev
  name: hiseas-center-course-dev
  labels:
    app: hiseas-center-course-dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hiseas-center-course-dev
  template:
    metadata:
      labels:
        app: hiseas-center-course-dev
    spec:
      containers:
        - name: hiseas-center-course-dev
          image: 10.211.55.11:5566/hiseas-cloud/hiseas-center-user:v32
          ports:
            - containerPort: 9000
      imagePullSecrets:
        - name: harbor-pass # 确保你已创建了 secret
---
apiVersion: v1
kind: Service
metadata:
  namespace: hiseas-dev
  name: hiseas-center-course-svc-dev
spec:
  selector:
    app: hiseas-center-course-dev
  ports:
    - protocol: TCP
      port: 9000
      targetPort: 9000
  type: ClusterIP # 如果要暴露为 LoadBalancer，可更改为 LoadBalancer

yaml

主要创建了一个Deployment和ClusterIP类型的service，其中镜像地址10.211.55.11:5566/hiseas-cloud/hiseas-center-user:v32中的v32可以随意写，因为我们的流水线会在推送完镜像后自动更新这个版本号

4、流水线部署逻辑#

再回头看看GitLab流水线定义的部署逻辑

Deploy Kubernetes:
  stage: 集群部署
  script:
    - echo "Deploying Kubernetes"
    - cd /usr/local/app/hiseas
    # 第一次执行
    - kubectl apply -f $PROJECT_NAME-$MY_PROFILE.yaml
    - kubectl set image deployment/$PROJECT_NAME-$MY_PROFILE $PROJECT_NAME-$MY_PROFILE=$DOCKER_REGISTRY/$PROJECT_NAME:v$IMAGE_VERSION -n hiseas-dev

shell

首先根据我们的Deployment文件创建资源，然后紧接着就将指定Deployment的image设置为了刚刚推送的镜像版本

5、kubectl配置#

由于Gitlab Runner所在服务器与Kubernetes master不在同一台服务器，需要远程连接到集群

将master的配置文件 ~/.kube/config 文件拷贝到 Runner 服务器的相同目录下

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJVmRqVG0rd3YvTjR3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TlRBeE1URXdPVFUzTlRSYUZ3MHpOVEF4TURreE1EQXlOVFJhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUUN1SFZpenlSa1JOcWlGb3lrQ2hTSUVZZ2NrdE44KzFlZTdSaUJncnBWckpGamxUSWNsaUxpOG1pMG8KTWIvTTF6Sk5mOXQ4WnQxRGxjaTFrZUZHekNIUnZoM0ZHcVNLYVprbGF3bXpuK3ZqZUp1eVliNFZDSzdQdG9xdApUbDVJcVhhdk51UHQzK2R1enA4MkJOQk04VGY5Mng5MHF0aU9oTmpHb3NvUGttdUNmYy9HREFvUWg0KzFjL0hJClVGUHM3ZXVHLzVwYVo5TEZ5REY5ekcyR3EvWGFSYVROTWpOMGFKY1J4bE9yTC9rSm9VWUo0K0hSZjRRazhudHQKdlZidTJXcStMWUE1SWV2TGEwczdRV0ZXWHMxK3lNK3ovS0kvVUxERVkrT2dmaVdURkdMbm9wMEl6QVkxeXN1TgpqQ0FpanhUbWdmYjNLendDTkNvUE9ta1pFSGNmQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJUc3VuYUcwRTlqOVN1UUdmQ1JyalJrZE5kTWRUQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQlU4Y0orbTVHMAozZmdvb0VZcFNTcEhRbXVuaVRreUVzcWlpM1l3MUJlTUNNVXh2NVl3U202UWFFRDU4eXdXVmdRVGxBeWlMbG9GCk9qOHVzMmRxNXpEc3dQVG9qTmZYWlA1alpoWm51QmhEZ3F5YmU5eGZNK0d3T01xQWgxaC9JR3dJekg2YUQxdmkKaUtIK3RPUWh2MDgxTmpZWEkzSktyRkRQbm9qKy9VVVJsRTdhVWtqNVZpRmJlMklhMGN3Y01lUS9JTldsaUsrOAo4VXdQZ0RKaFlLakxONmVtaHRsS0Y5Sys2QUx6WWdzSEdTRC8rZjJnbGJjdkY0dDh1aUtSb2lTREM5UTZQTlBqClMvY2dYeWpka1UyRGN2aXYyYkdHZG44Y2swN1FUOFc0SFNrVzFNcWZxN0ZGVzlPTHh4di9qc1VZOTVEQTFzVlAKcjA2T29RcUtzZ2piCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://10.211.55.7:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURLVENDQWhHZ0F3SUJBZ0lJWHBvNFVlNFljc1F3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TlRBeE1URXdPVFUzTlRSYUZ3MHlOakF4TVRFeE1EQXlOVFJhTUR3eApIekFkQmdOVkJBb1RGbXQxWW1WaFpHMDZZMngxYzNSbGNpMWhaRzFwYm5NeEdUQVhCZ05WQkFNVEVHdDFZbVZ5CmJtVjBaWE10WVdSdGFXNHdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFER0ZRU1QKWlNGZC9kVDl4U3UrQWY5TzNZMll3eks4OEVSanpRczUwMGNVc0JHOXN6N2F1RTVFb1ZJUVVjNHo2c0xlRURvaQpNMGtqdnFKUHNBUkxwazlsOEZnbXRTM1U3WnRlRDdjRndSM2gySTc4K2RDUWNBRHBVaDk5L3VDVVYrbVNzbWdiCi9aUXJxUzhYSTJiTjhHVld2c25DTnJXcU9UanQxU0ZhZEh5a0k1N3dYVEY2eUR1ZzlXVktWYTgyb0tFcTY5SlcKMkJwanVoU3hJRXVFZGJscER2N3dHb0hXaDRFbEtjS0RIWVcvRmFIS0xKY08remsrNDlHZ1l5cmx4WUdyMUcvSQo0T2xZTnNvdTBVYnRIY3hCNVdHT1JNT29RbXVJR2JQTjJ1aXJYM2hPTTBqY002clNRc3d1MXBQTEt0dlZtblk4ClZrTkw4M2lxNXUrdTAzYlBBZ01CQUFHalZqQlVNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUsKQmdnckJnRUZCUWNEQWpBTUJnTlZIUk1CQWY4RUFqQUFNQjhHQTFVZEl3UVlNQmFBRk95NmRvYlFUMlAxSzVBWgo4Skd1TkdSMDEweDFNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUF1ZUV1dlQvSE9BYXZRTUV3OFlHTnUwMXNlCjMxUXhVbVpuVzdrTHBCRmgvSndKR0k1WUs4K1U2N3ovMEFCZWtGbHRUQVZyMkZlbEhwbmhtWm5iaVZ1Wk43MDEKY1lxdFNnOXJaenhZTXFBYkhNS0VwTGM5bUhFN3NkTnhraWdWOXZ1VWdnWnZNL3ZVTkN3cjR4c2FIaHRDNWNaSgo2L0RINVQ3NUYxS0NsWHhEL2Q4d0UvNlVna2pMVmhMR2MwVm5qR2MzNzdydFYrRkllRWdLZ2h1b21kNjlSNEl1CmttczJmUERUOTc1ZU55MjNBMFg4YXBrOEpKOFp3bzRobW5LZWRTbXJoWDBTcUhmZ3FHNUhkT0tLdVV4MmxBMjMKT1VZMHdDNjVnRE96b1h0QkVEbFU3OXZhTlRydDA0SUp5SGx2Q3BVR3FEdEdtNzBaTDhoMXd5OHV3azdNCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb2dJQkFBS0NBUUVBeGhVRWsyVWhYZjNVL2NVcnZnSC9UdDJObU1NeXZQQkVZODBMT2ROSEZMQVJ2Yk0rCjJyaE9SS0ZTRUZIT00rckMzaEE2SWpOSkk3NmlUN0FFUzZaUFpmQllKclV0MU8yYlhnKzNCY0VkNGRpTy9QblEKa0hBQTZWSWZmZjdnbEZmcGtySm9HLzJVSzZrdkZ5Tm16ZkJsVnI3SndqYTFxams0N2RVaFduUjhwQ09lOEYweAplc2c3b1BWbFNsV3ZOcUNoS3V2U1Z0Z2FZN29Vc1NCTGhIVzVhUTcrOEJxQjFvZUJKU25DZ3gyRnZ4V2h5aXlYCkR2czVQdVBSb0dNcTVjV0JxOVJ2eU9EcFdEYktMdEZHN1IzTVFlVmhqa1REcUVKcmlCbXp6ZHJvcTE5NFRqTkkKM0RPcTBrTE1MdGFUeXlyYjFacDJQRlpEUy9ONHF1YnZydE4yendJREFRQUJBb0lCQUVtNnhRdEFGMTF1Ly9xOQppM3ljTGJYalRiTlE1Ty9pMXZIMi9uaTFwYjJUaytieWN6RGliVytxUkd1Q3A2ZzNkL1ZFUVA4OTY3bDhXZG9qCmdGWFFQN21IN3FmcHhIZC9RZGVQTGViY2JRMEZtS09kTVA2eVVnU0c1dVgxR3c3Y2NqcUNZYklaZnkrSk45ZjYKKzJjb1ZmQWw3OTkwa2NZY3FvNE1TTy9zY0FDQUFYcVRKT3VHeGdFdTQ0MHdCMHc4VVE4MHhkUjFONTZnd3A4Twp4TjhtWWpQVGkxREl5cDBNZE12TjlmSWV5aDdMbmJvWlZOSU5NU3dDQk1oWUp1c3JiUWUwUkEvenBKVmpScjZzCmZqbWlSYlBnRGVIYUp5QkhVbXh0ZzVINzNnR1NBOFZ0LzhiYW1ITThlTGl6eW4vQmVLbWlRNWZ6dUZwUjVCVkUKcVVKeVdFRUNnWUVBOWhKeWxrMGFMN0NTaHVpMXR4eGhqWE1CK3krOWZwQ3NmVjhXa0haeDdseVRTZmgyMjBVKwpSSUE2OW1sMlRMY1hKYW5lQ0gwaHhkT1BhbjRDL2ZtR0c0cS9XNmxCWkJkcThwVVNBQWxkSjBMZUpDMFE3OGpQCm85T3NBR211MHprVWNmM2pqTXkrZHhSVkx6aDJOZlFLeU5aN0IzcnZNdENTOUJ1c2VKQS83ekVDZ1lFQXpoTG8KRW1NNXRmZ3ZSVVI1MDlUSEVoWVhnRW5POFlDUU1Pc05ja21jR0M0bTRScHdBTzBFU2xqaDd3dXdIcGI0NnRoTgo1WWhjdzhYTXZvRzN4UE9jY0tkYTJRUlI0S0NjaURpNVhxQnFVeDRTdm9EcWhrczRraVRqRWVpemdOOTBsdDZEClJUYUk5YmpNR0RJMXhJc0trOHNFMlJVU0xyOVFxc1NTOHN0c1JmOENnWUJVM0MzS0taTFRTWGh1c0FnaXV0WGwKbXZqZThDclFZQUlzUmtPWTFGQWNISzdxdnBYMnJsMTM4Q1RsWDU5ZGUzS3ZHWEtweU1XQjNLREQzK3NXUFJ3RAptWWkraXBNemVtUDJmblF6c1FjQUNDNWxtV2tTMkZUTE1hOUtnU2U2VGgvLzQ4ZTZnYzFaY1ppaTdUelBCSGRICnBiZk43NVpZRnRmbStUZzBaOEswc1FLQmdGOEdBTlZadytiUkJwc2VORDhsd3BYb2NmWEtFdzd4aFl2SkYyNDMKMnk2Yk9YcGJRb0pxeUt1dnlKNzkrTHhpU1cyaFNHb3JNdnRqWGtJS2RyN3V4cGg3MHp4MjdYcHpGVHNMVFE4QgpVWVg2UEJKMFI3Q2RSdnk4M2FnSWNaMWZiTmVVeTF0REJteW9SaFJ5TXJEcW5Ib0NCNzVDSlJ4QlErVTZ0N2dWCmJWdnBBb0dBVStRNGZxN0VVRkowNHVWSHEreXdlZE5mMHp3M0cxR1pFWmZlMEUwT2hyQVM0RTRQbmRNb2NkRXMKNmNOT2hzLzA2YUs4UjdPNmZBdTNvTkxCbFRNVEQyRzgyb2I0YWhxelJ1dVlpSDk5ZUpIR04xM3B5Rzl1ZzhsUwpuWUtwc0R5TjYvdDhYRVpKQ1Q3LzFJNFNJMTU4alA5SzlSTGpOZHdwVHF3L0Y5YzdXSnc9Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==

shell

这样在Runner服务器上也能够执行kubectl指令

6、流水线测试#

尝试修改推送代码到Gitlab

在Gitlab项目主页查看流水线运行情况和日志

查看Harbor中的镜像信息

镜像版本是v10，检查集群中运行的Deployment的镜像版本

可以看到集群中运行的镜像版本号正确同步完成

7、服务运行验证#

将gateway服务的service 类型有ClusterIP 修改为NodePort 临时测试

使用任意集群ip+NodePort端口进行接口测试

curl -X POST http://10.211.55.7:31408/auth/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{
    "username": "zed",
    "password": "5566"
  }'

shell

通过聚合服务接口查询

首先查看iam服务的pod 获取登录的token

curl -X GET http://10.211.55.7:31408/mgmt-application/api/courses/3 \
  -H "Access-Token: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJsb2dpblR5cGUiOiJsb2dpbiIsImxvZ2luSWQiOjEsInJuU3RyIjoiY3lNNzd3d2hWMU1RVks5dU5acGJ1ZzdSRXQwZXlVa0YiLCJmdWxsTmFtZSI6Imhpc2VhcyJ9.-9N3jeKwH-cbxE9CjKt7qToqAa3kskghCwnP-dphzHY"

shell

通过聚合服务调用分布式事务测试接口

curl -X POST http://10.211.55.7:31408/mgmt-application/api/courseReview \
     -H "Content-Type: application/json" \
     -H "Access-Token: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJsb2dpblR5cGUiOiJsb2dpbiIsImxvZ2luSWQiOjEsInJuU3RyIjoiY3lNNzd3d2hWMU1RVks5dU5acGJ1ZzdSRXQwZXlVa0YiLCJmdWxsTmFtZSI6Imhpc2VhcyJ9.-9N3jeKwH-cbxE9CjKt7qToqAa3kskghCwnP-dphzHY" \
     -d '{
           "courseId": 3,
           "userId": 1,
           "rating": 5
         }'

shell

8、总结#

至此，已经完成了整个微服务项目的CICD流程，从代码提交，到镜像构建、镜像推送、集群Deployment更新庚哥流程都已完成自动化。

整个实验任然存在部分欠缺的地方需要完善和调整

由于Sentinel是部署在集群外部，在进行接口限流的时候，外部的Sentinel是无法访问到集群内部的服务ip的，所以Sentinel限流暂时无法完成，解决办法也比较简单，将Sentinel以Deployment的形式部署到集群内部，然后在服务的yaml配置文件中指定Sentinel在集群内部服务地址即可
所有基础服务，Nacos、Sentinel、Seata、Redis、Mysql等服务都是以单体形式部署，方便实验演示，企业集生产项目中一般都是会做高可用，这部分留在后续实验研究，有了Kubernetes和Helm之后，这些组件的高可用部署方案都会变得特别简单和高效。