为了测试 gitlab CI pipeline， 从而在家里的台式机搭建了测试环境。
为什么进行这次实验？因为自己平时只是用到了 gitlab，它对于我来说是一个黑盒。
为了搞清楚 CI job 里面的一些 function 用法，自己搭建一套环境。一是方便实验，二是自己不能在公司的环境上实验，担心造成破坏。
Update history
2021-07-13 安利学 git 的网站 Learn Git Branching 和 Git 命令太难学？我用一款游戏带你玩转它！
2021-05-24 终于认识到自己的膨胀了，倍感欣慰
2021-01-09 初稿，妹子什么时候才能好好学习呀
Environment
默认技能：
- docker， 虚拟机上安装 docker 的过程就省略了
- DNS， 因为是自定义的域名，所以需要绑定在本地 hosts
最开始 Ubuntu 机器只有 2 cpu 4GB， 安装 gitlab 非常卡顿， load 高到 40+， 增加配置后，顺利安装
| Name | Operating system | CPU | RAM | Disk | IP |
|---|---|---|---|---|---|
| gitlab-main | Ubuntu 18.04.5 LTS | 4 | 8G | 50G | 192.168.64.3 |
| gitlab-runner | CentOS 7.8.2003 | 2 | 2G | 20G | 192.168.64.4 |
| gitlab-runner | Ubuntu 20.04.1 LTS | 2 | 4G | 20G | 192.168.64.5 |
Install gitlab
gitlab
本次实验，为了快速安装选择了 docker 模式。 https://docs.gitlab.com/omnibus/docker/README.html
本地 22 端口被 SSH 占用了，所以改为了 2222，这里需要注意的是，我用的是 社区版本 ce1
2
3
4
5
6
7
8
9sudo docker run --detach \
--hostname gitlab.feiyang.com \
--publish 443:443 --publish 80:80 --publish 2222:22 \
--name gitlab \
--restart always \
--volume $GITLAB_HOME/config:/etc/gitlab \
--volume $GITLAB_HOME/logs:/var/log/gitlab \
--volume $GITLAB_HOME/data:/var/opt/gitlab \
gitlab/gitlab-ce:latest
如果还需要修改配置，可以 attach 到容器内部，进行修改。
例如修改 relative URL https://docs.gitlab.com/omnibus/settings/configuration.html#enable-relative-url-in-gitlab1
2
3
4
5
6
7
8
9
10
11docker exec -it gitlab bash
vi /etc/gitlab/gitlab.rb
# 指定host地址
external_url 'http://gitlab.feiyang.com'
# 重新加载配置文件并重启服务
gitlab-ctl reconfigure
gitlab-ctl restart
第一次登陆，一定要以 IP 地址登陆，而不是用 localhost。 否则设置 root 新密码就会遇到错误 8 errors prohibited this user from being saved
成功登陆网友以后，我创建了一个新用户，并且设置了 SSH Key，创建了一个 test repo
在这里需要的注意的是，如果没有设置域名
http://gitlab.feiyang.com那在repo 页面上 git clong 的 url 可能不对，导致无法解析并下载。
runner
本次实验，为了快速安装选择了 docker 模式。 https://docs.gitlab.com/runner/install/docker.html
1 | docker volume create gitlab-runner-config |
最好先进入容器设置一下 extra_hosts = [“gitlab.feiyang.com:192.168.64.3”] (这里后文有详细介绍 config.toml 可以搜索关键词) 否则注册 runner 的时候，只能输入 IP
下一步就是注册，前提你需要去 gitlab 网页上 CI/CD section runner 详情页上获得 token
1 | [root@centos7 jy576]# docker run --rm -it -v gitlab-runner-config:/etc/gitlab-runner gitlab/gitlab-runner:latest register |
回到网页端，我们需要修改一下 runner 设置，勾选上 Run untagged jobs
因为我们是自定义域名，所以我们需要在 gitlab-runner 里面进行设置，否则 CI job 无法 pull1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17Running with gitlab-runner 13.7.0 (943fc252)
on centos7 FSX9yRiU
Preparing the "docker" executor
00:04
Using Docker executor with image python:3.9.1-slim-buster ...
Pulling docker image python:3.9.1-slim-buster ...
Using docker image sha256:b55839ea7a0e9bb534237d00558cb96dce4013bf7f1092966fe0e27e98f8179f for python:3.9.1-slim-buster with digest python@sha256:4d92968b26bb6b7b62d957244de86fc1054f03793577d49e85c00864eb03ca07 ...
Preparing environment
00:01
Running on runner-fsx9yriu-project-2-concurrent-0 via 926cd5798468...
Getting source from Git repository
00:00
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/feiyang/test/.git/
Created fresh repository.
fatal: unable to access 'http://gitlab.feiyang.com/feiyang/test.git/': Could not resolve host: gitlab.feiyang.com
ERROR: Job failed: exit code 1
在这里，我们需要在 runner 里面定义好 domain gitlab.feiyang.com
参考文档： https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersdocker-section
extra_hosts: Specify hosts that should be defined in container environment1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28[root@centos7 jy576]# docker exec -it gitlab-runner bash
root@926cd5798468:/# cat /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "centos7"
url = "http://192.168.64.3/"
token = "FSX9yRiUGxok94hMPYdt"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "python:3.9.1-slim-buster"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
extra_hosts = ["gitlab.feiyang.com:192.168.64.3"]
Test CI
gitlab 和 runner 安装完成以后，我们就可以进行测试
Job artifacts
文档：https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#defining-artifacts-in-gitlab-ciyml
test repo 目录结构如下
├── .gitlab-ci.yml
├── lint.py
└── README.md
lint.py1
2
3
4import os
os.chdir(os.path.dirname(__file__))
print("feiyang test CI")
print(os.getcwd())
.gitlab-ci.yml1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31stages:
- lint
- build
- deploy
image: python:3.9.1-slim-buster
check_path:
stage: lint
before_script:
- pwd
script:
- cd /builds/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME && python lint.py
build_ami:
stage: build
before_script:
- apt update && apt install -y curl
script:
- echo $CI_JOB_ID >> ip.json && curl ipinfo.io >> ip.json
artifacts:
expire_in: 1 year
paths:
- ip.json
read_artifacts:
stage: deploy
before_script:
- pwd
script:
- cat ip.json
CI 结果
lint
1 | Running with gitlab-runner 13.7.0 (943fc252) |
build
1 | Running with gitlab-runner 13.7.0 (943fc252) |
deploy
1 | Running with gitlab-runner 13.7.0 (943fc252) |
总结 artifacts，前一个 stage 生存的文件，以 artifacts 保存下来，给下一个 stage 使用
CI/CD
Services
- 先看 docker:latest 和 docker:dind 镜像区别
- 官网文档 Use Docker to build Docker images
- CI/CD services 一句话，service 就是 CI 步骤里面需要的额外服务，比如 MySQL， redis， 或者 docker engine，不懂看完这一节，保证你会明白
- Passing CI/CD variables to services 这个是非常重要的， 因为 CI 的 variables 是不能自动传入到 services 的容器环境
dind
dind docker in docker, 在 docker 里面运行 docker，套娃。 我在工作上遇到了，用 gitlab CI docker build image， 我们先看下面的一段 ci.yml1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39# ------------- sleep 方便检测容器 -----------------
build_docker:
stage: build
image: docker:latest
variables:
DOCKER_HOST: tcp://docker:2375
# This instructs Docker not to start over TLS.
DOCKER_TLS_CERTDIR: ""
before_script:
- apk add --no-cache curl jq python3 py3-pip git
- head -1 /proc/self/cgroup|cut -d/ -f3
- docker info
- env
services:
- docker:dind
script:
- head -1 /proc/self/cgroup|cut -d/ -f3
- env
- docker build -t sre-turtle .
- sleep 300
# ------------- 测试 docker login 和 push ----------------
build_docker:
stage: build
image: docker:latest
variables:
DOCKER_HOST: tcp://docker:2375
# This instructs Docker not to start over TLS.
DOCKER_TLS_CERTDIR: ""
before_script:
- echo $DOCKERPASS | docker login --username $DOCKERUSER --password-stdin
services:
- docker:dind
script:
- export VERSION=${CI_JOB_ID}_$(echo $CI_COMMIT_SHA | head -c 8)
- docker build -t sre-turtle .
- docker tag sre-turtle:latest $DOCKERUSER/turtle:$VERSION
- docker push $DOCKERUSER/turtle:$VERSION
上面这一段代码的意思：
- 启动一个容器1 docker:latest 开始进行 CI
- 同时启动一个 service 运行在容器2 docker:dind 里面，service 生命周期仅限于 build_docker 这一步
- 当容器1 运行 docker build 的时候，容器1 会自动调用容器2 的 docker engine 。 打一个比方， 容器1 好比你电脑的命令行，容器2 好比你电脑上运行的 docker，当我们在电脑上运行 docker build ， 实际上也是去调用 电脑上运行的 docker 来 build image
- 补充知识 两容器之间的连接
- 测试 docker login 和 push 的成果 https://hub.docker.com/r/feiyang233/turtle
下图就很清楚的解释了 gitlab-runner 在运行上面这段代码时，容器的整体情况，会创建 2 个容器，一个负责 CI script， 一个负责 docker engine
为了更有说服力， 我们也亲自 SSH 到机器上面去验证
这里 docker:latest 运行所有的 script
这里 docker:dind 运行 docker engine ， 对外提供服务端口 2375
总结图，可以看到 build image 是存放在 docker engine, 在 CI shell 运行 docker images 可以调用 API 检查
build image
如果在构建镜像的过程中，需要访问一个私有的仓库 How to securely git clone/pip install a private repository into my docker image?
好消息！ docker 官网已经有了答案 Using SSH to access private data in builds。但是官网的文档真的不够详细，还是要参考这一细节 BuildKit 镜像构建。看一眼 Docker 一篇文章带你理解Dockerfile
安全的办法是采用 ssh key 访问:
- Generate an SSH key pair
OpenSSH 6.5 introduced ED25519 SSH keys in 2014 and they should be available on most operating systems.ED25519 有坑！见文末 - Don’t leak your Docker image’s build secrets
- Differences between deploy keys and deploy tokens
- 用 file type 存 key 到 CI/CD variable 里面
如下图所示，当我们在构建镜像的时候，实际上是 Dockerfile 指定的那一个镜像(图中是 python的镜像 )去执行 git clone 或者 pip install, 为了不让 SSH key 存在于我们的镜像中， 确保安全第一，我们必须用新版 docker 提供的 BuildKit， 下一代的镜像构建组件。
- 在容器1 运行一个 eval $(ssh-agent)， CI 容器1 中有 SSH key (gitlab deploy key)
- 容器1 通过 docker –ssh default=$SSH_AUTH_SOCK 配置好 allow the Docker Engine to forward SSH agent connections.
- 在 Dockerfile 构建的步骤里，哪一步需要 SSH 代理转发， 就挂载 ssh， RUN –mount=type=ssh <执行命令>
完整的实验代码1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17# syntax=docker/dockerfile:1.2
FROM python:3.9.1-slim-buster
WORKDIR /app
#COPY requirements.txt requirements.txt
COPY . .
RUN apt update && apt install -y openssh-client git dnsutils
RUN mkdir -p -m 0700 ~/.ssh && echo "[192.168.64.3]:2222 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAs5kvb4yuBB1C5yJfyav7F13MCP5Z4BJpCYq403tz2G/eyUMzJUzW+/FIIlLAffnLZxqkLpKZterXD2Ahn9KvA=" >> ~/.ssh/known_hosts
# RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.feiyang.com >> ~/.ssh/known_hosts 但是我自己自定义的域名，无法解析
RUN nslookup gitlab.feiyang.com
RUN --mount=type=ssh git clone ssh://git@192.168.64.3:2222/feiyang/test.git
RUN ls && env && apt-get purge -y --auto-remove git
CMD [ "python3", "-m" , "flask", "run", "--host=0.0.0.0"]
1 | build_docker: |
最后的结果 成品镜像
可以运行如下的命令去检测 SSH key 是否存在于,新构建的镜像中1
2
3docker pull feiyang233/turtle:177_0f142bbc
docker run -it feiyang233/turtle:177_0f142bbc bash
ls ~/.ssh/
bonus for sai
Maybe you have a question, why we need 2 docker containers, 1 stage task container, 1 service docker engine
The answer is
- It’s easier and faster to use an existing image and run it as an additional container than to install docker engine
- if you use dind image, please remember gitlab CI docker CMD entrypoint is
/bin/sh, because CI job you need run script on shell
But we also can try to test using only 1 container to build docker image:
- because gitlab start this container by
/bin/sh - we need manually run docker engine in backend and wait it startup
- we also need change the deafult env of DOCKER_HOST
I also manually to test, see the terminal logs:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35root@ubuntu20:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ad978e3c9250 dc8c389414c8 "dockerd-entrypoint.…" 16 seconds ago Up 15 seconds 2375-2376/tcp runner-supzrn4b-project-2-concurrent-0-ef2ee4c879673f88-build-2
dea3a81e5209 gitlab/gitlab-runner:latest "/usr/bin/dumb-init …" 4 months ago Up 46 minutes gitlab-runner
root@ubuntu20:~# docker exec -it ad978e3c9250 sh
/ # ps -ef
PID USER TIME COMMAND
1 root 0:00 /bin/sh
17 root 0:00 /bin/sh
27 root 0:00 sleep 120
28 root 0:00 sh
34 root 0:00 ps -ef
/ # env | grep DOCKER_HOST
CI_COMMIT_TITLE=export DOCKER_HOST=tcp://127.0.0.1:2375
CI_COMMIT_MESSAGE=export DOCKER_HOST=tcp://127.0.0.1:2375
/ # export DOCKER_HOST=tcp://127.0.0.1:2375
/ # env | grep DOCKER_HOST
CI_COMMIT_TITLE=export DOCKER_HOST=tcp://127.0.0.1:2375
DOCKER_HOST=tcp://127.0.0.1:2375
CI_COMMIT_MESSAGE=export DOCKER_HOST=tcp://127.0.0.1:2375
/ # dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2375
INFO[2021-06-02T15:20:31.852225191Z] Starting up
WARN[2021-06-02T15:20:31.853254325Z] could not change group /var/run/docker.sock to docker: group docker not found
WARN[2021-06-02T15:20:31.853700852Z] Binding to IP address without --tlsverify is insecure and gives root access on this machine to everyone who has access to your network. host="tcp://0.0.0.0:2375"
WARN[2021-06-02T15:20:31.853771630Z] Binding to an IP address, even on localhost, can also give access to scripts run in a browser. Be safe out there! host="tcp://0.0.0.0:2375"
WARN[2021-06-02T15:20:32.857263870Z] Binding to an IP address without --tlsverify is deprecated. Startup is intentionally being slowed down to show this message host="tcp://0.0.0.0:2375"
WARN[2021-06-02T15:20:32.857418830Z] Please consider generating tls certificates with client validation to prevent exposing unauthenticated root access to your network host="tcp://0.0.0.0:2375"
WARN[2021-06-02T15:20:32.857466269Z] You can override this by explicitly specifying '--tls=false' or '--tlsverify=false' host="tcp://0.0.0.0:2375"
WARN[2021-06-02T15:20:32.857660544Z] Support for listening on TCP without authentication or explicit intent to run without authentication will be removed in the next release host="tcp://0.0.0.0:2375"
INFO[2021-06-02T15:20:47.860746924Z] libcontainerd: started new containerd process pid=48
INFO[2021-06-02T15:20:47.860803013Z] parsed scheme: "unix" module=grpc
INFO[2021-06-02T15:20:47.860813924Z] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2021-06-02T15:20:47.860830282Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>} module=grpc
INFO[2021-06-02T15:20:47.860835997Z] ClientConn switching balancer to "pick_first" module=grpc
INFO[2021-06-02T15:20:47.877826134Z] starting containerd revision=05f951a3781f4f2c1911b05e61c160e9c30eaa8e version=v1.4.4
Finally, share with you the gitlab ci yaml file
1 | before_script: |
| gitlab docker in docker | pros | cons |
|---|---|---|
| 2 containers | official recommend Automatically wait docker engine startup No need setup env parallel start 2 docker is faster |
Use one more docker |
| 1 container | one docker is enough | slower than 2 dockers need more setup: env unsure how long to wait docker engin startup |
注意事项
- gitlab 自定义 SSH 端口
- gitlab container registry authenticate
设置 CI/CD 变量的时候, 上传 deploy key, 类型选择 file, 切记多留最后一行空白，否则会报错
1
Error loading key "/root/.ssh/id_rsa": invalid format
临时文件会在 runner 容器的 /build/user/project.temp/ 下面。 比如说
/builds/feiyang/test.tmp/SSH_TESTError loading key "/root/.ssh/id_ed25519": invalid formatdeploy key 的类型，最好是 RSA (at least 2048-bit key size), 新的 ED25519 (preferred) 在 docker build 会报错- Host key verification failed. 两个容器都需要添加
ssh-keyscan gitlab.feiyang.com >> ~/.ssh/known_hosts - Installing ssh-keyscan on Alpine linux? 这里安装包 openssh-client 就足够了， 如果安装 openssh， 就重很多， 自带了 server 功能，不推荐.
- Why eval the output of ssh-agent?