gitlab install and CI pipeline

为了测试 gitlab CI pipeline, 从而在家里的台式机搭建了测试环境。

为什么进行这次实验?因为自己平时只是用到了 gitlab,它对于我来说是一个黑盒。
为了搞清楚 CI job 里面的一些 function 用法,自己搭建一套环境。一是方便实验,二是自己不能在公司的环境上实验,担心造成破坏。

Update history

2021-07-13 安利学 git 的网站 Learn Git BranchingGit 命令太难学?我用一款游戏带你玩转它!
2021-05-24 终于认识到自己的膨胀了,倍感欣慰
2021-01-09 初稿,妹子什么时候才能好好学习呀

Environment

默认技能:

  • docker, 虚拟机上安装 docker 的过程就省略了
  • DNS, 因为是自定义的域名,所以需要绑定在本地 hosts

最开始 Ubuntu 机器只有 2 cpu 4GB, 安装 gitlab 非常卡顿, load 高到 40+, 增加配置后,顺利安装

Name Operating system CPU RAM Disk IP
gitlab-main Ubuntu 18.04.5 LTS 4 8G 50G 192.168.64.3
gitlab-runner CentOS 7.8.2003 2 2G 20G 192.168.64.4
gitlab-runner Ubuntu 20.04.1 LTS 2 4G 20G 192.168.64.5

Install gitlab

官方文档:https://docs.gitlab.com/

gitlab

本次实验,为了快速安装选择了 docker 模式。 https://docs.gitlab.com/omnibus/docker/README.html

本地 22 端口被 SSH 占用了,所以改为了 2222,这里需要注意的是,我用的是 社区版本 ce

1
2
3
4
5
6
7
8
9
sudo docker run --detach \
--hostname gitlab.feiyang.com \
--publish 443:443 --publish 80:80 --publish 2222:22 \
--name gitlab \
--restart always \
--volume $GITLAB_HOME/config:/etc/gitlab \
--volume $GITLAB_HOME/logs:/var/log/gitlab \
--volume $GITLAB_HOME/data:/var/opt/gitlab \
gitlab/gitlab-ce:latest

如果还需要修改配置,可以 attach 到容器内部,进行修改。

例如修改 relative URL https://docs.gitlab.com/omnibus/settings/configuration.html#enable-relative-url-in-gitlab

1
2
3
4
5
6
7
8
9
10
11
docker exec -it gitlab bash

vi /etc/gitlab/gitlab.rb


# 指定host地址
external_url 'http://gitlab.feiyang.com'

# 重新加载配置文件并重启服务
gitlab-ctl reconfigure
gitlab-ctl restart

第一次登陆,一定要以 IP 地址登陆,而不是用 localhost。 否则设置 root 新密码就会遇到错误 8 errors prohibited this user from being saved
成功登陆网友以后,我创建了一个新用户,并且设置了 SSH Key,创建了一个 test repo

在这里需要的注意的是,如果没有设置域名 http://gitlab.feiyang.com 那在repo 页面上 git clong 的 url 可能不对,导致无法解析并下载。

runner

本次实验,为了快速安装选择了 docker 模式。 https://docs.gitlab.com/runner/install/docker.html

1
2
3
4
5
6
docker volume create gitlab-runner-config

docker run -d --name gitlab-runner --restart always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v gitlab-runner-config:/etc/gitlab-runner \
gitlab/gitlab-runner:latest

最好先进入容器设置一下 extra_hosts = [“gitlab.feiyang.com:192.168.64.3”] (这里后文有详细介绍 config.toml 可以搜索关键词) 否则注册 runner 的时候,只能输入 IP
下一步就是注册,前提你需要去 gitlab 网页上 CI/CD section runner 详情页上获得 token

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
[root@centos7 jy576]# docker run --rm -it -v gitlab-runner-config:/etc/gitlab-runner gitlab/gitlab-runner:latest register
Runtime platform arch=amd64 os=linux pid=7 revision=943fc252 version=13.7.0
Running in system-mode.

Enter the GitLab instance URL (for example, https://gitlab.com/):
http://gitlab.feiyang.com/
Enter the registration token:
kXmj4UA9Z2_NL2sa2Z6z
Enter a description for the runner:
[43420abf216d]: centos7
Enter tags for the runner (comma-separated):
ci,ssh
ERROR: Registering runner... failed runner=kXmj4UA9 status=couldn't execute POST against http://gitlab.feiyang.com/api/v4/runners: Post http://gitlab.feiyang.com/api/v4/runners: dial tcp: lookup gitlab.feiyang.com on 8.8.8.8:53: no such host
PANIC: Failed to register the runner. You may be having network problems.
[root@centos7 jy576]# docker run --rm -it -v gitlab-runner-config:/etc/gitlab-runner gitlab/gitlab-runner:latest register
Runtime platform arch=amd64 os=linux pid=8 revision=943fc252 version=13.7.0
Running in system-mode.

Enter the GitLab instance URL (for example, https://gitlab.com/):
http://192.168.64.3/
Enter the registration token:
kXmj4UA9Z2_NL2sa2Z6z
Enter a description for the runner:
[a7068d34147b]: centos7
Enter tags for the runner (comma-separated):
ci,ssh
Registering runner... succeeded runner=kXmj4UA9
Enter an executor: custom, virtualbox, docker+machine, docker, docker-ssh, parallels, shell, ssh, docker-ssh+machine, kubernetes:
docker
Enter the default Docker image (for example, ruby:2.6):
python:3.9.1-slim-buster
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
[root@centos7 jy576]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
926cd5798468 gitlab/gitlab-runner:latest "/usr/bin/dumb-init …" 7 minutes ago Up 7 minutes gitlab-runner

回到网页端,我们需要修改一下 runner 设置,勾选上 Run untagged jobs

因为我们是自定义域名,所以我们需要在 gitlab-runner 里面进行设置,否则 CI job 无法 pull

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Running with gitlab-runner 13.7.0 (943fc252)
on centos7 FSX9yRiU
Preparing the "docker" executor
00:04
Using Docker executor with image python:3.9.1-slim-buster ...
Pulling docker image python:3.9.1-slim-buster ...
Using docker image sha256:b55839ea7a0e9bb534237d00558cb96dce4013bf7f1092966fe0e27e98f8179f for python:3.9.1-slim-buster with digest python@sha256:4d92968b26bb6b7b62d957244de86fc1054f03793577d49e85c00864eb03ca07 ...
Preparing environment
00:01
Running on runner-fsx9yriu-project-2-concurrent-0 via 926cd5798468...
Getting source from Git repository
00:00
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/feiyang/test/.git/
Created fresh repository.
fatal: unable to access 'http://gitlab.feiyang.com/feiyang/test.git/': Could not resolve host: gitlab.feiyang.com
ERROR: Job failed: exit code 1

在这里,我们需要在 runner 里面定义好 domain gitlab.feiyang.com
参考文档: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersdocker-section
extra_hosts: Specify hosts that should be defined in container environment
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[root@centos7 jy576]# docker exec -it gitlab-runner bash
root@926cd5798468:/# cat /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0

[session_server]
session_timeout = 1800

[[runners]]
name = "centos7"
url = "http://192.168.64.3/"
token = "FSX9yRiUGxok94hMPYdt"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "python:3.9.1-slim-buster"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
extra_hosts = ["gitlab.feiyang.com:192.168.64.3"]

Test CI

gitlab 和 runner 安装完成以后,我们就可以进行测试

Job artifacts

文档:https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#defining-artifacts-in-gitlab-ciyml

test repo 目录结构如下
├── .gitlab-ci.yml
├── lint.py
└── README.md

lint.py

1
2
3
4
import os
os.chdir(os.path.dirname(__file__))
print("feiyang test CI")
print(os.getcwd())

.gitlab-ci.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
stages:
- lint
- build
- deploy

image: python:3.9.1-slim-buster

check_path:
stage: lint
before_script:
- pwd
script:
- cd /builds/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME && python lint.py

build_ami:
stage: build
before_script:
- apt update && apt install -y curl
script:
- echo $CI_JOB_ID >> ip.json && curl ipinfo.io >> ip.json
artifacts:
expire_in: 1 year
paths:
- ip.json

read_artifacts:
stage: deploy
before_script:
- pwd
script:
- cat ip.json

CI 结果

lint

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Running with gitlab-runner 13.7.0 (943fc252)
on centos7 FSX9yRiU
Preparing the "docker" executor
00:05
Using Docker executor with image python:3.9.1-slim-buster ...
Pulling docker image python:3.9.1-slim-buster ...
Using docker image sha256:b55839ea7a0e9bb534237d00558cb96dce4013bf7f1092966fe0e27e98f8179f for python:3.9.1-slim-buster with digest python@sha256:4d92968b26bb6b7b62d957244de86fc1054f03793577d49e85c00864eb03ca07 ...
Preparing environment
00:00
Running on runner-fsx9yriu-project-2-concurrent-0 via 926cd5798468...
Getting source from Git repository
00:01
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/feiyang/test/.git/
Checking out b60b95c1 as master...
Removing ip.json
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:00
$ pwd
/builds/feiyang/test
$ cd /builds/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME && python lint.py
feiyang test CI
/builds/feiyang/test
Job succeeded

build

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
Running with gitlab-runner 13.7.0 (943fc252)
on centos7 FSX9yRiU
Preparing the "docker" executor
00:04
Using Docker executor with image python:3.9.1-slim-buster ...
Pulling docker image python:3.9.1-slim-buster ...
Using docker image sha256:b55839ea7a0e9bb534237d00558cb96dce4013bf7f1092966fe0e27e98f8179f for python:3.9.1-slim-buster with digest python@sha256:4d92968b26bb6b7b62d957244de86fc1054f03793577d49e85c00864eb03ca07 ...
Preparing environment
00:00
Running on runner-fsx9yriu-project-2-concurrent-0 via 926cd5798468...
Getting source from Git repository
00:01
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/feiyang/test/.git/
Checking out b60b95c1 as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:05
$ apt update && apt install -y curl
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [121 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [260 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [7860 B]
Fetched 8414 kB in 2s (4381 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
2 packages can be upgraded. Run 'apt list --upgradable' to see them.
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
krb5-locales libcurl4 libgssapi-krb5-2 libk5crypto3 libkeyutils1 libkrb5-3
libkrb5support0 libldap-2.4-2 libldap-common libnghttp2-14 libpsl5 librtmp1
libsasl2-2 libsasl2-modules libsasl2-modules-db libssh2-1 publicsuffix
Suggested packages:
krb5-doc krb5-user libsasl2-modules-gssapi-mit
| libsasl2-modules-gssapi-heimdal libsasl2-modules-ldap libsasl2-modules-otp
libsasl2-modules-sql
The following NEW packages will be installed:
curl krb5-locales libcurl4 libgssapi-krb5-2 libk5crypto3 libkeyutils1
libkrb5-3 libkrb5support0 libldap-2.4-2 libldap-common libnghttp2-14 libpsl5
librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db libssh2-1
publicsuffix
0 upgraded, 18 newly installed, 0 to remove and 2 not upgraded.
Need to get 2470 kB of archives.
After this operation, 5837 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 krb5-locales all 1.17-3+deb10u1 [95.4 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libkeyutils1 amd64 1.6-6 [15.0 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 libkrb5support0 amd64 1.17-3+deb10u1 [65.8 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 libk5crypto3 amd64 1.17-3+deb10u1 [122 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 libkrb5-3 amd64 1.17-3+deb10u1 [369 kB]
Get:6 http://deb.debian.org/debian buster/main amd64 libgssapi-krb5-2 amd64 1.17-3+deb10u1 [158 kB]
Get:7 http://deb.debian.org/debian buster/main amd64 libsasl2-modules-db amd64 2.1.27+dfsg-1+deb10u1 [69.1 kB]
Get:8 http://deb.debian.org/debian buster/main amd64 libsasl2-2 amd64 2.1.27+dfsg-1+deb10u1 [106 kB]
Get:9 http://deb.debian.org/debian buster/main amd64 libldap-common all 2.4.47+dfsg-3+deb10u4 [89.8 kB]
Get:10 http://deb.debian.org/debian buster/main amd64 libldap-2.4-2 amd64 2.4.47+dfsg-3+deb10u4 [224 kB]
Get:11 http://deb.debian.org/debian buster/main amd64 libnghttp2-14 amd64 1.36.0-2+deb10u1 [85.0 kB]
Get:12 http://deb.debian.org/debian buster/main amd64 libpsl5 amd64 0.20.2-2 [53.7 kB]
Get:13 http://deb.debian.org/debian buster/main amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2 [60.5 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 libssh2-1 amd64 1.8.0-2.1 [140 kB]
Get:15 http://deb.debian.org/debian buster/main amd64 libcurl4 amd64 7.64.0-4+deb10u1 [331 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 curl amd64 7.64.0-4+deb10u1 [264 kB]
Get:17 http://deb.debian.org/debian buster/main amd64 libsasl2-modules amd64 2.1.27+dfsg-1+deb10u1 [104 kB]
Get:18 http://deb.debian.org/debian buster/main amd64 publicsuffix all 20190415.1030-1 [116 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 2470 kB in 0s (5707 kB/s)
Selecting previously unselected package krb5-locales.
(Reading database ... 6829 files and directories currently installed.)
Preparing to unpack .../00-krb5-locales_1.17-3+deb10u1_all.deb ...
Unpacking krb5-locales (1.17-3+deb10u1) ...
Selecting previously unselected package libkeyutils1:amd64.
Preparing to unpack .../01-libkeyutils1_1.6-6_amd64.deb ...
Unpacking libkeyutils1:amd64 (1.6-6) ...
Selecting previously unselected package libkrb5support0:amd64.
Preparing to unpack .../02-libkrb5support0_1.17-3+deb10u1_amd64.deb ...
Unpacking libkrb5support0:amd64 (1.17-3+deb10u1) ...
Selecting previously unselected package libk5crypto3:amd64.
Preparing to unpack .../03-libk5crypto3_1.17-3+deb10u1_amd64.deb ...
Unpacking libk5crypto3:amd64 (1.17-3+deb10u1) ...
Selecting previously unselected package libkrb5-3:amd64.
Preparing to unpack .../04-libkrb5-3_1.17-3+deb10u1_amd64.deb ...
Unpacking libkrb5-3:amd64 (1.17-3+deb10u1) ...
Selecting previously unselected package libgssapi-krb5-2:amd64.
Preparing to unpack .../05-libgssapi-krb5-2_1.17-3+deb10u1_amd64.deb ...
Unpacking libgssapi-krb5-2:amd64 (1.17-3+deb10u1) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../06-libsasl2-modules-db_2.1.27+dfsg-1+deb10u1_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.27+dfsg-1+deb10u1) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../07-libsasl2-2_2.1.27+dfsg-1+deb10u1_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.27+dfsg-1+deb10u1) ...
Selecting previously unselected package libldap-common.
Preparing to unpack .../08-libldap-common_2.4.47+dfsg-3+deb10u4_all.deb ...
Unpacking libldap-common (2.4.47+dfsg-3+deb10u4) ...
Selecting previously unselected package libldap-2.4-2:amd64.
Preparing to unpack .../09-libldap-2.4-2_2.4.47+dfsg-3+deb10u4_amd64.deb ...
Unpacking libldap-2.4-2:amd64 (2.4.47+dfsg-3+deb10u4) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../10-libnghttp2-14_1.36.0-2+deb10u1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Selecting previously unselected package libpsl5:amd64.
Preparing to unpack .../11-libpsl5_0.20.2-2_amd64.deb ...
Unpacking libpsl5:amd64 (0.20.2-2) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../12-librtmp1_2.4+20151223.gitfa8646d.1-2_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Selecting previously unselected package libssh2-1:amd64.
Preparing to unpack .../13-libssh2-1_1.8.0-2.1_amd64.deb ...
Unpacking libssh2-1:amd64 (1.8.0-2.1) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../14-libcurl4_7.64.0-4+deb10u1_amd64.deb ...
Unpacking libcurl4:amd64 (7.64.0-4+deb10u1) ...
Selecting previously unselected package curl.
Preparing to unpack .../15-curl_7.64.0-4+deb10u1_amd64.deb ...
Unpacking curl (7.64.0-4+deb10u1) ...
Selecting previously unselected package libsasl2-modules:amd64.
Preparing to unpack .../16-libsasl2-modules_2.1.27+dfsg-1+deb10u1_amd64.deb ...
Unpacking libsasl2-modules:amd64 (2.1.27+dfsg-1+deb10u1) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../17-publicsuffix_20190415.1030-1_all.deb ...
Unpacking publicsuffix (20190415.1030-1) ...
Setting up libkeyutils1:amd64 (1.6-6) ...
Setting up libpsl5:amd64 (0.20.2-2) ...
Setting up libsasl2-modules:amd64 (2.1.27+dfsg-1+deb10u1) ...
Setting up libnghttp2-14:amd64 (1.36.0-2+deb10u1) ...
Setting up krb5-locales (1.17-3+deb10u1) ...
Setting up libldap-common (2.4.47+dfsg-3+deb10u4) ...
Setting up libkrb5support0:amd64 (1.17-3+deb10u1) ...
Setting up libsasl2-modules-db:amd64 (2.1.27+dfsg-1+deb10u1) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2) ...
Setting up libk5crypto3:amd64 (1.17-3+deb10u1) ...
Setting up libsasl2-2:amd64 (2.1.27+dfsg-1+deb10u1) ...
Setting up libssh2-1:amd64 (1.8.0-2.1) ...
Setting up libkrb5-3:amd64 (1.17-3+deb10u1) ...
Setting up publicsuffix (20190415.1030-1) ...
Setting up libldap-2.4-2:amd64 (2.4.47+dfsg-3+deb10u4) ...
Setting up libgssapi-krb5-2:amd64 (1.17-3+deb10u1) ...
Setting up libcurl4:amd64 (7.64.0-4+deb10u1) ...
Setting up curl (7.64.0-4+deb10u1) ...
Processing triggers for libc-bin (2.28-10) ...
$ echo $CI_JOB_ID >> ip.json && curl ipinfo.io >> ip.json
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 312 100 312 0 0 1362 0 --:--:-- --:--:-- --:--:-- 1362
Uploading artifacts for successful job
00:00
Uploading artifacts...
ip.json: found 1 matching files and directories
Uploading artifacts as "archive" to coordinator... ok id=14 responseStatus=201 Created token=AzTM9Cfs
Job succeeded

deploy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
Running with gitlab-runner 13.7.0 (943fc252)
on centos7 FSX9yRiU
Preparing the "docker" executor
00:04
Using Docker executor with image python:3.9.1-slim-buster ...
Pulling docker image python:3.9.1-slim-buster ...
Using docker image sha256:b55839ea7a0e9bb534237d00558cb96dce4013bf7f1092966fe0e27e98f8179f for python:3.9.1-slim-buster with digest python@sha256:4d92968b26bb6b7b62d957244de86fc1054f03793577d49e85c00864eb03ca07 ...
Preparing environment
00:00
Running on runner-fsx9yriu-project-2-concurrent-0 via 926cd5798468...
Getting source from Git repository
00:01
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/feiyang/test/.git/
Checking out b60b95c1 as master...
Removing ip.json
Skipping Git submodules setup
Downloading artifacts
00:00
Downloading artifacts for build_ami (14)...
Downloading artifacts from coordinator... ok id=14 responseStatus=200 OK token=AzTM9Cfs
Executing "step_script" stage of the job script
00:00
$ pwd
/builds/feiyang/test
$ cat ip.json
14
{
"ip": "x.x.x.x",
"hostname": "singnet.com.sg",
"city": "Singapore",
"region": "Singapore",
"country": "SG",
"loc": "1,103",
"org": "AS9506 Singtel Fibre Broadband",
"postal": "048508",
"timezone": "Asia/Singapore",
"readme": "https://ipinfo.io/missingauth"
}
Job succeeded

总结 artifacts,前一个 stage 生存的文件,以 artifacts 保存下来,给下一个 stage 使用

CI/CD

Services

dind

dind docker in docker, 在 docker 里面运行 docker,套娃。 我在工作上遇到了,用 gitlab CI docker build image, 我们先看下面的一段 ci.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# ------------- sleep 方便检测容器 -----------------
build_docker:
stage: build
image: docker:latest
variables:
DOCKER_HOST: tcp://docker:2375
# This instructs Docker not to start over TLS.
DOCKER_TLS_CERTDIR: ""
before_script:
- apk add --no-cache curl jq python3 py3-pip git
- head -1 /proc/self/cgroup|cut -d/ -f3
- docker info
- env
services:
- docker:dind
script:
- head -1 /proc/self/cgroup|cut -d/ -f3
- env
- docker build -t sre-turtle .
- sleep 300

# ------------- 测试 docker login 和 push ----------------

build_docker:
stage: build
image: docker:latest
variables:
DOCKER_HOST: tcp://docker:2375
# This instructs Docker not to start over TLS.
DOCKER_TLS_CERTDIR: ""
before_script:
- echo $DOCKERPASS | docker login --username $DOCKERUSER --password-stdin
services:
- docker:dind
script:
- export VERSION=${CI_JOB_ID}_$(echo $CI_COMMIT_SHA | head -c 8)
- docker build -t sre-turtle .
- docker tag sre-turtle:latest $DOCKERUSER/turtle:$VERSION
- docker push $DOCKERUSER/turtle:$VERSION

上面这一段代码的意思:

  1. 启动一个容器1 docker:latest 开始进行 CI
  2. 同时启动一个 service 运行在容器2 docker:dind 里面,service 生命周期仅限于 build_docker 这一步
  3. 当容器1 运行 docker build 的时候,容器1 会自动调用容器2 的 docker engine 。 打一个比方, 容器1 好比你电脑的命令行,容器2 好比你电脑上运行的 docker,当我们在电脑上运行 docker build , 实际上也是去调用 电脑上运行的 docker 来 build image
  4. 补充知识 两容器之间的连接
  5. 测试 docker login 和 push 的成果 https://hub.docker.com/r/feiyang233/turtle

下图就很清楚的解释了 gitlab-runner 在运行上面这段代码时,容器的整体情况,会创建 2 个容器,一个负责 CI script, 一个负责 docker engine

为了更有说服力, 我们也亲自 SSH 到机器上面去验证
这里 docker:latest 运行所有的 script

这里 docker:dind 运行 docker engine , 对外提供服务端口 2375

总结图,可以看到 build image 是存放在 docker engine, 在 CI shell 运行 docker images 可以调用 API 检查

build image

如果在构建镜像的过程中,需要访问一个私有的仓库 How to securely git clone/pip install a private repository into my docker image?
好消息! docker 官网已经有了答案 Using SSH to access private data in builds。但是官网的文档真的不够详细,还是要参考这一细节 BuildKit 镜像构建。看一眼 Docker 一篇文章带你理解Dockerfile
安全的办法是采用 ssh key 访问:

如下图所示,当我们在构建镜像的时候,实际上是 Dockerfile 指定的那一个镜像(图中是 python的镜像 )去执行 git clone 或者 pip install, 为了不让 SSH key 存在于我们的镜像中, 确保安全第一,我们必须用新版 docker 提供的 BuildKit, 下一代的镜像构建组件。

  • 在容器1 运行一个 eval $(ssh-agent), CI 容器1 中有 SSH key (gitlab deploy key)
  • 容器1 通过 docker –ssh default=$SSH_AUTH_SOCK 配置好 allow the Docker Engine to forward SSH agent connections.
  • 在 Dockerfile 构建的步骤里,哪一步需要 SSH 代理转发, 就挂载 ssh, RUN –mount=type=ssh <执行命令>

完整的实验代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# syntax=docker/dockerfile:1.2

FROM python:3.9.1-slim-buster

WORKDIR /app

#COPY requirements.txt requirements.txt
COPY . .
RUN apt update && apt install -y openssh-client git dnsutils
RUN mkdir -p -m 0700 ~/.ssh && echo "[192.168.64.3]:2222 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAs5kvb4yuBB1C5yJfyav7F13MCP5Z4BJpCYq403tz2G/eyUMzJUzW+/FIIlLAffnLZxqkLpKZterXD2Ahn9KvA=" >> ~/.ssh/known_hosts
# RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.feiyang.com >> ~/.ssh/known_hosts 但是我自己自定义的域名,无法解析
RUN nslookup gitlab.feiyang.com
RUN --mount=type=ssh git clone ssh://git@192.168.64.3:2222/feiyang/test.git

RUN ls && env && apt-get purge -y --auto-remove git

CMD [ "python3", "-m" , "flask", "run", "--host=0.0.0.0"]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
build_docker:
stage: build
image: docker:latest
variables:
DOCKER_HOST: tcp://docker:2375
# This instructs Docker not to start over TLS.
DOCKER_TLS_CERTDIR: ""
PASS_VAR: $FEIYANG
before_script:
- apk add --no-cache curl jq python3 py3-pip git bind-tools openssh-client
- echo $DOCKERPASS | docker login --username $DOCKERUSER --password-stdin
- mkdir -p -m 0700 ~/.ssh
#- 这里本来应该是 ssh-keyscan gitlab.feiyang.com >> ~/.ssh/known_hosts 但是我自己自定义的域名,无法解析
- echo "[192.168.64.3]:2222 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAs5kvb4yuBB1C5yJfyav7F13MCP5Z4BJpCYq403tz2G/eyUMzJUzW+/FIIlLAffnLZxqkLpKZterXD2Ahn9KvA=" >> ~/.ssh/known_hosts
- chmod 600 $SSH_TEST
- cat /etc/hosts
- nslookup gitlab.feiyang.com

services:
- docker:dind
script:
- export VERSION=${CI_JOB_ID}_$(echo $CI_COMMIT_SHA | head -c 8)
- export DOCKER_BUILDKIT=1
- eval $(ssh-agent)
- ssh-add $SSH_TEST
- docker build -t sre-turtle --ssh default=$SSH_AUTH_SOCK .
- docker tag sre-turtle:latest $DOCKERUSER/turtle:$VERSION
- docker push $DOCKERUSER/turtle:$VERSION

最后的结果 成品镜像

可以运行如下的命令去检测 SSH key 是否存在于,新构建的镜像中

1
2
3
docker pull feiyang233/turtle:177_0f142bbc
docker run -it feiyang233/turtle:177_0f142bbc bash
ls ~/.ssh/

bonus for sai

Maybe you have a question, why we need 2 docker containers, 1 stage task container, 1 service docker engine

The answer is

  • It’s easier and faster to use an existing image and run it as an additional container than to install docker engine
  • if you use dind image, please remember gitlab CI docker CMD entrypoint is /bin/sh, because CI job you need run script on shell

But we also can try to test using only 1 container to build docker image:

  • because gitlab start this container by /bin/sh
  • we need manually run docker engine in backend and wait it startup
  • we also need change the deafult env of DOCKER_HOST

I also manually to test, see the terminal logs:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
root@ubuntu20:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ad978e3c9250 dc8c389414c8 "dockerd-entrypoint.…" 16 seconds ago Up 15 seconds 2375-2376/tcp runner-supzrn4b-project-2-concurrent-0-ef2ee4c879673f88-build-2
dea3a81e5209 gitlab/gitlab-runner:latest "/usr/bin/dumb-init …" 4 months ago Up 46 minutes gitlab-runner
root@ubuntu20:~# docker exec -it ad978e3c9250 sh
/ # ps -ef
PID USER TIME COMMAND
1 root 0:00 /bin/sh
17 root 0:00 /bin/sh
27 root 0:00 sleep 120
28 root 0:00 sh
34 root 0:00 ps -ef
/ # env | grep DOCKER_HOST
CI_COMMIT_TITLE=export DOCKER_HOST=tcp://127.0.0.1:2375
CI_COMMIT_MESSAGE=export DOCKER_HOST=tcp://127.0.0.1:2375
/ # export DOCKER_HOST=tcp://127.0.0.1:2375
/ # env | grep DOCKER_HOST
CI_COMMIT_TITLE=export DOCKER_HOST=tcp://127.0.0.1:2375
DOCKER_HOST=tcp://127.0.0.1:2375
CI_COMMIT_MESSAGE=export DOCKER_HOST=tcp://127.0.0.1:2375
/ # dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2375
INFO[2021-06-02T15:20:31.852225191Z] Starting up
WARN[2021-06-02T15:20:31.853254325Z] could not change group /var/run/docker.sock to docker: group docker not found
WARN[2021-06-02T15:20:31.853700852Z] Binding to IP address without --tlsverify is insecure and gives root access on this machine to everyone who has access to your network. host="tcp://0.0.0.0:2375"
WARN[2021-06-02T15:20:31.853771630Z] Binding to an IP address, even on localhost, can also give access to scripts run in a browser. Be safe out there! host="tcp://0.0.0.0:2375"
WARN[2021-06-02T15:20:32.857263870Z] Binding to an IP address without --tlsverify is deprecated. Startup is intentionally being slowed down to show this message host="tcp://0.0.0.0:2375"
WARN[2021-06-02T15:20:32.857418830Z] Please consider generating tls certificates with client validation to prevent exposing unauthenticated root access to your network host="tcp://0.0.0.0:2375"
WARN[2021-06-02T15:20:32.857466269Z] You can override this by explicitly specifying '--tls=false' or '--tlsverify=false' host="tcp://0.0.0.0:2375"
WARN[2021-06-02T15:20:32.857660544Z] Support for listening on TCP without authentication or explicit intent to run without authentication will be removed in the next release host="tcp://0.0.0.0:2375"
INFO[2021-06-02T15:20:47.860746924Z] libcontainerd: started new containerd process pid=48
INFO[2021-06-02T15:20:47.860803013Z] parsed scheme: "unix" module=grpc
INFO[2021-06-02T15:20:47.860813924Z] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2021-06-02T15:20:47.860830282Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>} module=grpc
INFO[2021-06-02T15:20:47.860835997Z] ClientConn switching balancer to "pick_first" module=grpc
INFO[2021-06-02T15:20:47.877826134Z] starting containerd revision=05f951a3781f4f2c1911b05e61c160e9c30eaa8e version=v1.4.4

Finally, share with you the gitlab ci yaml file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
before_script:
- apk add --no-cache curl jq python3 py3-pip git
- mkdir -p -m 0600 ~/.ssh
- echo "[192.168.64.3]:2222 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAs5kvb4yuBB1C5yJfyav7F13MCP5Z4BJpCYq403tz2G/eyUMzJUzW+/FIIlLAffnLZxqkLpKZterXD2Ahn9KvA=" >> ~/.ssh/known_hosts
- chmod 600 $SSH_TEST
- export DOCKER_HOST=tcp://127.0.0.1:2375
- nohup dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2375 &
- cp $SSH_TEST ~/.ssh/id_rsa
- sleep 30 # it's hard to set a best vaule

script:
- export VERSION=${CI_JOB_ID}_$(echo $CI_COMMIT_SHA | head -c 8)
- export DOCKER_BUILDKIT=1
- eval $(ssh-agent)
- ssh-add $SSH_TEST
- docker build -t sre-turtle --ssh default=$SSH_AUTH_SOCK .
- docker tag sre-turtle:latest $DOCKERUSER/turtle:$VERSION
- echo $DOCKERPASS | docker login --username $DOCKERUSER --password-stdin
- docker push $DOCKERUSER/turtle:$VERSION
gitlab docker in docker pros cons
2 containers official recommend
Automatically wait docker engine startup
No need setup env
parallel start 2 docker is faster
Use one more docker
1 container one docker is enough slower than 2 dockers
need more setup: env
unsure how long to wait docker engin startup

注意事项

  • gitlab 自定义 SSH 端口
  • gitlab container registry authenticate
  • 设置 CI/CD 变量的时候, 上传 deploy key, 类型选择 file, 切记多留最后一行空白,否则会报错

    1
    Error loading key "/root/.ssh/id_rsa": invalid format

    临时文件会在 runner 容器的 /build/user/project.temp/ 下面。 比如说 /builds/feiyang/test.tmp/SSH_TEST

  • Error loading key "/root/.ssh/id_ed25519": invalid format deploy key 的类型,最好是 RSA (at least 2048-bit key size), 新的 ED25519 (preferred) 在 docker build 会报错

  • Host key verification failed. 两个容器都需要添加 ssh-keyscan gitlab.feiyang.com >> ~/.ssh/known_hosts
  • Installing ssh-keyscan on Alpine linux? 这里安装包 openssh-client 就足够了, 如果安装 openssh, 就重很多, 自带了 server 功能,不推荐.
  • Why eval the output of ssh-agent?