实验环境
这三台虚拟机，都运行在我的台式机上。
| Name | Operating system | CPU | RAM | Disk | IP |
|---|---|---|---|---|---|
| node-1 | Ubuntu 18.04.5 LTS | 4 | 8G | 50G | 192.168.64.3 |
| node-2 | CentOS 7.8.2003 | 2 | 2G | 20G | 192.168.64.4 |
| node-3 | Ubuntu 20.04.1 LTS | 2 | 4G | 10G | 192.168.64.5 |
实验环境的搭建，可以参考我的另外一篇博客 https://feiyang233.club/post/vm/
安装过程
参考文档：
- https://computingforgeeks.com/install-and-configure-vault-server-linux/
- https://learn.hashicorp.com/tutorials/vault/getting-started-install?in=vault/getting-started
- https://learn.hashicorp.com/tutorials/vault/raft-storage
分别在三台虚拟机上安装好 vault service1
2
3
4
5
6
7
8
9# ubuntu OS
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install vault
root@ubuntu20:/etc/vault.d# vault --version
Vault v1.6.2 (be65a227ef2e80f8588b3b13584b5c0d9238c1d7)
就下来就是配置 config, 如果没有创建存储的文件夹，还需要创建1
2mkdir -p /vault/data
chown -R vault:vault /vault
配置文件如下1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61// node-1
root@ubuntu18-108:/etc/vault.d# cat /etc/vault.d/vault.hcl
storage "raft" {
path = "./vault/data"
node_id = "node1"
}
listener "tcp" {
address = "192.168.64.3:8200"
cluster_address = "192.168.64.3:8201"
tls_disable = "true"
}
disable_mlock = true
api_addr = "http://192.168.64.3:8200"
cluster_addr = "http://192.168.64.3:8201"
//----------------------------------
//node2
[root@centos7 vault.d]# cat /etc/vault.d/vault.hcl
storage "raft" {
path = "./vault/data"
node_id = "node2"
}
listener "tcp" {
address = "192.168.64.4:8200"
tls_disable = "true"
cluster_address = "192.168.64.4:8201"
}
disable_mlock = true
api_addr = "http://192.168.64.4:8200"
cluster_addr = "http://192.168.64.4:8201"
// ----------------------------------
// node-3
root@ubuntu20:/etc/vault.d# cat /etc/vault.d/vault.hcl
storage "raft" {
path = "./vault/data"
node_id = "node3"
retry_join {
leader_api_addr = "http://192.168.64.3:8200"
}
retry_join {
leader_api_addr = "http://192.168.64.4:8200"
}
}
listener "tcp" {
address = "192.168.64.5:8200"
cluster_address = "192.168.64.5:8201"
tls_disable = "true"
}
api_addr = "http://192.168.64.5:8200"
cluster_addr = "http://192.168.64.5:8201"
init
配置完成后就是启动 vault， 并且初始化
1 | systemctl start vault |
unseal
这个时候只是初始化启动了 vault， 还需要我们用 key unseal to reconstruct the master key
node-1
1 | root@ubuntu18-108:/etc/vault.d# vault operator unseal |
这个时候 vault 才算是真正的启动，开始提供服务了，我们可以来验证一下。
1 | # login |
node-2
follower 节点稍微多一步，需要先加入 clustervault operator raft join http://192.168.64.3:8200
为了方便，避免重复，可以采用 shell 脚本1
2
3
4
5
6
7
8
9
10
11vim init.sh
chmod 755 init.sh
export VAULT_ADDR='http://192.168.64.4:8200'
vault operator raft join http://192.168.64.3:8200
vault operator unseal key1
vault operator unseal key3
vault operator unseal key5
vault login token-for-login
#-----------------
bash init.sh
node-3
1 | export VAULT_ADDR='http://192.168.64.5:8200' \ |
unseal-script
1 | feiyang-vault() { |
Enable Audit Logging
1 | mkdir /var/log/vault |
Rekeying
如果需要重新生成 unseal key 的话，可以用 rekey 命令1
2
3
4
5
6
7
8
9
10
11
12vault operator rekey -init -key-shares=3 -key-threshold=2 \
-format=json | jq -r ".nonce" > nonce.txt
# 前提是第一次 init 的结果存在 key.txt
vault operator rekey -nonce=$(cat nonce.txt) \
$(grep 'Key 1:' key.txt | awk '{print $NF}')
vault operator rekey -nonce=$(cat nonce.txt) \
$(grep 'Key 2:' key.txt | awk '{print $NF}')
vault operator rekey -nonce=$(cat nonce.txt) \
$(grep 'Key 3:' key.txt | awk '{print $NF}')
Rotating
Rotating the Encryption Key 就非常的简单了，只需要一行1
vault operator rotate
nginx 代理转发
因为我们的 vault 集群是监听在内网上的 192.168.64.0/24，而我家里局域网的网段是 192.168.0.0/24，因此要设置代理转发一下。
- 首先安装 Nginx
1
2
3
4sudo apt update
sudo apt install nginx
# 我已经关闭了防火墙的 - 设置代理转发，配置文件如下
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37vim /etc/nginx/conf.d/vault.conf
upstream vault {
server 192.168.64.3:8200;
keepalive 15;
}
server {
listen 192.168.1.108:8200;
server_name 192.168.1.108;
access_log /var/log/nginx/vault.access.log;
error_log /var/log/nginx/vault.error.log;
location / {
proxy_http_version 1.1;
proxy_buffer_size 64k;
proxy_buffers 32 32k;
proxy_busy_buffers_size 128k;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";
proxy_pass http://vault;
}
}
systemctl start nginx
systemctl enable nginx
# 记得删除 default 的 80 服务，以免和其他服务冲突，这里我这台机器 80 端口已经被 gitlab 占用了
rm -f /etc/nginx/sites-enabled/default
现在我们就可以登录 vault UI 界面了 http://192.168.1.108:8200/ui/vault/auth?with=token
enable nginx failed
我们通过 systemctl enable nginx 已经设置了开机自启动 Nginx， 但是实际环境中缺是失败了。我们检查日志发现了错误
1 | root@ubuntu18-108:/home/feiyang# systemctl status nginx |
从日志中，我们可以看出 Nginx 失败的原因是， 没有找到那个 IP 192.168.1.108 , 因为我们这个是双网卡的虚拟机，默认的 primary network interface 是192.168.64.3 , secondary network interface 是 192.168.1.108 , 在此我推测是因为当 Nginx 启动的时候， 第二张网卡还没有准备好。
这个时候，我们可以采用 systemd: Unit dependencies and order 来定义 service 之间的依赖项，启动相关顺序。我们先检查一下 Nginx 的 systemd config1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30root@ubuntu18-108:~# cat /lib/systemd/system/nginx.service
# Stop dance for nginx
# =======================
#
# ExecStop sends SIGSTOP (graceful stop) to the nginx process.
# If, after 5s (--retry QUIT/5) nginx is still running, systemd takes control
# and sends SIGTERM (fast shutdown) to the main process.
# After another 5s (TimeoutStopSec=5), and if nginx is alive, systemd sends
# SIGKILL to all the remaining processes in the process group (KillMode=mixed).
#
# nginx signals reference doc:
# http://nginx.org/en/docs/control.html
#
[Unit]
Description=A high performance web server and a reverse proxy server
Documentation=man:nginx(8)
After=network.target # 启动依赖，问题就在这里
[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload
ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid
TimeoutStopSec=5
KillMode=mixed
[Install]
WantedBy=multi-user.target
从这一行看出， Nginx 是在 After=network.target 之后启动的，但结果却是失败。我们对比了 docker service 的 systemd config 发现 docker 是在 After=network-online.target 后启动的， 我们尝试将 Nginx systemd 配置文件修改为 network-online.target ，终于可以成功开机自启动。
Auth Methods
认证方式有很多种。 https://www.vaultproject.io/docs/auth
Each auth method serves a different purpose, and some auth methods are better suited for machine authentication rather than used by human users.
machine authmethods include AppRole,Cloud-based authmethods, tokens, TLS, Kubernetes, and Radius.human authmethods include Okta, LDAP, GitHub, OIDC, and userpass.
userpass
创建一个 user 登录，这样就不需要每次都是 root token 了1
2
3
4
5
6
7
8
9
10
11
12
13
14# 本地编辑一个 sudo policy，保存
vim sudo.hcl
path "*" {
policy = "sudo"
}
# 创建一个名为 sudo 的 policy
vault policy write sudo sudo.hcl
# 创建一个 user
vault auth enable userpass
vault write auth/userpass/users/feiyang password="password" policies="sudo"
这样就可以用 feiyang 和 password 登录了
AppRole
AppRole 多用于机器上1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30# 上面 userpass 我们已经创建了 sudo policy
# 创建一个 my-role
vault write auth/approle/role/my-role \
secret_id_ttl=10m \
token_ttl=20m \
token_max_ttl=30m \
token_policies=["sudo"] \
period=0 \
bind_secret_id=true
# 获取 role list
vault list /auth/approle/role
Keys
----
my-role
# 获取 role id
vault read auth/approle/role/my-role/role-id
Key Value
--- -----
role_id 8511d740-3e3f-76d4-6084-df02b149e6fb
# 获取 role secret
vault write -f auth/approle/role/my-role/secret-id
Key Value
--- -----
secret_id 3ea8ee03-9239-7c64-ca24-2194d7a10b8d
secret_id_accessor 11d7ddcf-08ea-4e5d-921b-59d9608119ef
secret_id_ttl 10m
到此，我们有了 role_id 和 secret_id 就可以用于机器上访问 vault 了
HTTP API
首先为了操作方便，我们需要保持 token 到环境变量 $VAULT_TOKEN1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58export VAULT_TOKEN="s.Ga5jyNq6kNfRMVQk2LY1j9iu"
# HTTP API enable approle
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--request POST \
--data '{"type": "approle"}' \
http://127.0.0.1:8200/v1/sys/auth/approle
# create my-policy
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--request PUT \
--data '{"policy":"# Dev servers have version 2 of KV secrets engine mounted by default, so will\n# need these paths to grant permissions:\npath \"secret/data/*\" {\n capabilities = [\"create\", \"update\"]\n}\n\npath \"secret/data/foo\" {\n capabilities = [\"read\"]\n}\n"}' \
http://127.0.0.1:8200/v1/sys/policies/acl/my-policy
# enable KV v2 secrets engine at secret/ using API
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--request POST \
--data '{ "type":"kv-v2" }' \
http://127.0.0.1:8200/v1/sys/mounts/secret
# create my-role
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--request POST \
--data '{"policies": ["my-policy"]}' \
http://127.0.0.1:8200/v1/auth/approle/role/my-role
# get role-id
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
http://127.0.0.1:8200/v1/auth/approle/role/my-role/role-id | jq -r ".data"
# get secret-id
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--request POST \
http://127.0.0.1:8200/v1/auth/approle/role/my-role/secret-id | jq -r ".data"
# auth with the role-id and secret-id
curl --request POST \
--data '{"role_id": "3c301960-8a02-d776-f025-c3443d513a18", "secret_id": "22d1e0d6-a70b-f91f-f918-a0ee8902666b"}' \
http://127.0.0.1:8200/v1/auth/approle/login | jq -r ".auth"
# set new token from my-role auth
export VAULT_TOKEN="s.p5NB4dTlsPiUU94RA5IfbzXv"
# Create a version 1 of secret named creds with a key password and its value set to my-long-password.
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--request POST \
--data '{ "data": {"password": "my-long-password"} }' \
http://127.0.0.1:8200/v1/secret/data/creds | jq -r ".data"
# You can stop the server and unset the VAULT_TOKEN environment variable.
unset VAULT_TOKEN
CLI
Vault Commands CLI 除了 API, CLI 也是我常用的
- 自动补全 Autocompletion
1
vault -autocomplete-install
注意事项
- 每次重启服务，都需要 unseal
- 如果需要重新初始化，需要先删除 raft 存储的数据，
rm -rf /vault/data/* - enable nginx failed 请看上面 3.1 章节
vault 考试认证
HashiCorp Certified: Vault Associate
我买了这个练习题 HashiCorp Certified: Vault Associate Practice Exam 2021 ，还是不错的。 虽然不是原题，但对考试，还是非常有帮助的。
最终通过了考试 HashiCorp Certified: Vault Associate
vault-agent
安装都是一样的，Install Vault , 只是启动的时候用 agent 模式
我们需要做的就是，创建 vault-agent.service 以便于 systemd 托管控制1
2
3
4
5
6
7
8
9
10
11
12
13
14
15root@ubuntu18-119:/etc/vault.d# cat /lib/systemd/system/vault-agent.service
[Unit]
Description=Vault agent
Requires=network-online.target
[Service]
Environment=VAULT_ADDR=http://192.168.1.108:8200
Restart=on-failure
ExecStart=/usr/bin/vault agent -log-level=info -config=/etc/vault.d/vault-agent.conf
User=vault
Group=vault
[Install]
WantedBy=multi-user.target
记得 reload 一下配置文件1
systemctl daemon-reload
还有就是创建配置文件
1 | root@ubuntu18-119:/etc/vault.d# cat vault-agent.conf |
需要创建的文件：并且保证 User=vault Group=vault 有读写权限这些文件
- /var/run/vault-agent/vault-agent.pid
- /etc/vault.d/role_id
- /etc/vault.d/secret_id
- /var/run/vault-agent/vault-agent.token
- /var/run/vault-agent/vault-agent.socket
接下来我们就可以来测试一下，1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61systemctl start vault-agent
systemctl status vault-agent
root@ubuntu18-119:/etc/vault.d# systemctl status vault-agent
● vault-agent.service - Vault agent
Loaded: loaded (/lib/systemd/system/vault-agent.service; disabled; vendor preset: enabled)
Active: active (running) since Tue 2021-07-13 11:51:08 +08; 2h 32min ago
Main PID: 51688 (vault)
Tasks: 8 (limit: 4631)
CGroup: /system.slice/vault-agent.service
└─51688 /usr/bin/vault agent -log-level=info -config=/etc/vault.d/vault-agent.conf
Jul 13 14:20:48 ubuntu18-119 vault[51688]: 2021-07-13T14:20:48.288+0800 [INFO] auth.handler: authenticating
Jul 13 14:20:48 ubuntu18-119 vault[51688]: 2021-07-13T14:20:48.294+0800 [ERROR] auth.handler: error authenticating:
Jul 13 14:20:48 ubuntu18-119 vault[51688]: error=
Jul 13 14:20:48 ubuntu18-119 vault[51688]: | Error making API request.
Jul 13 14:20:48 ubuntu18-119 vault[51688]: |
Jul 13 14:20:48 ubuntu18-119 vault[51688]: | URL: PUT http://192.168.1.108:8200/v1/auth/approle/login
Jul 13 14:20:48 ubuntu18-119 vault[51688]: | Code: 400. Errors:
Jul 13 14:20:48 ubuntu18-119 vault[51688]: |
Jul 13 14:20:48 ubuntu18-119 vault[51688]: | * invalid secret id
Jul 13 14:20:48 ubuntu18-119 vault[51688]: backoff=4m2.47s
这是 token 过期了，最大期限应该是 token_max_ttl=30m ， 所以需要重新生成
root@ubuntu18-108:~# vault write -f auth/approle/role/my-role/secret-id
Key Value
--- -----
secret_id 63859ef3-7906-7eff-0586-a4ebb97ab20c
secret_id_accessor 3125d535-b357-097e-1a8d-68370f03327c
secret_id_ttl 10m
把 secret_id 放入 /etc/vault.d/secret_id
重启 vault agent
systemctl restart vault-agent
root@ubuntu18-119:/etc/vault.d# systemctl status vault-agent
● vault-agent.service - Vault agent
Loaded: loaded (/lib/systemd/system/vault-agent.service; disabled; vendor preset: enabled)
Active: active (running) since Tue 2021-07-13 14:25:53 +08; 1s ago
Main PID: 128325 (vault)
Tasks: 7 (limit: 4631)
CGroup: /system.slice/vault-agent.service
└─128325 /usr/bin/vault agent -log-level=info -config=/etc/vault.d/vault-agent.conf
Jul 13 14:25:53 ubuntu18-119 vault[128325]: 2021-07-13T14:25:53.962+0800 [INFO] sink.file: file sink configured: path=/var/run/vault-agent/vault-agent.token mode=-rw-r-----
Jul 13 14:25:53 ubuntu18-119 vault[128325]: 2021-07-13T14:25:53.981+0800 [INFO] auth.handler: starting auth handler
Jul 13 14:25:53 ubuntu18-119 vault[128325]: 2021-07-13T14:25:53.981+0800 [INFO] auth.handler: authenticating
Jul 13 14:25:53 ubuntu18-119 vault[128325]: 2021-07-13T14:25:53.984+0800 [INFO] sink.server: starting sink server
Jul 13 14:25:53 ubuntu18-119 vault[128325]: 2021-07-13T14:25:53.984+0800 [INFO] template.server: starting template server
Jul 13 14:25:53 ubuntu18-119 vault[128325]: 2021-07-13T14:25:53.984+0800 [INFO] template.server: no templates found
Jul 13 14:25:54 ubuntu18-119 vault[128325]: 2021-07-13T14:25:54.000+0800 [INFO] auth.handler: authentication successful, sending token to sinks
Jul 13 14:25:54 ubuntu18-119 vault[128325]: 2021-07-13T14:25:54.000+0800 [INFO] auth.handler: starting renewal process
Jul 13 14:25:54 ubuntu18-119 vault[128325]: 2021-07-13T14:25:54.001+0800 [INFO] sink.file: token written: path=/var/run/vault-agent/vault-agent.token
Jul 13 14:25:54 ubuntu18-119 vault[128325]: 2021-07-13T14:25:54.008+0800 [INFO] auth.handler: renewed auth token
test
root@ubuntu18-119:/etc/vault.d# curl http://127.0.0.1:8200/v1/sys/health
{"initialized":true,"sealed":false,"standby":false,"performance_standby":false,"replication_performance_mode":"disabled","replication_dr_mode":"disabled","server_time_utc":1626158428,"version":"1.7.2","cluster_name":"vault-cluster-0618ae06","cluster_id":"feca826b-0ffd-838f-412d-0c6687feee60"}
root@ubuntu18-119:/etc/vault.d# curl http://127.0.0.1:8200/v1/sys/leader
{"ha_enabled":true,"is_self":true,"active_time":"2021-07-13T02:11:43.591625814Z","leader_address":"http://192.168.64.3:8200","leader_cluster_address":"https://192.168.64.3:8201","performance_standby":false,"performance_standby_last_remote_wal":0,"raft_committed_index":13017,"raft_applied_index":13017}