vault 实践

在本地虚拟机环境搭建 vault cluster 实践。

实验环境

这三台虚拟机,都运行在我的台式机上。

Name Operating system CPU RAM Disk IP
node-1 Ubuntu 18.04.5 LTS 4 8G 50G 192.168.64.3
node-2 CentOS 7.8.2003 2 2G 20G 192.168.64.4
node-3 Ubuntu 20.04.1 LTS 2 4G 10G 192.168.64.5

实验环境的搭建,可以参考我的另外一篇博客 https://feiyang233.club/post/vm/

安装过程

参考文档:

分别在三台虚拟机上安装好 vault service

1
2
3
4
5
6
7
8
9
# ubuntu OS
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -

sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"

sudo apt-get update && sudo apt-get install vault

root@ubuntu20:/etc/vault.d# vault --version
Vault v1.6.2 (be65a227ef2e80f8588b3b13584b5c0d9238c1d7)

就下来就是配置 config, 如果没有创建存储的文件夹,还需要创建
1
2
mkdir -p /vault/data
chown -R vault:vault /vault

配置文件如下
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
// node-1
root@ubuntu18-108:/etc/vault.d# cat /etc/vault.d/vault.hcl
storage "raft" {
path = "./vault/data"
node_id = "node1"
}

listener "tcp" {
address = "192.168.64.3:8200"
cluster_address = "192.168.64.3:8201"
tls_disable = "true"
}

disable_mlock = true
api_addr = "http://192.168.64.3:8200"
cluster_addr = "http://192.168.64.3:8201"

//----------------------------------

//node2
[root@centos7 vault.d]# cat /etc/vault.d/vault.hcl
storage "raft" {
path = "./vault/data"
node_id = "node2"
}

listener "tcp" {
address = "192.168.64.4:8200"
tls_disable = "true"
cluster_address = "192.168.64.4:8201"
}

disable_mlock = true
api_addr = "http://192.168.64.4:8200"
cluster_addr = "http://192.168.64.4:8201"

// ----------------------------------

// node-3
root@ubuntu20:/etc/vault.d# cat /etc/vault.d/vault.hcl
storage "raft" {
path = "./vault/data"
node_id = "node3"

retry_join {
leader_api_addr = "http://192.168.64.3:8200"
}

retry_join {
leader_api_addr = "http://192.168.64.4:8200"
}
}

listener "tcp" {
address = "192.168.64.5:8200"
cluster_address = "192.168.64.5:8201"
tls_disable = "true"
}

api_addr = "http://192.168.64.5:8200"
cluster_addr = "http://192.168.64.5:8201"

init

配置完成后就是启动 vault, 并且初始化

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
systemctl  start vault

# 选一台作为 master 开始初始化,这里我选的 node-1
# export 设置 API 地址非常重要
export VAULT_ADDR='http://192.168.64.3:8200'

root@ubuntu18-108:/etc/vault.d# vault operator init
Unseal Key 1: key1
Unseal Key 2: key2
Unseal Key 3: key3
Unseal Key 4: key4
Unseal Key 5: key5

Initial Root Token: token

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
# 以上的初始化产生的 key 和 token 非常重要,一定要妥善安全保管

# 如果你嫌测试环境 太多 key 麻烦,可以修改参数,产生一个 key
vault operator init -key-shares=1 -key-threshold=1

-key-shares=<int>
Number of key shares to split the generated master key into. This is
the number of "unseal keys" to generate. This is aliased as "-n". The
default is 5.

-key-threshold=<int>
Number of key shares required to reconstruct the master key. This must
be less than or equal to -key-shares. This is aliased as "-t". The
default is 3.

unseal

这个时候只是初始化启动了 vault, 还需要我们用 key unseal to reconstruct the master key

node-1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
root@ubuntu18-108:/etc/vault.d# vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce 8950ac41-69ae-2a96-00db-610739dcbeee
Version 1.7.1
Storage Type raft
HA Enabled true


root@ubuntu18-108:/etc/vault.d# vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce 8950ac41-69ae-2a96-00db-610739dcbeee
Version 1.7.1
Storage Type raft
HA Enabled true


root@ubuntu18-108:/etc/vault.d# vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.7.1
Storage Type raft
Cluster Name vault-cluster-0618ae06
Cluster ID feca826b-0ffd-838f-412d-0c6687feee60
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address <none>
Raft Committed Index 24
Raft Applied Index 24

这个时候 vault 才算是真正的启动,开始提供服务了,我们可以来验证一下。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# login
root@ubuntu18-108:/etc/vault.d# vault login token-for-login
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key Value
--- -----
token token-for-login
token_accessor xxxxxxxxxxx
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]

# check nodes, 我这里设置好之后,如果此时你只安装好 node-1 列表只有一个
root@ubuntu18-108:/etc/vault.d# vault operator raft list-peers
Node Address State Voter
---- ------- ----- -----
node1 192.168.64.3:8201 leader true
node2 192.168.64.4:8201 follower false
node3 192.168.64.5:8201 follower false

node-2

follower 节点稍微多一步,需要先加入 cluster
vault operator raft join http://192.168.64.3:8200

为了方便,避免重复,可以采用 shell 脚本

1
2
3
4
5
6
7
8
9
10
11
vim init.sh
chmod 755 init.sh
export VAULT_ADDR='http://192.168.64.4:8200'
vault operator raft join http://192.168.64.3:8200
vault operator unseal key1
vault operator unseal key3
vault operator unseal key5
vault login token-for-login

#-----------------
bash init.sh

node-3

1
2
3
4
5
6
export VAULT_ADDR='http://192.168.64.5:8200' \
&& vault operator raft join http://192.168.64.3:8200 \
&& vault operator unseal key1 \
&& vault operator unseal key3 \
&& vault operator unseal key5 \
vault login token-for-login

Enable Audit Logging

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
mkdir /var/log/vault

chown -R vault:vault /var/log/vault

vault audit enable file file_path=/var/log/vault/vault-audit.log
Success! Enabled the file audit device at: file/

# check the log
root@ubuntu18-108:/var/log/vault# cat /var/log/vault/vault-audit.log | jq
{
"time": "2021-06-10T12:55:56.251461725Z",
"type": "request",
"auth": {
"token_type": "default"
},
"request": {
"id": "facaae9e-b62a-0a5e-e486-3737252ca003",
"operation": "update",
"namespace": {
"id": "root"
},
"path": "sys/audit/test"
}
}
{
"time": "2021-06-10T12:55:56.255890481Z",
"type": "response",
"auth": {
"client_token": "hmac-sha256:4ee53ef44a84288b7e08f4cb5d3cee7a8169f953e18fe0dd5074aaac477000b8",
"accessor": "hmac-sha256:1632d2bc5c5b4bf88ed7c2fab22f9932cf31470a95f111bc22fbe34737d56322",
"display_name": "root",
"policies": [
"root"
],
"token_policies": [
"root"
],
"token_type": "service",
"token_issue_time": "2021-05-09T20:42:59+08:00"
},
"request": {
"id": "8b29ec93-b89d-feb9-56e0-47bdb7dea4f0",
"operation": "update",
"mount_type": "system",
"client_token": "hmac-sha256:4ee53ef44a84288b7e08f4cb5d3cee7a8169f953e18fe0dd5074aaac477000b8",
"client_token_accessor": "hmac-sha256:1632d2bc5c5b4bf88ed7c2fab22f9932cf31470a95f111bc22fbe34737d56322",
"namespace": {
"id": "root"
},
"path": "sys/audit/file",
"data": {
"description": "hmac-sha256:9a3dedcf661ac628346b8b4787ff2e1b84e506aa614c359cf8e6500c834aca3d",
"local": false,
"options": {
"file_path": "hmac-sha256:1a445f97febdb58c62eee3997a99dac448c7dd32674dda94f8c1366d18c530e5"
},
"type": "hmac-sha256:29023c2cba01f448e1052da9d286f10d600f2ed99aaedd46dd98122078faa77a"
},
"remote_address": "192.168.64.3"
},
"response": {
"mount_type": "system"
}
}

Rekeying

如果需要重新生成 unseal key 的话,可以用 rekey 命令

1
2
3
4
5
6
7
8
9
10
11
12
vault operator rekey -init -key-shares=3 -key-threshold=2 \
-format=json | jq -r ".nonce" > nonce.txt

# 前提是第一次 init 的结果存在 key.txt
vault operator rekey -nonce=$(cat nonce.txt) \
$(grep 'Key 1:' key.txt | awk '{print $NF}')

vault operator rekey -nonce=$(cat nonce.txt) \
$(grep 'Key 2:' key.txt | awk '{print $NF}')

vault operator rekey -nonce=$(cat nonce.txt) \
$(grep 'Key 3:' key.txt | awk '{print $NF}')

Rotating

Rotating the Encryption Key 就非常的简单了,只需要一行

1
vault operator rotate

nginx 代理转发

因为我们的 vault 集群是监听在内网上的 192.168.64.0/24,而我家里局域网的网段是 192.168.0.0/24,因此要设置代理转发一下。

  • 首先安装 Nginx
    1
    2
    3
    4
    sudo apt update
    sudo apt install nginx

    # 我已经关闭了防火墙的
  • 设置代理转发,配置文件如下
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    vim /etc/nginx/conf.d/vault.conf

    upstream vault {
    server 192.168.64.3:8200;
    keepalive 15;
    }

    server {

    listen 192.168.1.108:8200;
    server_name 192.168.1.108;
    access_log /var/log/nginx/vault.access.log;
    error_log /var/log/nginx/vault.error.log;

    location / {

    proxy_http_version 1.1;
    proxy_buffer_size 64k;
    proxy_buffers 32 32k;
    proxy_busy_buffers_size 128k;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Connection "Keep-Alive";
    proxy_set_header Proxy-Connection "Keep-Alive";

    proxy_pass http://vault;

    }
    }


    systemctl start nginx
    systemctl enable nginx

    # 记得删除 default 的 80 服务,以免和其他服务冲突,这里我这台机器 80 端口已经被 gitlab 占用了
    rm -f /etc/nginx/sites-enabled/default

现在我们就可以登录 vault UI 界面了 http://192.168.1.108:8200/ui/vault/auth?with=token

enable nginx failed

我们通过 systemctl enable nginx 已经设置了开机自启动 Nginx, 但是实际环境中缺是失败了。我们检查日志发现了错误

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
root@ubuntu18-108:/home/feiyang# systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2021-06-12 09:45:56 +08; 1min 12s ago
Docs: man:nginx(8)
Process: 1365 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)

Jun 12 09:45:55 ubuntu18-108 systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 12 09:45:56 ubuntu18-108 nginx[1365]: nginx: [emerg] bind() to 192.168.1.108:8200 failed (99: Cannot assign requested address)
Jun 12 09:45:56 ubuntu18-108 nginx[1365]: nginx: configuration file /etc/nginx/nginx.conf test failed
Jun 12 09:45:56 ubuntu18-108 systemd[1]: nginx.service: Control process exited, code=exited status=1
Jun 12 09:45:56 ubuntu18-108 systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 12 09:45:56 ubuntu18-108 systemd[1]: Failed to start A high performance web server and a reverse proxy server.

# journalctl check log
root@ubuntu18-108:/home/feiyang# journalctl -u nginx
-- Reboot --
Jun 12 08:42:37 ubuntu18-108 systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 12 08:42:37 ubuntu18-108 nginx[1348]: nginx: [emerg] bind() to 192.168.1.108:8200 failed (99: Cannot assign requested address)
Jun 12 08:42:37 ubuntu18-108 nginx[1348]: nginx: configuration file /etc/nginx/nginx.conf test failed
Jun 12 08:42:37 ubuntu18-108 systemd[1]: nginx.service: Control process exited, code=exited status=1
Jun 12 08:42:37 ubuntu18-108 systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 12 08:42:37 ubuntu18-108 systemd[1]: Failed to start A high performance web server and a reverse proxy server.

从日志中,我们可以看出 Nginx 失败的原因是, 没有找到那个 IP 192.168.1.108 , 因为我们这个是双网卡的虚拟机,默认的 primary network interface 是192.168.64.3 , secondary network interface 是 192.168.1.108 , 在此我推测是因为当 Nginx 启动的时候, 第二张网卡还没有准备好。

这个时候,我们可以采用 systemd: Unit dependencies and order 来定义 service 之间的依赖项,启动相关顺序。我们先检查一下 Nginx 的 systemd config

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
root@ubuntu18-108:~# cat /lib/systemd/system/nginx.service
# Stop dance for nginx
# =======================
#
# ExecStop sends SIGSTOP (graceful stop) to the nginx process.
# If, after 5s (--retry QUIT/5) nginx is still running, systemd takes control
# and sends SIGTERM (fast shutdown) to the main process.
# After another 5s (TimeoutStopSec=5), and if nginx is alive, systemd sends
# SIGKILL to all the remaining processes in the process group (KillMode=mixed).
#
# nginx signals reference doc:
# http://nginx.org/en/docs/control.html
#
[Unit]
Description=A high performance web server and a reverse proxy server
Documentation=man:nginx(8)
After=network.target # 启动依赖,问题就在这里

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload
ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid
TimeoutStopSec=5
KillMode=mixed

[Install]
WantedBy=multi-user.target

从这一行看出, Nginx 是在 After=network.target 之后启动的,但结果却是失败。我们对比了 docker service 的 systemd config 发现 docker 是在 After=network-online.target 后启动的, 我们尝试将 Nginx systemd 配置文件修改为 network-online.target ,终于可以成功开机自启动。

Auth Methods

认证方式有很多种。 https://www.vaultproject.io/docs/auth
Each auth method serves a different purpose, and some auth methods are better suited for machine authentication rather than used by human users.

  • machine auth methods include AppRole,
  • Cloud-based auth methods, tokens, TLS, Kubernetes, and Radius.
  • human auth methods include Okta, LDAP, GitHub, OIDC, and userpass.

userpass

创建一个 user 登录,这样就不需要每次都是 root token 了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 本地编辑一个 sudo policy,保存
vim sudo.hcl

path "*" {
policy = "sudo"
}

# 创建一个名为 sudo 的 policy
vault policy write sudo sudo.hcl

# 创建一个 user
vault auth enable userpass

vault write auth/userpass/users/feiyang password="password" policies="sudo"

这样就可以用 feiyang 和 password 登录了

AppRole

AppRole 多用于机器上

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# 上面 userpass 我们已经创建了 sudo policy

# 创建一个 my-role
vault write auth/approle/role/my-role \
secret_id_ttl=10m \
token_ttl=20m \
token_max_ttl=30m \
token_policies=["sudo"] \
period=0 \
bind_secret_id=true

# 获取 role list
vault list /auth/approle/role
Keys
----
my-role

# 获取 role id
vault read auth/approle/role/my-role/role-id
Key Value
--- -----
role_id 8511d740-3e3f-76d4-6084-df02b149e6fb

# 获取 role secret
vault write -f auth/approle/role/my-role/secret-id
Key Value
--- -----
secret_id 3ea8ee03-9239-7c64-ca24-2194d7a10b8d
secret_id_accessor 11d7ddcf-08ea-4e5d-921b-59d9608119ef
secret_id_ttl 10m

到此,我们有了 role_id 和 secret_id 就可以用于机器上访问 vault 了

HTTP API

首先为了操作方便,我们需要保持 token 到环境变量 $VAULT_TOKEN

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
export VAULT_TOKEN="s.Ga5jyNq6kNfRMVQk2LY1j9iu"

# HTTP API enable approle
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--request POST \
--data '{"type": "approle"}' \
http://127.0.0.1:8200/v1/sys/auth/approle

# create my-policy
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--request PUT \
--data '{"policy":"# Dev servers have version 2 of KV secrets engine mounted by default, so will\n# need these paths to grant permissions:\npath \"secret/data/*\" {\n capabilities = [\"create\", \"update\"]\n}\n\npath \"secret/data/foo\" {\n capabilities = [\"read\"]\n}\n"}' \
http://127.0.0.1:8200/v1/sys/policies/acl/my-policy

# enable KV v2 secrets engine at secret/ using API
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--request POST \
--data '{ "type":"kv-v2" }' \
http://127.0.0.1:8200/v1/sys/mounts/secret

# create my-role
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--request POST \
--data '{"policies": ["my-policy"]}' \
http://127.0.0.1:8200/v1/auth/approle/role/my-role

# get role-id
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
http://127.0.0.1:8200/v1/auth/approle/role/my-role/role-id | jq -r ".data"

# get secret-id
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--request POST \
http://127.0.0.1:8200/v1/auth/approle/role/my-role/secret-id | jq -r ".data"

# auth with the role-id and secret-id
curl --request POST \
--data '{"role_id": "3c301960-8a02-d776-f025-c3443d513a18", "secret_id": "22d1e0d6-a70b-f91f-f918-a0ee8902666b"}' \
http://127.0.0.1:8200/v1/auth/approle/login | jq -r ".auth"

# set new token from my-role auth
export VAULT_TOKEN="s.p5NB4dTlsPiUU94RA5IfbzXv"

# Create a version 1 of secret named creds with a key password and its value set to my-long-password.
curl \
--header "X-Vault-Token: $VAULT_TOKEN" \
--request POST \
--data '{ "data": {"password": "my-long-password"} }' \
http://127.0.0.1:8200/v1/secret/data/creds | jq -r ".data"

# You can stop the server and unset the VAULT_TOKEN environment variable.
unset VAULT_TOKEN

CLI

Vault Commands CLI 除了 API, CLI 也是我常用的

  • 自动补全 Autocompletion
    1
    vault -autocomplete-install

注意事项

  • 每次重启服务,都需要 unseal
  • 如果需要重新初始化,需要先删除 raft 存储的数据, rm -rf /vault/data/*
  • enable nginx failed 请看上面 3.1 章节

vault 考试认证

HashiCorp Certified: Vault Associate

未完待续