feiyang's blog

费洋的博客

0%

vault k8s integration

Posted on Edited on In Disqus:

本文记录了 vault 与 kubernetes 集成实验

参考文档

实验环境

  • vault cluster version: v1.7.2
  • kubernetes cluster version: v1.21.1

kubernetes config

Below is an example configuration of a working k8s auth. In this below example:

  • vault-auth is the service account with which k8s auth is configured.
  • test-cloud is the service account used to login to vault using the k8s auth method
  • test is the namespace of the service account test-cloud
  • role-tokenreview-binding is the name of the cluster role binding the service accounts (vault-auth and test-cloud) need to be associated with

这里是准备了 2 个 service account， 一个 vault-auth 是专门用来给 vault 回调 call k8s 验证来自 k8s 的request 的真实性。另外一个 test-cloud 是我们服务运行时， 用这个 test-cloud 的 secrets 向 vault 发起请求。 role-tokenreview-binding 是一个权限增加配置， 运行两个 service account 去 call rbac.authorization.k8s.io 验证身份。
这里我的理解是，有点像 AWS 的 role， 互相可以 assume ，验证 role 的身份。如果想更简单，就一个 service account 也是可以的。
这里让我联想到了 AWS STS, 每次当 AWS call vault ，vault 都需要验证来访的真实性。

1
2
3
4
5
6
7
8
9
10
11
12
13
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1639532537842",
      "Action": [
        "sts:GetCallerIdentity"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

以下命令，都在 kubernetes 那台机器上执行。 我已经把返回值加#注释

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
kubectl create namespace test

# namespace/test created

cat <<EOF | kubectl create -f -
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-cloud
  namespace: test
---
apiVersion: v1
kind: Secret
metadata:
 name: test-cloud
 namespace: test
 annotations:
   kubernetes.io/service-account.name: test-cloud
type: kubernetes.io/service-account-token
EOF

# serviceaccount/test-cloud created
# secret/test-cloud created

cat <<EOF | kubectl create -f -
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: default
  - kind: ServiceAccount
    name: test-cloud
    namespace: test
EOF

# serviceaccount/vault-auth created
# secret/vault-auth created
# clusterrolebinding.rbac.authorization.k8s.io/role-tokenreview-binding created

接下来是得到一些 config 用来给 vault 设置 kubernetes auth integration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
TOKEN_REVIEW_JWT=$(kubectl get secret vault-auth -o go-template='{{ .data.token }}' | base64 --decode) 
echo  $TOKEN_REVIEW_JWT
eyJhbGciOiJSUzI1NiIsImtpZCI6IlZNU3F3QURqYThpVndNdXQ0Nmlsc19LanFDM1Z4NzRwVXNxSTY0X3M5VW8ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LWF1dGgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidmF1bHQtYXV0aCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjAxNWUzNjU4LWM4MjEtNDQzNi1iMTcwLWNmZTUwMGRmZDMwZCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OnZhdWx0LWF1dGgifQ.R86ZHaSIYK4evpUhT9ayQp5XgJS-Lj_GIsIRT2n1M3Q8cFiO8A3HKx44VI0XToPHKpkyrRvzAX3n-GodR2L7AEcsesr1vQ-qlb2qreb07uh9ZxMo8t8r9bJ91eRaoV-xygW6bu3_dJupKDFL7peuCJSB16VYkPoSONQA_v1gZdXK7hmPe-JSrKtyz31FpsoI5mEhqxCwMTON_7Ew41S2wN089lMR--Rvn1ZtEaLRy1fpzEvJmwiUrN28byGrszZXMnmHTA26UUAHs59fKo-My_jM7EpV029aJGZEytX8czlndwiijPCRnhiuSb7oCJNI8yFLXyTr7LVbe6q7ip6Qqw
echo  $TOKEN_REVIEW_JWT >> TOKEN_REVIEW_JWT

KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
echo $KUBE_CA_CERT
-----BEGIN CERTIFICATE----- MIIC5zCCAc+gAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl cm5ldGVzMB4XDTIxMDUxMzAxMzAxOFoXDTMxMDUxMTAxMzAxOFowFTETMBEGA1UE AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKjq 3U72Z95kACf7ZbWYN2ktCxQVzcgJiVsD9UP2vm9c0hsI+eQWxodEf3+T6XVMCJce YgrYpLFyM1Mv7W/ZJ5F6UAGaPG4jOrF4CUgnYRV3j39wRjuvguOaf3lSMVFkAkOF GAN2eEO+OVaIeH7IxIbQ+vzC72V3WNTD0RC3OQoJIIQ2nLebcxkbksSTpDgARg+O j34IbHyUAUw3+msaCsKtqF+Tg6cLWGOu+iNekfL27rGJotXD3NCInE2aFX9Ipbmd BjnsvauwcCuBDKkSxxr46kcd7NZ2SFcvokkkuGl/c8AVdlL0TeDOrl8AOz/eUiQj 6ihQ7s07SUdNU3MefIkCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB /wQFMAMBAf8wHQYDVR0OBBYEFFIG5EyHEyWO9GE2Gm1D7nRY7iXFMA0GCSqGSIb3 DQEBCwUAA4IBAQCFcFHMgDAQ3GMfrLX0EudpyeDqJ/+Mj6C5xHet/bxQopMyCbmC ZVVXx2jpRaMSo81U/eDE0hfUSD3JGVUeM6aajivsGrGjqjuF0t/5hUxPJT74lIbc 5OQNXyYYuPeRyRw7Cnx8YpdJh4NJvWI+LBWSOSaDyFjf5yMpWQvT3bfWV4PGvi/0 YHxglWM3Jfpn7JBF6OJPs4P+6E0FPmS88d5ry6znmrAg9/PCOefZ1r93EAIwP1P4 EhWsyF9xemSyTIeDA740X7ZFcCEugkS6mOoswWtXPsL0JURlkx0kRR5HAYcQ01N1 Cv0epL7mgWJpxcwnil9pWXVDRgWTXvFP5aaD -----END CERTIFICATE-----
echo $KUBE_CA_CERT >> KUBE_CA_CERT

KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')
echo $KUBE_HOST
https://192.168.1.119:6443
echo $KUBE_HOST >> KUBE_HOST

kubectl get secret  test-cloud -n test -o go-template='{{ .data.token }}' | base64 --decode >> test-cloud
cat test-cloud
eyJhbGciOiJSUzI1NiIsImtpZCI6IlZNU3F3QURqYThpVndNdXQ0Nmlsc19LanFDM1Z4NzRwVXNxSTY0X3M5VW8ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJ0ZXN0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InRlc3QtY2xvdWQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidGVzdC1jbG91ZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjlkNDE0NjNkLWEyNTctNGRjNy1iZGQwLTVlYmRhODliOGE5NiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDp0ZXN0OnRlc3QtY2xvdWQifQ.MaPfB0w-rj9ymn1nr7Y1IJPfTM1-HpCjRhRF-qqJ8qFbREVeYgMwqg6FCYTL_bH9thLW4oNSFTw80IN8H9-0D-TVlA6UsRQGbp4v6tvb4YZ_rPJ7DRC5iaHproGPPJ0Nr-1cPLbfJVfWZwqvrrkz_pNmF23qM9WliGNGHgxV0n58KOD06lb03wk-IVBAyTKBwRTKgA3yub7SzoR0HeCS_4fvB4T73vGeNUBoaS2kIC1MF7Hlunmr0rzkGv7zs6HseFSXnGg8wZ1jUqOrP0AKuOXpE8eUXoqxd3-t0Y9zAj4pjZMCsJJZQlEVrKxAhuyeTvh_SYy5SCxjIKhpjT8j8A

vault config

这里我们需要把上面从 kubernetes 得到的 TOKEN_REVIEW_JWT KUBE_CA_CERT KUBE_HOST test-cloud 拷贝到 vault 这台机器上

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
vault auth enable kubernetes

# Success! Enabled kubernetes auth method at: kubernetes/

# 这一步很容易出错，因为 cert 的格式不对， 后面我会讲
vault write auth/kubernetes/config   token_reviewer_jwt="$TOKEN_REVIEW_JWT"   kubernetes_host="$KUBE_HOST"   kubernetes_ca_cert="$KUBE_CA_CERT" disable_local_ca_jwt="true”
# Success! Data written to: auth/kubernetes/config

# 这里 kubernetes_ca_cert 格式不对
vault read auth/kubernetes/config
Key                       Value
---                       -----
disable_iss_validation    true
disable_local_ca_jwt      true
issuer                    n/a
kubernetes_ca_cert        -----BEGIN CERTIFICATE----- MIIC5zCCAc+gAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl cm5ldGVzMB4XDTIxMDUxMzAxMzAxOFoXDTMxMDUxMTAxMzAxOFowFTETMBEGA1UE AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKjq 3U72Z95kACf7ZbWYN2ktCxQVzcgJiVsD9UP2vm9c0hsI+eQWxodEf3+T6XVMCJce YgrYpLFyM1Mv7W/ZJ5F6UAGaPG4jOrF4CUgnYRV3j39wRjuvguOaf3lSMVFkAkOF GAN2eEO+OVaIeH7IxIbQ+vzC72V3WNTD0RC3OQoJIIQ2nLebcxkbksSTpDgARg+O j34IbHyUAUw3+msaCsKtqF+Tg6cLWGOu+iNekfL27rGJotXD3NCInE2aFX9Ipbmd BjnsvauwcCuBDKkSxxr46kcd7NZ2SFcvokkkuGl/c8AVdlL0TeDOrl8AOz/eUiQj 6ihQ7s07SUdNU3MefIkCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB /wQFMAMBAf8wHQYDVR0OBBYEFFIG5EyHEyWO9GE2Gm1D7nRY7iXFMA0GCSqGSIb3 DQEBCwUAA4IBAQCFcFHMgDAQ3GMfrLX0EudpyeDqJ/+Mj6C5xHet/bxQopMyCbmC ZVVXx2jpRaMSo81U/eDE0hfUSD3JGVUeM6aajivsGrGjqjuF0t/5hUxPJT74lIbc 5OQNXyYYuPeRyRw7Cnx8YpdJh4NJvWI+LBWSOSaDyFjf5yMpWQvT3bfWV4PGvi/0 YHxglWM3Jfpn7JBF6OJPs4P+6E0FPmS88d5ry6znmrAg9/PCOefZ1r93EAIwP1P4 EhWsyF9xemSyTIeDA740X7ZFcCEugkS6mOoswWtXPsL0JURlkx0kRR5HAYcQ01N1 Cv0epL7mgWJpxcwnil9pWXVDRgWTXvFP5aaD -----END CERTIFICATE-----
kubernetes_host           https://192.168.1.119:6443
pem_keys                  []

vault policy write devwebapp - <<EOF
path "secret/data/devwebapp/config" {
  capabilities = ["read”]
}
EOF

# Success! Uploaded policy: devwebapp

vault write auth/kubernetes/role/devweb-app \
  bound_service_account_names=test-cloud \
  bound_service_account_namespaces=test \
  policies=devwebapp \
  ttl=24h
# Success! Data written to: auth/kubernetes/role/devweb-app

最后就是测试 login 了， 在 vault 那台机器上

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
root@ubuntu18-108:~# curl     --request POST   \
--data '{"jwt": "eyJhbGciOiJSUzI1NiIsImtpZCI6IlZNU3F3QURqYThpVndNdXQ0Nmlsc19LanFDM1Z4NzRwVXNxSTY0X3M5VW8ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJ0ZXN0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InRlc3QtY2xvdWQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidGVzdC1jbG91ZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjlkNDE0NjNkLWEyNTctNGRjNy1iZGQwLTVlYmRhODliOGE5NiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDp0ZXN0OnRlc3QtY2xvdWQifQ.MaPfB0w-rj9ymn1nr7Y1IJPfTM1-HpCjRhRF-qqJ8qFbREVeYgMwqg6FCYTL_bH9thLW4oNSFTw80IN8H9-0D-TVlA6UsRQGbp4v6tvb4YZ_rPJ7DRC5iaHproGPPJ0Nr-1cPLbfJVfWZwqvrrkz_pNmF23qM9WliGNGHgxV0n58KOD06lb03wk-IVBAyTKBwRTKgA3yub7SzoR0HeCS_4fvB4T73vGeNUBoaS2kIC1MF7Hlunmr0rzkGv7zs6HseFSXnGg8wZ1jUqOrP0AKuOXpE8eUXoqxd3-t0Y9zAj4pjZMCsJJZQlEVrKxAhuyeTvh_SYy5SCxjIKhpjT8j8A", "role": "devweb-app"}'  \
http://192.168.64.3:8200/v1/auth/kubernetes/login |jq

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1574  100   653  100   921  19787  27909 --:--:-- --:--:-- --:--:-- 47696
{
  "request_id": "2b9978ef-4010-cb5a-38b5-01fdfb4ea5d5",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": null,
  "wrap_info": null,
  "warnings": null,
  "auth": {
    "client_token": "s.ElCkHSzi20xOlZj0CfoDuGyv",
    "accessor": "mCzG5MoxhCUSVdSNRNfEhdBS",
    "policies": [
      "default",
      "devwebapp"
    ],
    "token_policies": [
      "default",
      "devwebapp"
    ],
    "metadata": {
      "role": "devweb-app",
      "service_account_name": "test-cloud",
      "service_account_namespace": "test",
      "service_account_secret_name": "test-cloud",
      "service_account_uid": "9d41463d-a257-4dc7-bdd0-5ebda89b8a96"
    },
    "lease_duration": 86400,
    "renewable": true,
    "entity_id": "7ab08e2f-f956-d2c6-c13e-3ec06e4a2cf3",
    "token_type": "service",
    "orphan": true
  }
}

可以看到测试成功，结果返回了我们需要的 vault client_token ，kubernetes 用这个 token 去读取存在 vault 里面的 credentials

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
root@ubuntu18-108:~# curl --header "X-Vault-Token: s.ElCkHSzi20xOlZj0CfoDuGyv" http://192.168.64.3:8200/v1/secret/devwebapp/config | jq

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   205  100   205    0     0   200k      0 --:--:-- --:--:-- --:--:--  200k
{
  "request_id": "c465525a-4af5-50d2-6a66-67ffb8a48629",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 2764800,
  "data": {
    "password": "salsa",
    "username": "giraffe"
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

踩过的坑

  • cert 格式不对
    这个时候， login 返回error 是 {“errors”:[“permission denied”]}，实际上可能是证书格式不对

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    vault read auth/kubernetes/config

    Key                       Value
    ---                       -----
    disable_iss_validation    true
    disable_local_ca_jwt      true
    issuer                    n/a
    kubernetes_ca_cert        -----BEGIN CERTIFICATE----- MIIC5zCCAc+gAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl cm5ldGVzMB4XDTIxMDUxMzAxMzAxOFoXDTMxMDUxMTAxMzAxOFowFTETMBEGA1UE AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKjq 3U72Z95kACf7ZbWYN2ktCxQVzcgJiVsD9UP2vm9c0hsI+eQWxodEf3+T6XVMCJce YgrYpLFyM1Mv7W/ZJ5F6UAGaPG4jOrF4CUgnYRV3j39wRjuvguOaf3lSMVFkAkOF GAN2eEO+OVaIeH7IxIbQ+vzC72V3WNTD0RC3OQoJIIQ2nLebcxkbksSTpDgARg+O j34IbHyUAUw3+msaCsKtqF+Tg6cLWGOu+iNekfL27rGJotXD3NCInE2aFX9Ipbmd BjnsvauwcCuBDKkSxxr46kcd7NZ2SFcvokkkuGl/c8AVdlL0TeDOrl8AOz/eUiQj 6ihQ7s07SUdNU3MefIkCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB /wQFMAMBAf8wHQYDVR0OBBYEFFIG5EyHEyWO9GE2Gm1D7nRY7iXFMA0GCSqGSIb3 DQEBCwUAA4IBAQCFcFHMgDAQ3GMfrLX0EudpyeDqJ/+Mj6C5xHet/bxQopMyCbmC ZVVXx2jpRaMSo81U/eDE0hfUSD3JGVUeM6aajivsGrGjqjuF0t/5hUxPJT74lIbc 5OQNXyYYuPeRyRw7Cnx8YpdJh4NJvWI+LBWSOSaDyFjf5yMpWQvT3bfWV4PGvi/0 YHxglWM3Jfpn7JBF6OJPs4P+6E0FPmS88d5ry6znmrAg9/PCOefZ1r93EAIwP1P4 EhWsyF9xemSyTIeDA740X7ZFcCEugkS6mOoswWtXPsL0JURlkx0kRR5HAYcQ01N1 Cv0epL7mgWJpxcwnil9pWXVDRgWTXvFP5aaD -----END CERTIFICATE-----
    kubernetes_host           https://192.168.1.119:6443
    pem_keys                  []

    # 证书应该是多行
    kubectl config view --raw --minify codeatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --dec
    -----BEGIN CERTIFICATE-----
    MIIC5zCCAc+gAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
    cm5ldGVzMB4XDTIxMDUxMzAxMzAxOFoXDTMxMDUxMTAxMzAxOFowFTETMBEGA1UE
    AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKjq
    3U72Z95kACf7ZbWYN2ktCxQVzcgJiVsD9UP2vm9c0hsI+eQWxodEf3+T6XVMCJce
    YgrYpLFyM1Mv7W/ZJ5F6UAGaPG4jOrF4CUgnYRV3j39wRjuvguOaf3lSMVFkAkOF
    GAN2eEO+OVaIeH7IxIbQ+vzC72V3WNTD0RC3OQoJIIQ2nLebcxkbksSTpDgARg+O
    j34IbHyUAUw3+msaCsKtqF+Tg6cLWGOu+iNekfL27rGJotXD3NCInE2aFX9Ipbmd
    BjnsvauwcCuBDKkSxxr46kcd7NZ2SFcvokkkuGl/c8AVdlL0TeDOrl8AOz/eUiQj
    6ihQ7s07SUdNU3MefIkCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
    /wQFMAMBAf8wHQYDVR0OBBYEFFIG5EyHEyWO9GE2Gm1D7nRY7iXFMA0GCSqGSIb3
    DQEBCwUAA4IBAQCFcFHMgDAQ3GMfrLX0EudpyeDqJ/+Mj6C5xHet/bxQopMyCbmC
    ZVVXx2jpRaMSo81U/eDE0hfUSD3JGVUeM6aajivsGrGjqjuF0t/5hUxPJT74lIbc
    5OQNXyYYuPeRyRw7Cnx8YpdJh4NJvWI+LBWSOSaDyFjf5yMpWQvT3bfWV4PGvi/0
    YHxglWM3Jfpn7JBF6OJPs4P+6E0FPmS88d5ry6znmrAg9/PCOefZ1r93EAIwP1P4
    EhWsyF9xemSyTIeDA740X7ZFcCEugkS6mOoswWtXPsL0JURlkx0kRR5HAYcQ01N1
    Cv0epL7mgWJpxcwnil9pWXVDRgWTXvFP5aaD
    -----END CERTIFICATE-----

    # 重新配置后
    vault read auth/kubernetes/config

    Key                       Value
    ---                       -----
    disable_iss_validation    true
    disable_local_ca_jwt      true
    issuer                    n/a
    kubernetes_ca_cert        -----BEGIN CERTIFICATE-----
    MIIC5zCCAc+gAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
    cm5ldGVzMB4XDTIxMDUxMzAxMzAxOFoXDTMxMDUxMTAxMzAxOFowFTETMBEGA1UE
    AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKjq
    3U72Z95kACf7ZbWYN2ktCxQVzcgJiVsD9UP2vm9c0hsI+eQWxodEf3+T6XVMCJce
    YgrYpLFyM1Mv7W/ZJ5F6UAGaPG4jOrF4CUgnYRV3j39wRjuvguOaf3lSMVFkAkOF
    GAN2eEO+OVaIeH7IxIbQ+vzC72V3WNTD0RC3OQoJIIQ2nLebcxkbksSTpDgARg+O
    j34IbHyUAUw3+msaCsKtqF+Tg6cLWGOu+iNekfL27rGJotXD3NCInE2aFX9Ipbmd
    BjnsvauwcCuBDKkSxxr46kcd7NZ2SFcvokkkuGl/c8AVdlL0TeDOrl8AOz/eUiQj
    6ihQ7s07SUdNU3MefIkCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
    /wQFMAMBAf8wHQYDVR0OBBYEFFIG5EyHEyWO9GE2Gm1D7nRY7iXFMA0GCSqGSIb3
    DQEBCwUAA4IBAQCFcFHMgDAQ3GMfrLX0EudpyeDqJ/+Mj6C5xHet/bxQopMyCbmC
    ZVVXx2jpRaMSo81U/eDE0hfUSD3JGVUeM6aajivsGrGjqjuF0t/5hUxPJT74lIbc
    5OQNXyYYuPeRyRw7Cnx8YpdJh4NJvWI+LBWSOSaDyFjf5yMpWQvT3bfWV4PGvi/0
    YHxglWM3Jfpn7JBF6OJPs4P+6E0FPmS88d5ry6znmrAg9/PCOefZ1r93EAIwP1P4
    EhWsyF9xemSyTIeDA740X7ZFcCEugkS6mOoswWtXPsL0JURlkx0kRR5HAYcQ01N1
    Cv0epL7mgWJpxcwnil9pWXVDRgWTXvFP5aaD
    -----END CERTIFICATE-----
    kubernetes_host           https://192.168.1.119:6443
    pem_keys                  []

  • kubernetes 里面的 service account 缺少权限， 导致 {“errors”:[“permission denied”]}

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: role-tokenreview-binding
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: system:auth-delegator
    subjects:
      - kind: ServiceAccount
        name: vault-auth
        namespace: default

  • {“errors”:[“claim "iss" is invalid”]} 参考问题
    这个是因为没有配置 issuer, 如果你的 kubernetes 高于 1.21.1 你需要设置 issuer 在 vault auth with kubernetes 的时候， 如何得到 issuer 地址，请参考这个
    Discovering the service account issuer