1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407<!-- omit in toc -->
# Trust custom Certificate Authority
If your AWX has to trust custom Certificate Authority, you can pass the CA certificates to AWX. This is helpful in cases:
- Use private Git repository via SSL, without ignoring SSL verification.
- Use LDAPS to authenticate users.
Refer to [the official documentation](https://github.com/ansible/awx-operator#trusting-a-custom-certificate-authority) for more information.
Note that if you want to trust custom CA for jobs (running playbooks or accessing inventory sources), a different approach is required. Refer to [Appendix: Trust custom CA for jobs](#appendix-trust-custom-ca-for-jobs).
<!-- omit in toc -->
## Table of Contents
- [Overview](#overview)
- [Prepare required CA certificates](#prepare-required-ca-certificates)
- [Modify `base/kustomization.yaml`](#modify-basekustomizationyaml)
- [Modify `base/awx.yaml`](#modify-baseawxyaml)
- [Apply configuration](#apply-configuration)
- [Troubleshooting](#troubleshooting)
- [Appendix: Trust custom CA for jobs](#appendix-trust-custom-ca-for-jobs)
- [Method 1: Use Container Group](#method-1-use-container-group)
- [Create Secret](#create-secret)
- [Create Container Group on AWX](#create-container-group-on-awx)
- [Specify Container Group](#specify-container-group)
- [Method 2: Mount host filesystem](#method-2-mount-host-filesystem)
- [Update certificate store](#update-certificate-store)
- [Modify options on AWX](#modify-options-on-awx)
- [Add environment variables to jobs](#add-environment-variables-to-jobs)
- [Method 3: Add certificate file to EE image](#method-3-add-certificate-file-to-ee-image)
- [Add files to the existing image](#add-files-to-the-existing-image)
- [Build new EE using Ansible Builder](#build-new-ee-using-ansible-builder)
## Overview
Trusting custom Certificate Authority can be achieved by following steps:
1. Creating new Secret which includes your certificates
2. Passing it to your AWX by specifying the name of the Secret as your AWX's specification
There are two kinds of certificate, one is used to trust LDAP server, and the other is used as the CA bundle.
| Fields in the specification for AWX | Keys in Secret | Containers that the certificate will be mounted | Paths that the certificate will be mounted as |
|-|-|-|-|
| `ldap_cacert_secret` | `ldap-ca.crt` | `awx-web` | `/etc/openldap/certs/ldap-ca.crt` |
| `bundle_cacert_secret` | `bundle-ca.crt` | `awx-web`, `awx-task`, and `awx-ee` | `/etc/pki/ca-trust/source/anchors/bundle-ca.crt` |
Note that the `awx-ee` container is used to run management jobs only, this is not EE which runs your playbooks. If the EE running your playbook needs certificates, refer [Appendix: Trust custom CA for jobs](#appendix-trust-custom-ca-for-jobs) on the bottom.
## Prepare required CA certificates
Place your certificates under `base` directory.
```bash
$ ls -l base
total 32
-rw-rw-r--. 1 kuro kuro 801 Feb 27 00:23 awx.yaml
-rw-rw-r--. 1 kuro kuro 1339 Feb 27 00:44 cacert.pem 👈👈👈
-rw-rw-r--. 1 kuro kuro 610 Feb 27 00:23 kustomization.yaml
...
```
Note that **your certificates have to have PEM format**. You can check the format of the certificates depending on which of the following commands succeeds.
```bash
# Works for PEM format
openssl x509 -in cacert.crt -text
# Works for DER format
openssl x509 -in cacert.crt -inform DER -text
# Works for PKCS #7 format
openssl pkcs7 -in cacert.crt -text
# Works for PKCS #12 format
openssl pkcs12 -in cacert.crt -info
```
If your certificate doesn't have PEM format, you can convert it by followings:
```bash
# Convert DER to PEM
openssl x509 -in cacert.crt -inform DER -out cacert.pem -outform PEM
# Convert PKCS #7 to PEM
openssl pkcs7 -print_certs -in cacert.crt -out cacert.pem -outform PEM
# Convert PKCS #12 to PEM
openssl pkcs12 -in cacert.crt -out cacert.pem -nokeys -nodes
```
## Modify `base/kustomization.yaml`
Add following lines under `secretGenerator` in `base/kustomization.yaml`.
Note that this example provides both `ldap-ca.crt` and `bundle-ca.crt`, but you can remove unnecessary line if you don't need both of them. `ldap-ca.crt` will be used as the CA certificate for LDAP server, and `bundle-ca.crt` will be used as the CA bundle.
```yaml
...
secretGenerator:
...
- name: awx-custom-certs 👈👈👈
type: Opaque 👈👈👈
files: 👈👈👈
- ldap-ca.crt=<Name Of Your Certificate File> 👈👈👈
- bundle-ca.crt=<Name Of Your Certificate File> 👈👈👈
...
```
## Modify `base/awx.yaml`
Add following lines under `secretGenerator` in `base/kustomization.yaml`.
Note that this example provides both `ldap_cacert_secret` (should have `ldap-ca.crt`) and `bundle_cacert_secret` (should have `bundle-ca.crt`), but you can remove unnecessary line if you don't need both of them.
```yaml
...
spec:
...
ldap_cacert_secret: awx-custom-certs 👈👈👈
bundle_cacert_secret: awx-custom-certs 👈👈👈
...
```
## Apply configuration
Invoke `apply` command. This will start re-deployment of your AWX.
```base
kubectl apply -k base
```
You can monitor the progress of the re-deployment by following command:
```bash
kubectl -n awx logs -f deployments/awx-operator-controller-manager
```
## Troubleshooting
If you have problem with SSL connection such as LDAPS, you can verify your certificates inside the pod.
```bash
# Open Bash shell of the "awx-web" container
$ kubectl -n awx exec -it deployment/awx-web -c awx-web -- bash
bash-5.1$
```
First of all, you should ensure your CA certificate is mounted and has PEM format. The certificate should be be dumped as readable plain text by following command, without any error.
```bash
# The secret ldap_cacert_secret is mounted as /etc/openldap/certs/ldap-ca.crt
bash-5.1$ openssl x509 -in /etc/openldap/certs/ldap-ca.crt -text
# The secret bundle_cacert_secret is mounted as /etc/pki/ca-trust/source/anchors/bundle-ca.crt
bash-5.1$ openssl x509 -in /etc/pki/ca-trust/source/anchors/bundle-ca.crt -text
```
Note that your certificate file should contain both intermediate CA and root CA, if your server certificate is signed by intermediate CA.
```bash
# Example output of concatenated CA cert; one for intermediate CA, one for root CA
bash-5.1$ cat /etc/openldap/certs/ldap-ca.crt
-----BEGIN CERTIFICATE-----
MIIDizCCAnOgAwIBAgIUftINZYmeHvcovY0qBHp+SqZWrlswDQYJKoZIhvcNAQEL
...
3Eyhv0l7mJw/86twDMFFax+cKOCRFV6NoPOpzK1mzAXmxth6vk8DeRm0ipVpQVQ=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDizCCAnOgAwIBAgIUftINZYmeHvcovY0qBHp+SqZWrlwwDQYJKoZIhvcNAQEL
...
lVsDxZfbZVpRGkDr8odNurNmz0Xcttr+ZVRkoTy5KUxqIZhQuS6ySJj7yoLawWY=
-----END CERTIFICATE-----
```
Now you can test SSL connection.
```bash
# This is an example to test connection to LDAP server over SSL using /etc/openldap/certs/ldap-ca.crt
bash-5.1$ echo | openssl s_client -connect ldap.example.com:636 -no-CAfile -CAfile /etc/openldap/certs/ldap-ca.crt
CONNECTED(00000003)
depth=2 C = JP, ST = Example State, O = EXAMPLE.COM, CN = rca.example.com
verify return:1
depth=1 C = JP, ST = Example State, O = EXAMPLE.COM, CN = ica.example.com
verify return:1
depth=0 C = JP, ST = Example State, O = EXAMPLE.COM, CN = ldap.example.com
verify return:1
---
Certificate chain 👈👈👈 Ensure that the full certificate chain is recognized
0 s:C = JP, ST = Example State, O = EXAMPLE.COM, CN = ldap.example.com
i:C = JP, ST = Example State, O = EXAMPLE.COM, CN = ica.example.com
...
1 s:C = JP, ST = Example State, O = EXAMPLE.COM, CN = ica.example.com
i:C = JP, ST = Example State, O = EXAMPLE.COM, CN = rca.example.com
...
2 s:C = JP, ST = Example State, O = EXAMPLE.COM, CN = rca.example.com
i:C = JP, ST = Example State, O = EXAMPLE.COM, CN = rca.example.com
...
---
...
---
SSL handshake has read 3210 bytes and written 413 bytes
Verification: OK 👈👈👈 Ensure there is no verification error
---
...
SSL-Session:
...
Verify return code: 0 (ok) 👈👈👈 Ensure there is no verification error
...
```
## Appendix: Trust custom CA for jobs
The `bundle_cacert_secret` stated above does not add any certificates to automation job pods, so it can't be used to trust custom CA to run jobs e.g. running playbooks, accessing inventory sources.
To trust custom CA for your jobs, there are several solutions depend on your modules or plugins that used in your jobs.
- Update system certificate store with custom CA certificate on automation job pods.
- Place custom CA certificates on automation job pods and pass its path through environment variable or parameters for modules or plugins.
This guide provides some ways as examples to achieve above solutions by invoking `update-ca-trust` and adding environment variable `REQUESTS_CA_BUNDLE` for `requests` python module (since `requests` does not refer system certificate store, custom CA certificates should be passed through this environment variable).
<!-- no toc -->
- [Method 1: Use Container Group](#method-1-use-container-group)
- Add required certificates and environment variable on the automation job pods by mounting it using [Container Group](../containergroup) with custom pod specification.
- [Method 2: Mount host filesystem](#method-2-mount-host-filesystem)
- Add required certificates and environment variable on the automation job pods by mounting host filesystem.
- [Method 3: Add certificate file to EE image](#method-3-add-certificate-file-to-ee-image)
- Add required certificates and environment variable on the automation job pods by adding it to the EE image.
In this example, the value of the environment variable `REQUESTS_CA_BUNDLE` is specified as `/etc/pki/tls/cert.pem`. This file is a symbolic link to `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` that updated by `update-ca-trust` by combining and includes all CA certificates in the system certificate store into one file.
### Method 1: Use Container Group
This method can be used for standard jobs, such as the case that some modules in your playbook require custom CA certificates. In addition, this method can also be used to access inventory sources.
#### Create Secret
Prepare CA certificate file with PEM format (refer to [Prepare required CA certificates](#prepare-required-ca-certificates)) first, then create Secret with the certificate.
```bash
kubectl -n awx create secret generic awx-custom-job-certs \
--from-file=job-ca.crt=cacert.pem
```
#### Create Container Group on AWX
Create new Container Group by `Administration` > `Instance Group` > `Add` > `Add container group` on AWX, then enable `Customize pod specification` and put following specification.
This specification supports both jobs that refers system certificate store and the store for `requests`.
```yaml
apiVersion: v1
kind: Pod
metadata:
namespace: awx
spec:
serviceAccountName: default
automountServiceAccountToken: false
initContainers:
- name: init
image: quay.io/ansible/awx-ee:latest
command:
- /bin/sh
- -c
- |
mkdir -p /etc/pki/ca-trust/extracted/{java,pem,openssl,edk2}
update-ca-trust
volumeMounts:
- name: ca-trust-extracted
mountPath: /etc/pki/ca-trust/extracted
- name: awx-custom-job-certs
mountPath: /etc/pki/ca-trust/source/anchors/job-ca.crt
subPath: job-ca.crt
readOnly: true
containers:
- image: quay.io/ansible/awx-ee:latest
name: worker
args:
- ansible-runner
- worker
- '--private-data-dir=/runner'
env:
- name: REQUESTS_CA_BUNDLE
value: /etc/pki/tls/cert.pem
resources:
requests:
cpu: 250m
memory: 100Mi
volumeMounts:
- name: ca-trust-extracted
mountPath: /etc/pki/ca-trust/extracted
- name: awx-custom-job-certs
mountPath: /etc/pki/ca-trust/source/anchors/job-ca.crt
subPath: job-ca.crt
readOnly: true
volumes:
- name: ca-trust-extracted
emptyDir: {}
- name: awx-custom-job-certs
secret:
secretName: awx-custom-job-certs
items:
- key: job-ca.crt
path: job-ca.crt
```
This specification has following changes from the default one:
- Init container (`initContainers`) is added to invoke `update-ca-trust` to update certificate store using CA certificate that mounted through Secret resource.
- Worker container mounts certificate store that updated by init container (`volumeMounts`).
- Worker container has environment variable `REQUESTS_CA_BUNDLE` (`env`) that has path for updated certificate file.
- Volumes (`volumes`) are added to achieve above changes.
If you just want to place CA certificates somewhere on automation job pods and `update-ca-trust` is not required for you, you can ignore `initContainers` and `ca-trust-extracted` (under `volumes` and `volumeMounts`), and change `mountPath` for `awx-custom-job-certs`.
#### Specify Container Group
The Container Group can be specified as `Instance Groups` for the Job Template or Organization.
If you want to use this Container Group to access inventory sources, specify it as `Instance Groups` for the Inventory or Organization.
### Method 2: Mount host filesystem
This method can be used for both standard jobs and inventory sync.
Note that this method changes the global configuration of your K3s host and AWX, so whole system and other applications on your K3s host, and all automation job pods running on the AWX are also affected.
#### Update certificate store
Prepare CA certificate file with PEM format (refer to [Prepare required CA certificates](#prepare-required-ca-certificates)) first, then place the certificate file under `/etc/pki/ca-trust/source/anchors` on your K3s host, and invoke `update-ca-trust`.
```bash
$ ls -l /etc/pki/ca-trust/source/anchors
total 4
-rw-r--r--. 1 root root 1489 Apr 27 22:49 cacert.pem
$ sudo update-ca-trust
```
If your Kubernetes cluster has multiple nodes, this step have to be invoked on all nodes where your automation job pods will be launched on.
If you just want to place CA certificates somewhere on automation job pods and `update-ca-trust` is not required for you, you can ignore `update-ca-trust`.
#### Modify options on AWX
On `Settings` > `Jobs settings` > `Edit` page, modify following two options.
- Enable `Expose host paths for Container Groups`.
- Ensure `"/etc/pki/ca-trust:/etc/pki/ca-trust:O"` is specified in `Paths to expose to isolated jobs`.
#### Add environment variables to jobs
To add environment variables to your jobs, on `Settings` > `Jobs settings` > `Edit` page, modify `Extra Environment Variables` with following value. If the environment variable is not required for you, this step can be ignored.
```json
{
"REQUESTS_CA_BUNDLE": "/etc/pki/tls/cert.pem"
}
```
### Method 3: Add certificate file to EE image
This method can be used for both standard jobs and inventory sync, and can be achieved by following two ways.
- [Add files to the existing image](#add-files-to-the-existing-image)
- [Build new EE using Ansible Builder](#build-new-ee-using-ansible-builder)
#### Add files to the existing image
Prepare CA certificate file with PEM format (refer to [Prepare required CA certificates](#prepare-required-ca-certificates)), then you can add the certificate to existing image using following Dockerfile.
This example shows the way to add local certificate file (`cacert.pem`) and environment variable to `quay.io/ansible/awx-ee:latest`.
```dockerfile
FROM quay.io/ansible/awx-ee:latest
USER root
COPY cacert.pem /etc/pki/ca-trust/source/anchors/cacert.pem
RUN update-ca-trust
ENV REQUESTS_CA_BUNDLE=/etc/pki/tls/cert.pem
USER 1000
```
Build and push image to container registry, then define new EE on AWX, and specify it for your jobs.
#### Build new EE using Ansible Builder
Prepare CA certificate file with PEM format (refer to [Prepare required CA certificates](#prepare-required-ca-certificates)), then you can add `additional_build_files` and `additional_build_steps.append_final` to your `execution-environment.yml` to add the certificate and environment variable to your EE.
```yaml
...
additional_build_files:
- src: certs/cacert.pem
dest: certs
additional_build_steps:
append_final:
- COPY _build/certs/cacert.pem /etc/pki/ca-trust/source/anchors/cacert.pem
- RUN update-ca-trust
- ENV REQUESTS_CA_BUNDLE=/etc/pki/tls/cert.pem
```
Build and push image to container registry, then define new EE on AWX, and specify it for your jobs.