1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170# CI/CD for Kestra flows
This repository explains several ways of implementing a CI/CD pipeline for various kinds of workflows.
## CI/CD with GitHub Actions
This repository provides an end-to-end example of how you can use the [deploy](https://github.com/marketplace/actions/kestra-deploy-action) and [validate](https://github.com/marketplace/actions/kestra-validate-action) GitHub Actions.
Make sure that the directory structure of your flows corresponds to the structure of your namespaces.
| Directory | Namespace |
| ---------------------- | -------------- |
| ./flows/prod | prod |
| ./flows/prod.marketing | prod.marketing |
Here is a full CI/CD example using a GitHub Actions workflow:
```yaml
name: Kestra CI/CD
on:
push:
branches:
- main
jobs:
prod:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: validate-all flows
uses: kestra-io/validate-action@develop
with:
directory: ./flows/prod
resource: flow
server: ${{secrets.KESTRA_HOST}}
user: ${{secrets.KESTRA_USER}}
password: ${{secrets.KESTRA_PASSWORD}}
- name: deploy-prod
uses: kestra-io/deploy-action@develop
with:
namespace: kestra
directory: ./flows/prod
resource: flow
server: ${{secrets.KESTRA_HOST}}
user: ${{secrets.KESTRA_USER}}
password: ${{secrets.KESTRA_PASSWORD}}
delete: false
- name: deploy-prod-marketing
uses: kestra-io/deploy-action@develop
with:
namespace: kestra.marketing
directory: ./flows/prod.marketing
resource: flow
server: ${{secrets.KESTRA_HOST}}
user: ${{secrets.KESTRA_USER}}
password: ${{secrets.KESTRA_PASSWORD}}
delete: false
```
## CI/CD from a flow using a GitHub webhook trigger
Alternatively, you can use a Kestra flow that will deploy production flows based on the current state of the default branch. You can either run this flow on schedule or in response to a GitHub webhook event.
```yaml
id: ci-cd
namespace: kestra
tasks:
- id: deploy
type: io.kestra.core.tasks.flows.WorkingDirectory
tasks:
- id: cloneRepository
type: io.kestra.plugin.git.Clone
url: https://github.com/anna-geller/kestra-ci-cd
branch: main
- id: validateFlows
type: io.kestra.plugin.scripts.shell.Commands
commands:
- /app/kestra flow validate flows/
- id: deployFlows
type: io.kestra.plugin.scripts.shell.Commands
commands:
- /app/kestra flow namespace update prod flows/prod/ --no-delete
- /app/kestra flow namespace update prod.marketing flows/prod.marketing/ --no-delete
```
### CI/CD from a flow with a self-hosted remote server
```yaml
id: ci-cd
namespace: kestra
variables:
host: "http://your_host_name:8080/"
auth: "username:password"
tasks:
- id: deploy
type: io.kestra.core.tasks.flows.WorkingDirectory
tasks:
- id: cloneRepository
type: io.kestra.plugin.git.Clone
url: https://github.com/anna-geller/kestra-ci-cd
branch: main
- id: validateFlows
type: io.kestra.plugin.scripts.shell.Commands
commands:
- /app/kestra flow validate flows/ --server={{vars.host}} --user={{vars.auth}}
- id: deployFlows
type: io.kestra.plugin.scripts.shell.Commands
commands:
- /app/kestra flow namespace update prod flows/prod/ --no-delete --server={{vars.host}} --user={{vars.auth}}
- /app/kestra flow namespace update prod.marketing flows/prod.marketing/ --no-delete --server={{vars.host}} --user={{vars.auth}}
triggers:
- id: github
type: io.kestra.core.models.triggers.types.Webhook
key: "yourSecretKey"
```
### CI/CD from a flow with Kestra Enterprise
For Kestra Enterprise, make sure to change `/app/kestra` to `/app/kestra-ee`.
```yaml
id: ci-cd
namespace: kestra
variables:
host: "https://demo.kestra.io/"
auth: "cicd:{{secret('CICD_PASSWORD')}}" # cicd is a username - syntax is username:password
tasks:
- id: deploy
type: io.kestra.core.tasks.flows.WorkingDirectory
tasks:
- id: cloneRepository
type: io.kestra.plugin.git.Clone
url: https://github.com/anna-geller/kestra-ci-cd
- id: validateFlows
type: io.kestra.plugin.scripts.shell.Commands
commands:
- /app/kestra-ee flow validate flows/ --server={{vars.host}} --user={{vars.auth}}
- id: deployFlows
type: io.kestra.plugin.scripts.shell.Commands
commands:
- /app/kestra-ee flow namespace update prod flows/prod/ --no-delete --server={{vars.host}} --user={{vars.auth}}
- /app/kestra-ee flow namespace update prod.marketing flows/prod.marketing/ --no-delete --server={{vars.host}} --user={{vars.auth}}
triggers:
- id: github
type: io.kestra.core.models.triggers.types.Webhook
key: "yourSecretKey"
```
## CI/CD using Terraform
While Terraform might be more challenging to understand at first, it provides the highest degree of flexibility. Using Kestra and Terraform together, your flows can be deployed along with other infrastructure resources in your stack, making it easier to adopt Infrastructure as Code.
The [main.tf](main.tf) file provides a simple Terraform configuration that you can use to automate the deployment of flows stored in a `flows` directory.
Run the following commands from your Terminal:
```bash
terraform init # downloads the terraform provider for Kestra
terraform validate # validates the configuration incl. the syntax of your flows
terraform apply -auto-approve # deploys your flows - can be used in a CI/CD process
```