1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53# Data Infrastructure as Code
There are two important paradigms that significantly shaped the Data Infrastructure landscape:
1. Infrastructure as Code
2. Workflow as Code
Combining the two helps implement a reliable data engineering lifecycle with:
- reliable infrastructure management
- data pipelines built as code
- version control of both, data workflows and the underlying infrastructure.
This repository includes examples helping you adopt both of these paradigms at the same time.
## Install Kestra
You can start Kestra using Docker-Compose:
```sh
curl -o docker-compose.yml https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml
docker-compose up
```
Before starting Kestra, make sure to create the `.env` file as shown in the [.env_example](.env_example) file. Add any secrets there as environment variables so that you can use them in your workflows in a secure way.
## Install Terraform
You can install Terraform on your local machine using `Homebrew` (for detailed instructions of your OS, check the [Terraform CLI install guide Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)):
```sh
brew tap hashicorp/tap
brew install hashicorp/tap/terraform
```
# Write in code, deploy in one command
Navigate to the relevant project e.g. [aws_s3_tf](aws_s3) and initialize the relevant Terraform providers:
```bash
cd aws_s3/
terraform init
```
Then, deploy the workflow and the underlying infrastructure using the command:
```bash
terraform apply -auto-approve
```