Introduction
This tutorial shows the hardware and software needed to build a Kubernetes cluster with workflow processing and S3 like storage capabilities using Raspberry Pis as the nodes.
The primary purpose is to get you up and running. After which, you can explore the various tools in more detail on you own.
The example used here is specifically for a 4 node Raspberry Pi cluster (lex), but much of the information presented can be used for all types of installations (specifically when it comes to Kubernetes and its packages).
Though this guide is for a Raspberry Pi Kubernetes cluster, the same Kubernetes packages can be used on higher-end clusters (local, AWS, etc.).
Notes:
- To help simplify the setup, this guide doesn't get into the setup of all the security (i.e., certificates for accessing Argo). Running within a protected environment (e.g., a home intranet), will work fine, but if you want the cluster to be more public, it should set it up with better security.
- MinIO has the capability to store data across all the node's SSDs. This allows the total storage to be greater than that of one node. But MinIO requires at least 4 nodes to use this capability.
- You should be able to use as the remote host any machine that has an operating system that can use Conda. The operating systems that have been tested for this tutorial are Ubuntu, Raspberry Pi, Fedora and MacOS.
If you click on the logo at the top of each section, you can get more details on that product.
Raspberry Pi Cluster Hardware
Hardware | Description |
lex | Name of cluster with each node of that cluster having the names lex00-lex03. |
Raspberry Pi | Motherboard of the cluster node. Note: this installation uses Raspberry Pi 4s. |
POE Hat | Power over Ethernet hat. This allows the Raspberry Pi to receive power from the Ethernet switch through the network cable. |
SSD | Storage connected to one of the USB 3.0 ports. An SSD is much faster than an SD card. (Each lex node has a 1TB SSD). |
Ethernet Switch | This allows the nodes to communicate directly with one another. Also, this switch is POE capable, allowing it to power the Raspberry Pis. |
Only lex00 has its wireless activated. This forces the nodes to communicate with each other via the Ethernet switch for best performance. All nodes do have access to the internet, but through lex00.
To access lex01-lex03 remotely, you need to first login to lex00 and, from there, log into the desired node.
The following images show the cluster hardware including the cabling and case:
Top left to right: POE Hat, Raspberry Pi, SSD and case with 3 assembled nodes. Bottom left to right: internet cable and SSD to USB cable.
Below shows lex fully assembled and running:
Connect the SSD to a USB 3.0 port (blue) for best performance. Before installing the SSDs, install them with Raspbian OS (see below).
Raspberry Pi Cluster Software
The following diagram illustrates the interaction between the Kubernetes related software:
Software | Description |
Kubernetes (K3s) | K3s is a certified light weight Kubernetes distribution designed for production workloads in resource-constrained environments. |
Argo Workflows | Workflow manager installed in Kubernetes. |
Hera Workflows | A python wrapper for Argo Workflows. It was developed to make Argo workflow construction and submission more accessible by eliminating the developer's need to understand Kubernetes. |
MinIO | MinIO is a high-performance, S3 compatible object store that can be installed in K3s. MinIO can be configured to utilize all node SSDs as one object store. |
Miniforge | A lightweight alternative to Anaconda or Miniconda, offering a way to manage Python packages and environments. |
Podman | For creating container images for the Kubernetes system. |
Use commands and code with caution.
Configuration and Software Installation
Update the Raspberry Pi firmware to have it boot from the SSD:
If using the Raspberry Pi 5 for the nodes, as of 2026, a firmware update is not required. So you can skip this step.
After this firmware update, your Raspberry Pis will boot and use the SSD, eliminating the need for an SD card (which is normally used on Raspberry Pis).
Image to SSD
Plug each SSD into each Raspberry Pi node using the SSD-to-USB cable.
Use the Raspberry Pi imager: https://www.raspberrypi.com/software
Use the Raspberry Pi imager: https://www.raspberrypi.com/software
- Raspberry Pi Device
- Choose Pi device you are using.
- Operating System
- Choose “Raspberry Pi OS (other)”.
- Raspberry Pi OS Light (64-bit).
- Storage
- Choose the drive you wish to burn image to (be careful to choose the correct drive).
- Press NEXT
- Press EDIT SETTINGS
- Select GENERAL tab
- Set hostname: (e.g., lex03)
- Username: pi
- Password: <password>
- lex00 only:
- Configure wireless LAN
- SSID: <router name>
- Password: <router password>
- Wireless LAN country: (e.g., US)
- Set local settings
- Time zone (e.g., America/New York)
- Select SERVICES tab
- Enable SSH
- Press SAVE.
- Would you like to apply OS customization setting?
- Press YES.
- All existing data on <SSD> will be erased. Are you sure you want to continue?
- Ensure the SSD is the correct one then press YES.
- The image is written to the SSD.
Once all the SSDs have their OS installed, assemble the nodes into the cluster and boot it up.
Allow time for all the nodes to finish booting up.
From the remote computer:
Remote computer
ping lex00
When you see a response from lex00, you'll know it's ready to log in.
Login to lex00:
Remote computer
ssh pi@lex00
Note: To log in to the other nodes, you must first log in to lex00.
Update the Operating System on lex00:
lex00
sudo apt update sudo apt upgrade -y sudo apt autoremove -y
Configure lex00 to use the Ethernet switch
Configure "Wired connection 1" (eth0) with lex00 as the head node:
lex00
sudo nmcli con mod "Wired connection 1" ipv4.addresses 192.168.0.10/24 ipv4.method manual
Reset the eth0 connection to pick up the new configuration:
lex00
sudo nmcli con down "Wired connection 1" sudo nmcli con up "Wired connection 1"
Your connection to lex00 may lock briefly, but wait until it comes back.
Set up lex00 so it and the other nodes can communicate though the Ethernet switch
Install the DHCP server:
lex00
sudo apt install isc-dhcp-server -y
You will see some error messages, but they are benign.
Configure the DHCP by editing its configuration file:
lex00
sudo nano /etc/dhcp/dhcpd.conf
Add the following to the bottom:
lex00 (/etc/dhcp/dhcpd.conf)
ddns-update-style none; authoritative; log-facility local7; # No service will be given on this subnet subnet 192.168.1.0 netmask 255.255.255.0 { } # The internal cluster network group { option broadcast-address 192.168.0.255; option routers 192.168.0.10; default-lease-time 600; max-lease-time 7200; option domain-name "lex00"; option domain-name-servers 8.8.8.8, 8.8.4.4; subnet 192.168.0.0 netmask 255.255.255.0 { range 192.168.0.20 192.168.0.250; # Head Node host lex00 { hardware ethernet dc:a6:32:6a:16:90; fixed-address 192.168.0.10; } } }
Configure the DHCP server by editing:
lex00
sudo nano /etc/default/isc-dhcp-server
and update the following:
lex00 (/etc/default/isc-dhcp-server)
DHCPDv4_CONF=/etc/dhcp/dhcp.conf DHCPDv4_PID=/var/run/dhcpd.pid INTERFACESv4="eth0"
Enable port forwarding by editing:
lex00
sudo nano /etc/sysctl.d/98-rpi.conf
and add the following to the bottom:
lex00 (/etc/sysctl.d/980rpi.conf)
net.ipv4.ip_forward=1
Enable gateway ports for port forwarding:
lex00
sudo nano /etc/ssh/sshd_config
Un-comment and set to 'yes':
lex00 (/etc/ssh/sshd_config)
GatewayPorts yes
Exit lex00.
Shut down all nodes by removing their Ethernet cables.
Plug the Ethernet cable into lex00 and wait until its Ethernet connection light (green) flashes.
Plug in the remaining nodes one at a time (in order lex01, lex02 and lex03), waiting until each node’s Ethernet connection light (green) flashes before plugging in the next node.
Log back into lex00 and check its connection with the other nodes:
lex00
sudo dhcp-lease-list
You should see lex01-lex03 represented with ports 192.168.0.20, 192.168.0.21 and 192.168.0.22 respectively.
Edit /etc/hosts:
lex00
sudo nano /etc/hosts
and add to end:
lex00 (/etc/hosts)
192.168.0.10 lex00 192.168.0.20 lex01 192.168.0.21 lex02 192.168.0.22 lex03
Give all nodes access to the internet through wlan0
lex01-03 will not have wireless access, but they can access the internet through lex00's wlan0.
Set up iptables:
lex00
sudo apt install iptables -y sudo iptables --table nat --append POSTROUTING --out-interface wlan0 -j MASQUERADE sudo iptables --append FORWARD --in-interface eth0 -j ACCEPT
Check the iptable configuration:
lex00
sudo iptables -S sudo iptables -t nat -S
The output should look like this:
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A FORWARD -i eth0 -j ACCEPT
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -o wlan0 -j MASQUERADE
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A FORWARD -i eth0 -j ACCEPT
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -o wlan0 -j MASQUERADE
Save the iptable rules:
lex00
sudo apt install iptables-persistent -y
Answer 'yes' to both questions.
Log in to each node (lex01-03) and update their OS (note: to access lex01-03, you must first log onto lex00 first):
lex01, lex02 and lex03
sudo apt update sudo apt upgrade -y sudo apt autoremove -y
Edit /etc/hosts on lex01-03:
lex01, lex02 and lex03
sudo nano /etc/hosts
and add to the end:
lex01, lex02 and lex03 (/etc/hosts)
192.168.0.10 lex00
Conda Installation (Miniforge)
Conda is an environment manager which allows you to install software packages in an isolated environment. We specifically use Conda to install Python and the Hera Python package. Miniforge is used as it is a lightweight implementation of Conda.
Install Conda:
Remote computer
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh bash Miniforge3-$(uname)-$(uname -m).sh -b -p ${HOME}/conda rm -f Miniforge3-$(uname)-$(uname -m).sh source $HOME/conda/bin/activate conda init
After installation, create a conda (argodev) environment and activate it:
Remote computer
conda create -n argodev -y conda activate argodev
The following installs packages into argodev for use by this tutorial:
Remote computer (argodev)
conda install python -y pip install hera
Kubernetes (K3s) Installation
System Configuration
On each node, set the group settings needed by K3s (note: to access lex01-03, you must first log onto lex00 first):
lex00, lex01, lex02 and lex03
sudo nano /boot/firmware/cmdline.txt
and add to the end of the line:
lex00, lex01, lex02 and lex03 (/boot/firmware/cmdline.txt)
cgroup_memory=1 cgroup_enable=memory
Reboot all nodes
Install K3s
On lex00:
lex00
sudo curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -s -
Determine and record the token (this will be used when configuring the other nodes):
lex00
sudo cat /var/lib/rancher/k3s/server/node-token > token sudo cat token
On lex01-03. Note: Replace TOKEN below with the token determined above:
lex01, lex02 and lex03
sudo curl -sfL https://get.k3s.io | K3S_TOKEN=TOKEN K3S_URL=https://lex00:6443 sh -
Check the cluster by going to lex00 and executing the following:
lex00
sudo kubectl get nodes
All 4 nodes should be represented and have a status of "Ready”.
MinIO Installation
Install MinIO:
lex00
sudo apt install -y dpkg wget https://github.com/minio/operator/releases/download/v5.0.14/kubectl-minio_5.0.14_$(uname | tr '[:upper:]' '[:lower:]')_$(dpkg --print-architecture) -O kubectl-minio chmod +x kubectl-minio sudo mv kubectl-minio /usr/local/bin/. kubectl minio version
Shows the installed version when done.
Install and start MinIO in Kubernetes:
lex00
kubectl minio init
Check the MinIO pods:
lex00
kubectl get all -n minio-operator
Execute until you see the console and at least one of the minio-operator pods are running.
Once the pods are running, create the proxy:
lex00
kubectl minio proxy -n minio-operator &
You will see something like this:
Starting port forward of the Console UI.
To connect open a browser and go to http://localhost:9090
Current JWT to login: <TOKEN>
Forwarding from 0.0.0.0:9090 -> 9090
Note and record the <TOKEN>. The JWT token is used to log in to the MinIO Operator Console.
Open a browser and go to the MinIO operator site:
http://lex00:9090
Enter the JWT token to sign in. You’ll see the following:
Press "Create a Tenant". Edit the fields marked in red.
Note: "Namespace" must be "minio-operator", but the rest can be set to your desired preferences. Also, because MinIO uses erasure coding for redundancy to tolerate multiple drive failures, the “Total Size” needs to be much less then the available disk space. With the “Erasure Code Parity” of “2” and a “Total Size” of 1.5TiB, the actual amount of disk space used is ab 1.5x that or 2.5TiB.
When done, press “Create”.
The following screen pops up:
Note and record the "Access Key" and "Secret Key"
If you wish, press "Download for import". This stores the access key and secret key in the file: credentials.json
Check the installation:
lex00
kubectl get all -n minio-operator
Note the console port of "service/minio": 433:<Console Port> like shown in red below:
"lex00:<Console Port>" is used to access the MinIO object store. This, along with the access key and secret key, allows access to the MinIO repository from the workflow.
Create a file that will give the Hera workflow access to MinIO with the following (determined during MinIO installation above):
lex00:<console port>
<MinIO Access Key>
<MinIO Secret Key>
<MinIO Access Key>
<MinIO Secret Key>
lex00
nano minio_access.txt
For example:
lex00:30791 RwDiPu1z4F31mu1E 2pXcivwp7XkWKqHskGFLwk25DqU56eR8
Generate certificates for MinIO
On lex00, get the certificate generator and generate the certificates:
lex00
wget https://github.com/minio/certgen/releases/latest/download/certgen-$(uname)-$(dpkg --print-architecture) chmod +x certgen-$(uname)-$(dpkg --print-architecture) ./certgen-$(uname)-$(dpkg --print-architecture) -host localhost,lex00 rm -f certgen-$(uname)-$(dpkg --print-architecture)
This will generate the following files: private.key and public.crt.
Copy the certificates from lex00 to the remote machine:
Remote computer
cd ~/argo rsync -av pi@lex00:public.crt . rsync -av pi@lex00:private.key .
Go to the MinIO Operator site using the web browser (http://lex00:9090). You should see:
Click on the miniostorage tenet box.
Select Security:
Custom Certificates: Switch ON.
MinIO Server Certificates
- Click the Cert attachment button
- Find and select the public.crt file you created.
- Click the Key attachment button
- Find and select the private.key file you created.
Press “Save”. Then “Restart”:
You should now see the attached certificate. For example:
Argo Workflows Installation
On lex00, create an Argo installation YAML file:
lex00
nano argo.yaml
with the following contents:
argo.yaml
apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: clusterworkflowtemplates.argoproj.io spec: group: argoproj.io names: kind: ClusterWorkflowTemplate listKind: ClusterWorkflowTemplateList plural: clusterworkflowtemplates shortNames: - clusterwftmpl - cwft singular: clusterworkflowtemplate scope: Cluster versions: - name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true required: - metadata - spec type: object served: true storage: true --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: cronworkflows.argoproj.io spec: group: argoproj.io names: kind: CronWorkflow listKind: CronWorkflowList plural: cronworkflows shortNames: - cwf - cronwf singular: cronworkflow scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true status: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true required: - metadata - spec type: object served: true storage: true --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: workflowartifactgctasks.argoproj.io spec: group: argoproj.io names: kind: WorkflowArtifactGCTask listKind: WorkflowArtifactGCTaskList plural: workflowartifactgctasks shortNames: - wfat singular: workflowartifactgctask scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true status: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true required: - metadata - spec type: object served: true storage: true subresources: status: {} --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: workfloweventbindings.argoproj.io spec: group: argoproj.io names: kind: WorkflowEventBinding listKind: WorkflowEventBindingList plural: workfloweventbindings shortNames: - wfeb singular: workfloweventbinding scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true required: - metadata - spec type: object served: true storage: true --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: workflows.argoproj.io spec: group: argoproj.io names: kind: Workflow listKind: WorkflowList plural: workflows shortNames: - wf singular: workflow scope: Namespaced versions: - additionalPrinterColumns: - description: Status of the workflow jsonPath: .status.phase name: Status type: string - description: When the workflow was started format: date-time jsonPath: .status.startedAt name: Age type: date - description: Human readable message indicating details about why the workflow is in this condition. jsonPath: .status.message name: Message type: string name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true status: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true required: - metadata - spec type: object served: true storage: true subresources: {} --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: workflowtaskresults.argoproj.io spec: group: argoproj.io names: kind: WorkflowTaskResult listKind: WorkflowTaskResultList plural: workflowtaskresults singular: workflowtaskresult scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string message: type: string metadata: type: object outputs: properties: artifacts: items: properties: archive: properties: none: type: object tar: properties: compressionLevel: format: int32 type: integer type: object zip: type: object type: object archiveLogs: type: boolean artifactGC: properties: podMetadata: properties: annotations: additionalProperties: type: string type: object labels: additionalProperties: type: string type: object type: object serviceAccountName: type: string strategy: enum: - "" - OnWorkflowCompletion - OnWorkflowDeletion - Never type: string type: object artifactory: properties: passwordSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object url: type: string usernameSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object required: - url type: object azure: properties: accountKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object blob: type: string container: type: string endpoint: type: string useSDKCreds: type: boolean required: - blob - container - endpoint type: object deleted: type: boolean from: type: string fromExpression: type: string gcs: properties: bucket: type: string key: type: string serviceAccountKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object required: - key type: object git: properties: branch: type: string depth: format: int64 type: integer disableSubmodules: type: boolean fetch: items: type: string type: array insecureIgnoreHostKey: type: boolean passwordSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object repo: type: string revision: type: string singleBranch: type: boolean sshPrivateKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object usernameSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object required: - repo type: object globalName: type: string hdfs: properties: addresses: items: type: string type: array force: type: boolean hdfsUser: type: string krbCCacheSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object krbConfigConfigMap: properties: key: type: string name: type: string optional: type: boolean required: - key type: object krbKeytabSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object krbRealm: type: string krbServicePrincipalName: type: string krbUsername: type: string path: type: string required: - path type: object http: properties: auth: properties: basicAuth: properties: passwordSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object usernameSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object type: object clientCert: properties: clientCertSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object clientKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object type: object oauth2: properties: clientIDSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object clientSecretSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object endpointParams: items: properties: key: type: string value: type: string required: - key type: object type: array scopes: items: type: string type: array tokenURLSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object type: object type: object headers: items: properties: name: type: string value: type: string required: - name - value type: object type: array url: type: string required: - url type: object mode: format: int32 type: integer name: type: string optional: type: boolean oss: properties: accessKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object bucket: type: string createBucketIfNotPresent: type: boolean endpoint: type: string key: type: string lifecycleRule: properties: markDeletionAfterDays: format: int32 type: integer markInfrequentAccessAfterDays: format: int32 type: integer type: object secretKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object securityToken: type: string required: - key type: object path: type: string raw: properties: data: type: string required: - data type: object recurseMode: type: boolean s3: properties: accessKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object bucket: type: string createBucketIfNotPresent: properties: objectLocking: type: boolean type: object encryptionOptions: properties: enableEncryption: type: boolean kmsEncryptionContext: type: string kmsKeyId: type: string serverSideCustomerKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object type: object endpoint: type: string insecure: type: boolean key: type: string region: type: string roleARN: type: string secretKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object useSDKCreds: type: boolean type: object subPath: type: string required: - name type: object type: array exitCode: type: string parameters: items: properties: default: type: string description: type: string enum: items: type: string type: array globalName: type: string name: type: string value: type: string valueFrom: properties: configMapKeyRef: properties: key: type: string name: type: string optional: type: boolean required: - key type: object default: type: string event: type: string expression: type: string jqFilter: type: string jsonPath: type: string parameter: type: string path: type: string supplied: type: object type: object required: - name type: object type: array result: type: string type: object phase: type: string progress: type: string required: - metadata type: object served: true storage: true --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: workflowtasksets.argoproj.io spec: group: argoproj.io names: kind: WorkflowTaskSet listKind: WorkflowTaskSetList plural: workflowtasksets shortNames: - wfts singular: workflowtaskset scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true status: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true required: - metadata - spec type: object served: true storage: true subresources: status: {} --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: workflowtemplates.argoproj.io spec: group: argoproj.io names: kind: WorkflowTemplate listKind: WorkflowTemplateList plural: workflowtemplates shortNames: - wftmpl singular: workflowtemplate scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true required: - metadata - spec type: object served: true storage: true --- apiVersion: v1 kind: ServiceAccount metadata: name: argo namespace: argo --- apiVersion: v1 kind: ServiceAccount metadata: name: argo-server namespace: argo --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: argo-role namespace: argo rules: - apiGroups: - coordination.k8s.io resources: - leases verbs: - create - get - update - apiGroups: - "" resources: - secrets verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: rbac.authorization.k8s.io/aggregate-to-admin: "true" name: argo-aggregate-to-admin rules: - apiGroups: - argoproj.io resources: - workflows - workflows/finalizers - workfloweventbindings - workfloweventbindings/finalizers - workflowtemplates - workflowtemplates/finalizers - cronworkflows - cronworkflows/finalizers - clusterworkflowtemplates - clusterworkflowtemplates/finalizers - workflowtasksets - workflowtasksets/finalizers - workflowtaskresults - workflowtaskresults/finalizers verbs: - create - delete - deletecollection - get - list - patch - update - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: rbac.authorization.k8s.io/aggregate-to-edit: "true" name: argo-aggregate-to-edit rules: - apiGroups: - argoproj.io resources: - workflows - workflows/finalizers - workfloweventbindings - workfloweventbindings/finalizers - workflowtemplates - workflowtemplates/finalizers - cronworkflows - cronworkflows/finalizers - clusterworkflowtemplates - clusterworkflowtemplates/finalizers - workflowtaskresults - workflowtaskresults/finalizers verbs: - create - delete - deletecollection - get - list - patch - update - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: rbac.authorization.k8s.io/aggregate-to-view: "true" name: argo-aggregate-to-view rules: - apiGroups: - argoproj.io resources: - workflows - workflows/finalizers - workfloweventbindings - workfloweventbindings/finalizers - workflowtemplates - workflowtemplates/finalizers - cronworkflows - cronworkflows/finalizers - clusterworkflowtemplates - clusterworkflowtemplates/finalizers - workflowtaskresults - workflowtaskresults/finalizers verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: argo-cluster-role rules: - apiGroups: - "" resources: - pods - pods/exec verbs: - create - get - list - watch - update - patch - delete - apiGroups: - "" resources: - configmaps verbs: - get - watch - list - apiGroups: - "" resources: - persistentvolumeclaims - persistentvolumeclaims/finalizers verbs: - create - update - delete - get - apiGroups: - argoproj.io resources: - workflows - workflows/finalizers - workflowtasksets - workflowtasksets/finalizers - workflowartifactgctasks verbs: - get - list - watch - update - patch - delete - create - apiGroups: - argoproj.io resources: - workflowtemplates - workflowtemplates/finalizers - clusterworkflowtemplates - clusterworkflowtemplates/finalizers verbs: - get - list - watch - apiGroups: - argoproj.io resources: - workflowtaskresults verbs: - list - watch - deletecollection - apiGroups: - "" resources: - serviceaccounts verbs: - get - list - apiGroups: - argoproj.io resources: - cronworkflows - cronworkflows/finalizers verbs: - get - list - watch - update - patch - delete - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - policy resources: - poddisruptionbudgets verbs: - create - get - delete --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: argo-server-cluster-role rules: - apiGroups: - "" resources: - configmaps verbs: - get - watch - list - apiGroups: - "" resources: - secrets verbs: - get - create - apiGroups: - "" resources: - pods - pods/exec - pods/log verbs: - get - list - watch - delete - apiGroups: - "" resources: - events verbs: - watch - create - patch - apiGroups: - "" resources: - serviceaccounts verbs: - get - list - watch - apiGroups: - argoproj.io resources: - eventsources - sensors - workflows - workfloweventbindings - workflowtemplates - cronworkflows - clusterworkflowtemplates verbs: - create - get - list - watch - update - patch - delete --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: argo-binding namespace: argo roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: argo-role subjects: - kind: ServiceAccount name: argo namespace: argo --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: argo-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: argo-cluster-role subjects: - kind: ServiceAccount name: argo namespace: argo --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: argo-server-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: argo-server-cluster-role subjects: - kind: ServiceAccount name: argo-server namespace: argo --- apiVersion: v1 kind: ConfigMap metadata: name: workflow-controller-configmap namespace: argo --- apiVersion: v1 kind: Service metadata: name: argo-server namespace: argo spec: ports: - name: web port: 2746 targetPort: 2746 selector: app: argo-server --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: workflow-controller value: 1000000 --- apiVersion: apps/v1 kind: Deployment metadata: name: argo-server namespace: argo spec: selector: matchLabels: app: argo-server template: metadata: labels: app: argo-server spec: containers: - args: - server - --namespaced - --auth-mode - server - --auth-mode - client env: [] image: quay.io/argoproj/argocli:v3.4.9 name: argo-server ports: - containerPort: 2746 name: web readinessProbe: httpGet: path: / port: 2746 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 20 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true volumeMounts: - mountPath: /tmp name: tmp nodeSelector: kubernetes.io/os: linux securityContext: runAsNonRoot: true serviceAccountName: argo-server volumes: - emptyDir: {} name: tmp --- apiVersion: apps/v1 kind: Deployment metadata: name: workflow-controller namespace: argo spec: selector: matchLabels: app: workflow-controller template: metadata: labels: app: workflow-controller spec: containers: - args: [] command: - workflow-controller env: - name: LEADER_ELECTION_IDENTITY valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name image: quay.io/argoproj/workflow-controller:v3.4.9 livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 6060 initialDelaySeconds: 90 periodSeconds: 60 timeoutSeconds: 30 name: workflow-controller ports: - containerPort: 9090 name: metrics - containerPort: 6060 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true nodeSelector: kubernetes.io/os: linux priorityClassName: workflow-controller securityContext: runAsNonRoot: true serviceAccountName: argo
Create the Argo namespace and install using the argo.yaml file:
lex00
kubectl create ns argo kubectl apply -n argo -f argo.yaml
Check the Argo installation:
lex00
kubectl get pods -n argo
Once all pods are running, make Argo available for remote access:
lex00
kubectl patch svc argo-server -n argo -p '{"spec": {"type": "LoadBalancer"}}'
You can access the Argo Workflows page from a web browser on the remote computer:
https://lex00:2746
Note: The browser will issue warnings that the site is not secure, but you can safely ignore these warnings and navigate to the site.
Once you get to the site, you may get an few introduction pages. After getting past them, you should see the following:
Hera Workflows Installation
On lex00, create a Hera installation yaml file:
sudo nano hera.yaml
with the following contents:
apiVersion: v1 kind: Pod metadata: name: hera spec: serviceAccount: hera containers: - image: nginx name: hera
Using the hera.yaml file created above, create a Kubernetes role for hera in the argo namespace:
sudo kubectl create role hera --verb=list,update,create --resource=workflows.argoproj.io -n argo sudo kubectl create sa hera -n argo sudo kubectl create rolebinding hera --role=hera --serviceaccount=argo:hera -n argo sudo kubectl apply -f hera.yaml -n argo
Check the Hera installation:
sudo kubectl get pods -n argo
Execute until you see that "hera" has a status of "Running”
Create the Container Image for Workflows using Podman
On lex00, install Podman:
lex00
sudo apt install podman -y
Create the image that will be used by kubernetes using a container file:
lex00
nano Containerfile
with the following contents:
This will create an Ubuntu container image that includes the MinIO Python package.
Note: The Python package for Hera is also included so that Hera workflows can initiate other Hera workflows. This is not covered in this tutorial, but installing it is beneficial for future projects when you want your Hera workflows to kickoff other Hera workflows within Kubernetes.
Build the image using podman and make an image installation file from it:
lex00
rm -f argodemo.tar podman build -t argodemo:v1 -o argodemo . podman save -o argodemo.tar argodemo:v1
Install the image installation file and check that was installed successfully:
lex00
sudo k3s ctr images import argodemo.tar sudo k3s ctr -a /run/k3s/containerd/containerd.sock images ls | grep argodemo
Copy the image to the other nodes and install the image
Copy the image to each node (lex01-03):
lex00
rsync -av lexicon.tar lex01:/home/pi/. rsync -av lexicon.tar lex02:/home/pi/. rsync -av lexicon.tar lex03:/home/pi/.
Login to each of the nodes and install the container image into Kubernetes and check that was installed successfully:
lex00, lex01, lex02 and lex03
sudo k3s crictl rmi lexicon:v1 sudo k3s ctr images import lexicon.tar sudo ctr -a /run/k3s/containerd/containerd.sock images ls | grep lexicon
Testing the System with a Workflow
On the remote machine, create a test workflow:
Remote computer (argodev)
nano workflow_test.py
with the following code:
Workflow Test
#! /usr/bin/env python # This example has one task that stores a file to MinIO which is retrieved # by a second task from hera.shared import global_config from hera.workflows import DAG, Task, Workflow, service, script # define the container image to use global_config.image = "localhost/argodemo:v1" ################### # WARNING # The bearer token and authentication certificate not used for security. # The following disables warnings caused by insecure certificate # verification.For a production system, these should be set up. import urllib3 urllib3.disable_warnings() ################### # Create and upload a file to MinIO @script() def file_uploader(): from minio import Minio from minio.error import S3Error # this tells MinIO where to find the certificate os.environ['SSL_CERT_FILE'] = "public.crt" file = open("minio_access.txt", 'r') endpoint = file.readline().strip() accessKey = file.readline().strip() secretKey = file.readline().strip() file.close() # create a client with the MinIO server, its access key # and secret key. client = Minio( endpoint=endpoint, access_key=accessKey, secret_key=secretKey, secure=True ) try: # Make 'spirit' bucket if not exist. found = client.bucket_exists("spirit") if not found: client.make_bucket("spirit") else: print("Bucket 'spirit' already exists") # create a file to upload to the bucket file = open("/ghost.txt", "w") file.write("abcd\n") file.write("efgh\n") file.close() # upload '/ghost.txt' as object name # 'ghost.txt' to bucket 'spirit' client.fput_object( "spirit", "ghost.txt", "/ghost.txt", ) print( "'/ghost.txt' is successfully uploaded as " "object 'ghost' to bucket 'spirit'." ) except S3Error as exc: print("error occurred.", exc) exit(-1) # read file stored in MinIO by file_uploader() @script() def file_reader(): from minio import Minio from minio.error import S3Error # this tells MinIO where to find the certificate os.environ['SSL_CERT_FILE'] = "public.crt" file = open("minio_access.txt", 'r') endpoint = file.readline().strip() accessKey = file.readline().strip() secretKey = file.readline().strip() file.close() # create a client with the MinIO server, its access key # and secret key. client = Minio( endpoint=endpoint, access_key=accessKey, secret_key=secretKey, secure=True ) try: response = client.get_object( bucket_name = "spirit", object_name = "ghost.txt") print("Contents of file 'ghost.txt'") print(response.data.decode()) except S3Error as exc: print("error occurred.", exc) exit(-1) # define and execute the miniotest argo workflow namespace = "argo" service = service.WorkflowsService(host="https://lex00:2746", verify_ssl=False) with Workflow(generate_name="miniotest-", namespace=namespace, automount_service_account_token=True, workflows_service=service, entrypoint="m") as w: with DAG(name="m"): # define the tasks in the workflow t1: Task = file_uploader() t2: Task = file_reader() # execute the tasks sequentially t1 >> t2 # create, execute workflow and wait until workflow is complete w.create()
Make the script executable and run it:
Remote computer (workflow_test.py)
chmod +x workflow_test.py ./workflow_test.py
This workflow has two tasks. One to create and store a file to MinIO and another that reads the file from MinIO and displays its contents.
You can review the workflow’s progress on the Argo Workflows page:
https://localhost:2746
You should see the miniotest workflow:
Once the workflow has completed successfully, you’ll see a green check mark:
Click within the miniotest workflow box. You should see the following:
This is a graphical representation of the executed workflow. The top node represents the workflow itself. The middle node represents the file-uploader task and the bottom node represents the file-reader task.
If you click on the “LOGS” button near the top left of the screen, you’ll see the logs generated by the tasks.
Conclusion
You now have what you need to begin developing and testing Argo workflows using Hera on your own Kubernetes cluster. For more information, you can visit Hera's documentation website. Additionally, using workflow_test.py as a template, you can quickly begin writing your own workflows.
Cleanup
This section provides the necessary command for if/when you wish to remove your argodev Conda environment used in this tutorial.
To Remove the argodev Conda Environment
# if currently in the argodev environment, you need to deactivate it conda deactivate # remove argodev environment conda env remove -n argodev -y
Made with Bullet