⚠️ Disclaimer: Read Before Proceeding
This tutorial is provided for informational purposes only. The steps, commands, and code snippets are presented as a guide and may not be suitable for all systems or environments.
By proceeding with this installation, you acknowledge and agree to the following:
- You follow this tutorial at your own risk. The author(s) and publisher(s) are not responsible for any data loss, system damage, or other negative outcomes that may arise from following these instructions.
- It is highly recommended to back up your system and any important data before starting.
- Software versions, operating system updates, or other factors may cause the steps in this guide to become outdated or inaccurate.
- If you are unsure about any step, consult with a qualified professional or refer to the official software documentation.
- Any mention of third-party software or websites does not constitute an endorsement. You are responsible for verifying the legitimacy and security of any external resources.
Table of Contents
IntroductionRaspberry Pi Cluster HardwareRaspberry Pi Cluster SoftwareConfiguration and Software InstallationUpdate the Raspberry Pi firmware to have it boot from the SSD:Image to SSDConfigure lex00 to use the ethernet switchGive all nodes access to the internet through wlan0Kubernetes (k3s) InstallationSystem ConfigurationInstall k3sArgo Workflows InstallationHera Workflows InstallationMinIO InstallationGenerate certificates for MinIOCreate Image for WorkflowsSign on to each node and install the imageTesting the System with a Workflow
Introduction
This tutorial shows the hardware and software needed to build a Kubernetes cluster
with workflow processing and S3 like storage capabilities using Raspberry Pis as the nodes.
with workflow processing and S3 like storage capabilities using Raspberry Pis as the nodes.
The primary purpose is to get you up and running. After which, you can explore in more detail the various tools on you own.
The example used here is specifically for a 4 node Raspberry Pi cluster (lex), but much of the information presented can be used for all types of installations (specifically when
it comes to Kubernetes and its packages).
it comes to Kubernetes and its packages).
Though this tutorial is for a Raspberry Pi Kubernetes cluster, the same Kubernetes
packages can be used on higher end clusters (local, AWS, etc.).
packages can be used on higher end clusters (local, AWS, etc.).
Notes:
- To help simplify the setup, this tutorial doesn't get into the setup of all the security (i.e., certificates). Running within a protected environment (e.g., a home intranet), this will be fine, but if you want the cluster to be more public, it should have better security.
- MinIO has the capability of storage over all the node's SSDs. This allows the total storage to be greater than that of one node. But MinIO requires at least 4 nodes to use this capability.
Raspberry Pi Cluster Hardware
Hardware | Description |
lex | Name of cluster with each node of that cluster having the names lex00-lex03. |
Raspberry Pi | Motherboard of the cluster node. Note, this installation uses Raspberry Pi 4's. |
POE Hat | Power over ethernet hat. Allows the Raspberry Pi to be powered by the ethernet switch through the network cable. |
SSD | Storage connected to one of the USB 3.0 ports. An SSD is much faster than an SD card. (Each lex node has a 1TB SSD). |
Ethernet Switch | Allows the nodes to directly communicate with one another. Also, this switch is POE capable, allowing it to power the Raspberry Pis. |
Only lex00 has its wireless activated. This forces the nodes to communicate with each other via the ethernet switch for best performance. All nodes do have access to the internet, but through lex00.
To access lex01-lex03 remotely, you need to first login to lex00 and, from there, log into the desired node.
To access lex01-lex03 remotely, you need to first login to lex00 and, from there, log into the desired node.
The following images show the cluster hardware including the cabling and case:
Top left to right: POE Hat, Raspberry Pi, SSD and case with 3 nodes assembled. Bottom left to right: internet cable and SSD to USB cable
Below shows lex fully assembled and running:
Connect the SSD to a USB 3.0 port (blue) for best performance. Before installing the SSDs, install them with Raspbian OS (see below)
Raspberry Pi Cluster Software
Software | Description |
Kubernetes (k3s) | K3s is a certified light weight Kubernetes distribution that is designed for production workloads in environments with less resources. |
Argo Workflows | Workflow manager installed in Kubernetes. |
Hera Workflows | A python wrapper for Argo Workflows. It was developed to make Argo workflow construction and submission more accessible by eliminating the need for the developer to understand Kubernetes. |
MinIO | MinIO is a high-performance, S3 compatible object store which can be installed in k3s. MinIO can be configured to utilize all node SSDs as one object store. |
Configuration and Software Installation
Note: for software installation and container image generation you need python, pip and docker installed on your remote computer.
Update the Raspberry Pi firmware to have it boot from the SSD:
After this firmware update, your Raspberry Pis will boot and use the SSD, so you don't need an SD card (normally used on Raspberry Pis).
Image to SSD
Plug each SSD to each Rasperry Pi node using the SDD to USB cable.
Use the Raspberry Pi imager: https://www.raspberrypi.com/software
- Raspberry Pi Device
- Choose Pi device you are using
- Operating System
- Choose “Raspberry Pi OS (other)”
- Raspberry Pi OS Light (64-bit)
- Storage
- Choose the drive you wish to burn image to (be careful to choose the correct drive)
- NEXT
- EDIT SETTINGS
- GENERAL tab
- Set hostname: (e.g., lex01)
- Username: pi
- Password: <password>
- lex00 only:
- Configure wireless LAN
- SSID: <router name>
- Password: <router password>
- Wireless LAN country: (e.g., US)
- Set local settings
- Time zone (e.g., America/New York)
- SERVICES tab
- Enable SSH
- SAVE
- Would you like to apply OS customization setting?
- YES
- All existing data on <SSD> will be erased. Are you sure you want to continue?
- Make sure the SSD is the correct one.
Once all the SSDs have their OS installed, assemble the nodes into the cluster and boot up the cluster.
Give time for all the nodes to finish booting up.
From remote computer:
ping lex00
When you see a response from lex00, you'll know it ready to login.
Login to lex00:
ssh pi@lex00
Note: to login to the other nodes, you need to login to lex00 first.
Update the Operating System on lex00:
sudo apt update -y sudo apt upgrade -y sudo apt autoremove -y
Configure lex00 to use the ethernet switch
Configure "Wired connection 1" (eth0) with lex00 as the head node:
sudo nmcli con mod "Wired connection 1" ipv4.addresses 192.168.0.10/24 ipv4.method manual
Reset the eth0 connection so it picks up the new configuration:
sudo nmcli con down "Wired connection 1" sudo nmcli con up "Wired connection 1"
Your connection to lex00 may lock for a little time, but wait until it comes back.
Set up lex00 so it and the other nodes can communicate through the ethernet switch
Install the dhcp server:
sudo apt install isc-dhcp-server -y
You will see some error messages, but they are benign.
Configure the dhcp by editing its configuration file:
sudo nano /etc/dhcp/dhcpd.conf
Add to bottom:
ddns-update-style none; authoritative; log-facility local7; # No service will be given on this subnet subnet 192.168.1.0 netmask 255.255.255.0 { } # The internal cluster network group { option broadcast-address 192.168.0.255; option routers 192.168.0.10; default-lease-time 600; max-lease-time 7200; option domain-name "lex00"; option domain-name-servers 8.8.8.8, 8.8.4.4; subnet 192.168.0.0 netmask 255.255.255.0 { range 192.168.0.20 192.168.0.250; # Head Node host lex00 { hardware ethernet dc:a6:32:6a:16:90; fixed-address 192.168.0.10; } } }
Configure the dhcp server by editing:
sudo nano /etc/default/isc-dhcp-server
and updating the following:
DHCPDv4_CONF=/etc/dhcp/dhcp.conf DHCPDv4_PID=/var/run/dhcpd.pid INTERFACESv4="eth0"
Enable port forwarding by editing:
sudo nano /etc/sysctl.conf
and un-commenting the following:
net.ipv4.ip_forward=1
Enable gateway ports for port forwarding:
sudo nano /etc/ssh/sshd_config
Un-comment and set to 'yes':
GatewayPorts yes
Exit from lex00.
Shutdown all nodes by removing their ethernet cable.
Plug in the ethernet cable in lex00 and wait until its ethernet connection light (green) is flashing.
Plug in the remaining nodes one at a time and wait to plug in the next node by waiting until the current node’s ethernet connect light (green) is flashing.
Log in to to lex00 and check its connection with the other nodes:
sudo dhcp-lease-list
You should see lex01-lex03 represented with ports 192.168.0.20, 192.168.0.21 and 192.168.0.22 respectively.
Edit /etc/hosts on lex00:
sudo nano /etc/hosts
and add to end:
192.168.0.10 lex00 192.168.0.20 lex01 192.168.0.21 lex02 192.168.0.22 lex03
Note: Instead of spaces between the ip addresses and the node names use a tab.
Give all nodes access to the internet through wlan0
lex01-03 will not have wireless access, but they can access the internet through lex00's wlan0.
Login to lex00.
Set up iptables:
sudo apt install iptables -y sudo iptables --table nat --append POSTROUTING --out-interface wlan0 -j MASQUERADE sudo iptables --append FORWARD --in-interface eth0 -j ACCEPT
Check iptable configuration:
sudo iptables -S sudo iptables -t nat -S
The output should look like this:
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A FORWARD -i eth0 -j ACCEPT
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -o wlan0 -j MASQUERADE
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A FORWARD -i eth0 -j ACCEPT
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -o wlan0 -j MASQUERADE
Save iptables rules:
sudo apt install iptables-persistent -y
Answer 'yes' to both questions.
Log on to each node (lex01-03) and update their OS (note to access lex01-03, you need to log onto lex00 first):
sudo apt update -y sudo apt upgrade -y sudo apt autoremove -y
Edit /etc/host on lex01-03:
sudo nano /etc/hosts
and add to end:
192.168.0.10 lex00
Note: Instead of spaces between the ip addresses and the node names use a tab.
Kubernetes (k3s) Installation
System Configuration
On each node, set group settings needed by k3s (note: to access lex01-03, you need to log onto lex00 first):
sudo nano /boot/firmware/cmdline.txt
and add to end of line:
cgroup_memory=1 cgroup_enable=memory
Reboot all nodes so they pick up the above commands.
Install k3s
On lex00:
sudo curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -s -
Determine and record the token (will be used in configuring the other nodes):
sudo cat /var/lib/rancher/k3s/server/node-token > token sudo cat token
On lex01-03. Note: replace TOKEN below with token determined above:
sudo curl -sfL https://get.k3s.io | K3S_TOKEN=TOKEN K3S_URL=https://lex00:6443 sh -
Check cluster by going to lex00 and executing the following:
sudo kubectl get nodes
All 4 nodes should be represented and have a status of "Ready”.
Argo Workflows Installation
On lex00, create an Argo install yaml (install.yaml) file with the following contents:
install.yaml
apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: clusterworkflowtemplates.argoproj.io spec: group: argoproj.io names: kind: ClusterWorkflowTemplate listKind: ClusterWorkflowTemplateList plural: clusterworkflowtemplates shortNames: - clusterwftmpl - cwft singular: clusterworkflowtemplate scope: Cluster versions: - name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true required: - metadata - spec type: object served: true storage: true --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: cronworkflows.argoproj.io spec: group: argoproj.io names: kind: CronWorkflow listKind: CronWorkflowList plural: cronworkflows shortNames: - cwf - cronwf singular: cronworkflow scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true status: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true required: - metadata - spec type: object served: true storage: true --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: workflowartifactgctasks.argoproj.io spec: group: argoproj.io names: kind: WorkflowArtifactGCTask listKind: WorkflowArtifactGCTaskList plural: workflowartifactgctasks shortNames: - wfat singular: workflowartifactgctask scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true status: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true required: - metadata - spec type: object served: true storage: true subresources: status: {} --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: workfloweventbindings.argoproj.io spec: group: argoproj.io names: kind: WorkflowEventBinding listKind: WorkflowEventBindingList plural: workfloweventbindings shortNames: - wfeb singular: workfloweventbinding scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true required: - metadata - spec type: object served: true storage: true --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: workflows.argoproj.io spec: group: argoproj.io names: kind: Workflow listKind: WorkflowList plural: workflows shortNames: - wf singular: workflow scope: Namespaced versions: - additionalPrinterColumns: - description: Status of the workflow jsonPath: .status.phase name: Status type: string - description: When the workflow was started format: date-time jsonPath: .status.startedAt name: Age type: date - description: Human readable message indicating details about why the workflow is in this condition. jsonPath: .status.message name: Message type: string name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true status: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true required: - metadata - spec type: object served: true storage: true subresources: {} --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: workflowtaskresults.argoproj.io spec: group: argoproj.io names: kind: WorkflowTaskResult listKind: WorkflowTaskResultList plural: workflowtaskresults singular: workflowtaskresult scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string message: type: string metadata: type: object outputs: properties: artifacts: items: properties: archive: properties: none: type: object tar: properties: compressionLevel: format: int32 type: integer type: object zip: type: object type: object archiveLogs: type: boolean artifactGC: properties: podMetadata: properties: annotations: additionalProperties: type: string type: object labels: additionalProperties: type: string type: object type: object serviceAccountName: type: string strategy: enum: - "" - OnWorkflowCompletion - OnWorkflowDeletion - Never type: string type: object artifactory: properties: passwordSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object url: type: string usernameSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object required: - url type: object azure: properties: accountKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object blob: type: string container: type: string endpoint: type: string useSDKCreds: type: boolean required: - blob - container - endpoint type: object deleted: type: boolean from: type: string fromExpression: type: string gcs: properties: bucket: type: string key: type: string serviceAccountKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object required: - key type: object git: properties: branch: type: string depth: format: int64 type: integer disableSubmodules: type: boolean fetch: items: type: string type: array insecureIgnoreHostKey: type: boolean passwordSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object repo: type: string revision: type: string singleBranch: type: boolean sshPrivateKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object usernameSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object required: - repo type: object globalName: type: string hdfs: properties: addresses: items: type: string type: array force: type: boolean hdfsUser: type: string krbCCacheSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object krbConfigConfigMap: properties: key: type: string name: type: string optional: type: boolean required: - key type: object krbKeytabSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object krbRealm: type: string krbServicePrincipalName: type: string krbUsername: type: string path: type: string required: - path type: object http: properties: auth: properties: basicAuth: properties: passwordSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object usernameSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object type: object clientCert: properties: clientCertSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object clientKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object type: object oauth2: properties: clientIDSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object clientSecretSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object endpointParams: items: properties: key: type: string value: type: string required: - key type: object type: array scopes: items: type: string type: array tokenURLSecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object type: object type: object headers: items: properties: name: type: string value: type: string required: - name - value type: object type: array url: type: string required: - url type: object mode: format: int32 type: integer name: type: string optional: type: boolean oss: properties: accessKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object bucket: type: string createBucketIfNotPresent: type: boolean endpoint: type: string key: type: string lifecycleRule: properties: markDeletionAfterDays: format: int32 type: integer markInfrequentAccessAfterDays: format: int32 type: integer type: object secretKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object securityToken: type: string required: - key type: object path: type: string raw: properties: data: type: string required: - data type: object recurseMode: type: boolean s3: properties: accessKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object bucket: type: string createBucketIfNotPresent: properties: objectLocking: type: boolean type: object encryptionOptions: properties: enableEncryption: type: boolean kmsEncryptionContext: type: string kmsKeyId: type: string serverSideCustomerKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object type: object endpoint: type: string insecure: type: boolean key: type: string region: type: string roleARN: type: string secretKeySecret: properties: key: type: string name: type: string optional: type: boolean required: - key type: object useSDKCreds: type: boolean type: object subPath: type: string required: - name type: object type: array exitCode: type: string parameters: items: properties: default: type: string description: type: string enum: items: type: string type: array globalName: type: string name: type: string value: type: string valueFrom: properties: configMapKeyRef: properties: key: type: string name: type: string optional: type: boolean required: - key type: object default: type: string event: type: string expression: type: string jqFilter: type: string jsonPath: type: string parameter: type: string path: type: string supplied: type: object type: object required: - name type: object type: array result: type: string type: object phase: type: string progress: type: string required: - metadata type: object served: true storage: true --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: workflowtasksets.argoproj.io spec: group: argoproj.io names: kind: WorkflowTaskSet listKind: WorkflowTaskSetList plural: workflowtasksets shortNames: - wfts singular: workflowtaskset scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true status: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true required: - metadata - spec type: object served: true storage: true subresources: status: {} --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: workflowtemplates.argoproj.io spec: group: argoproj.io names: kind: WorkflowTemplate listKind: WorkflowTemplateList plural: workflowtemplates shortNames: - wftmpl singular: workflowtemplate scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: properties: apiVersion: type: string kind: type: string metadata: type: object spec: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true required: - metadata - spec type: object served: true storage: true --- apiVersion: v1 kind: ServiceAccount metadata: name: argo namespace: argo --- apiVersion: v1 kind: ServiceAccount metadata: name: argo-server namespace: argo --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: argo-role namespace: argo rules: - apiGroups: - coordination.k8s.io resources: - leases verbs: - create - get - update - apiGroups: - "" resources: - secrets verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: rbac.authorization.k8s.io/aggregate-to-admin: "true" name: argo-aggregate-to-admin rules: - apiGroups: - argoproj.io resources: - workflows - workflows/finalizers - workfloweventbindings - workfloweventbindings/finalizers - workflowtemplates - workflowtemplates/finalizers - cronworkflows - cronworkflows/finalizers - clusterworkflowtemplates - clusterworkflowtemplates/finalizers - workflowtasksets - workflowtasksets/finalizers - workflowtaskresults - workflowtaskresults/finalizers verbs: - create - delete - deletecollection - get - list - patch - update - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: rbac.authorization.k8s.io/aggregate-to-edit: "true" name: argo-aggregate-to-edit rules: - apiGroups: - argoproj.io resources: - workflows - workflows/finalizers - workfloweventbindings - workfloweventbindings/finalizers - workflowtemplates - workflowtemplates/finalizers - cronworkflows - cronworkflows/finalizers - clusterworkflowtemplates - clusterworkflowtemplates/finalizers - workflowtaskresults - workflowtaskresults/finalizers verbs: - create - delete - deletecollection - get - list - patch - update - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: rbac.authorization.k8s.io/aggregate-to-view: "true" name: argo-aggregate-to-view rules: - apiGroups: - argoproj.io resources: - workflows - workflows/finalizers - workfloweventbindings - workfloweventbindings/finalizers - workflowtemplates - workflowtemplates/finalizers - cronworkflows - cronworkflows/finalizers - clusterworkflowtemplates - clusterworkflowtemplates/finalizers - workflowtaskresults - workflowtaskresults/finalizers verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: argo-cluster-role rules: - apiGroups: - "" resources: - pods - pods/exec verbs: - create - get - list - watch - update - patch - delete - apiGroups: - "" resources: - configmaps verbs: - get - watch - list - apiGroups: - "" resources: - persistentvolumeclaims - persistentvolumeclaims/finalizers verbs: - create - update - delete - get - apiGroups: - argoproj.io resources: - workflows - workflows/finalizers - workflowtasksets - workflowtasksets/finalizers - workflowartifactgctasks verbs: - get - list - watch - update - patch - delete - create - apiGroups: - argoproj.io resources: - workflowtemplates - workflowtemplates/finalizers - clusterworkflowtemplates - clusterworkflowtemplates/finalizers verbs: - get - list - watch - apiGroups: - argoproj.io resources: - workflowtaskresults verbs: - list - watch - deletecollection - apiGroups: - "" resources: - serviceaccounts verbs: - get - list - apiGroups: - argoproj.io resources: - cronworkflows - cronworkflows/finalizers verbs: - get - list - watch - update - patch - delete - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - policy resources: - poddisruptionbudgets verbs: - create - get - delete --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: argo-server-cluster-role rules: - apiGroups: - "" resources: - configmaps verbs: - get - watch - list - apiGroups: - "" resources: - secrets verbs: - get - create - apiGroups: - "" resources: - pods - pods/exec - pods/log verbs: - get - list - watch - delete - apiGroups: - "" resources: - events verbs: - watch - create - patch - apiGroups: - "" resources: - serviceaccounts verbs: - get - list - watch - apiGroups: - argoproj.io resources: - eventsources - sensors - workflows - workfloweventbindings - workflowtemplates - cronworkflows - clusterworkflowtemplates verbs: - create - get - list - watch - update - patch - delete --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: argo-binding namespace: argo roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: argo-role subjects: - kind: ServiceAccount name: argo namespace: argo --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: argo-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: argo-cluster-role subjects: - kind: ServiceAccount name: argo namespace: argo --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: argo-server-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: argo-server-cluster-role subjects: - kind: ServiceAccount name: argo-server namespace: argo --- apiVersion: v1 kind: ConfigMap metadata: name: workflow-controller-configmap namespace: argo --- apiVersion: v1 kind: Service metadata: name: argo-server namespace: argo spec: ports: - name: web port: 2746 targetPort: 2746 selector: app: argo-server --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: workflow-controller value: 1000000 --- apiVersion: apps/v1 kind: Deployment metadata: name: argo-server namespace: argo spec: selector: matchLabels: app: argo-server template: metadata: labels: app: argo-server spec: containers: - args: - server - --namespaced - --auth-mode - server - --auth-mode - client env: [] image: quay.io/argoproj/argocli:v3.4.9 name: argo-server ports: - containerPort: 2746 name: web readinessProbe: httpGet: path: / port: 2746 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 20 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true volumeMounts: - mountPath: /tmp name: tmp nodeSelector: kubernetes.io/os: linux securityContext: runAsNonRoot: true serviceAccountName: argo-server volumes: - emptyDir: {} name: tmp --- apiVersion: apps/v1 kind: Deployment metadata: name: workflow-controller namespace: argo spec: selector: matchLabels: app: workflow-controller template: metadata: labels: app: workflow-controller spec: containers: - args: [] command: - workflow-controller env: - name: LEADER_ELECTION_IDENTITY valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name image: quay.io/argoproj/workflow-controller:v3.4.9 livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 6060 initialDelaySeconds: 90 periodSeconds: 60 timeoutSeconds: 30 name: workflow-controller ports: - containerPort: 9090 name: metrics - containerPort: 6060 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true nodeSelector: kubernetes.io/os: linux priorityClassName: workflow-controller securityContext: runAsNonRoot: true serviceAccountName: argo
Create Argo namespace and install using the install.yaml file:
sudo kubectl create ns argo sudo kubectl apply -n argo -f install.yaml
Check the Argo installation:
sudo kubectl get pods -n argo
Once you see all the pods running, make Argo available for remote access:
sudo kubectl -n argo port-forward deployment/argo-server 2746:2746 &
On the remote computer, port forward to Argo on the cluster:
ssh -f -q -N -L 2746:localhost:2746 pi@lex00
You can access the Argo Workflows page from a web browser on remote computer:
https://localhost:2746
Hera Workflows Installation
On lex00, create the file pod.yaml:
sudo nano pod.yaml
with the following contents:
apiVersion: v1 kind: Pod metadata: name: hera spec: serviceAccount: hera containers: - image: nginx name: hera
Using the pod.yaml file created above, create Kubernetes role for hera in the argo namespace:
sudo kubectl create role hera --verb=list,update,create --resource=workflows.argoproj.io -n argo sudo kubectl create sa hera -n argo sudo kubectl create rolebinding hera --role=hera --serviceaccount=argo:hera -n argo sudo kubectl apply -f pod.yaml -n argo
Check Hera installation:
sudo kubectl get pods -n argo
Execute until you see that hera has a status of "Running"
On the remote computer, add the Hera python package using pip:
pip install hera
Note: Python and pip needs to be installed first on the remote computer.
MinIO Installation
While all the other installations in this tutorial automatically use the latest version, the installation for MinIO uses version 5.0.14. This is the last version that allows configuring MinIO from a web page. Once the the new setup procedures are understood, this tutorial will be updated accordingly.
On lex00, install the MinIO version of kubectl:
sudo wget https://github.com/minio/operator/releases/download/v5.0.14/kubectl-minio_5.0.14_linux_arm64 -O kubectl-minio sudo chmod +x kubectl-minio sudo mv kubectl-minio /usr/local/bin/. sudo kubectl minio version
Shows the version installed when done.
Install and start MinIO in Kubernetes:
sudo kubectl minio init
Check installation:
sudo kubectl get all -n minio-operator
Note the port of "service/minio": 433:<Console Port>.
lex00:<Console Port> is what is used to access the MinIO object store. This with the access key and secret key determined later allows access to the minIO repository.
Once you see all pods running, create the proxy:
sudo kubectl minio proxy -n minio-operator &
You will see something like:
Starting port forward of the Console UI.
Current JWT to login: <TOKEN>
Forwarding from 0.0.0.0:9090 -> 9090
Note and record the <TOKEN>. The token is used to login to the MinIO Operator Console.
Open a browser and go to the MinIO operator site:
http://lex00:9090
Press "Create a Tenant" and edit these fields:
- Name: miniostorage
- Namespace: minio-operator
- Total Size: 2048
Note: "Name" and "Total Size" can be based on your preferences, but "Namespace" must be "minio-operator"
Press "Create"
Note the "Access Key" and "Secret Key"
Press "Download for import"
Stores the access key and secret key to the file: credentials.json
Generate certificates for MinIO
From your remote computer, get the certificates generator:
wget https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 chmod +x certgen-linux-amd64 ./certgen-linux-amd64 -host localhost,lex00
This will generate the following files: private.key and public.crt
Go to the MinIO Operator site using the web browser on the remote computer (http://lex00:9090).
Click on the miniostorage tenet.
Select Security.
Custom Certificates: switch ON.
MinIO Server Certificates
- Click on Cert attachment button
- Find and select the public.crt file your created.
- Click on the Key attachment button
- Find and select the private.key file you created.
Press “Save”.
To access the MinIO Object Store:
https://lex00:9443
Create Image for Workflows
Notes:
- To create the image based on the instructions below, you will need docker installed on your remote computer.
- Kubernetes does not require the image be from docker, but the instructions below are for docker.
Create a file that gives the Hera program access to MinIO (minio_access.txt) with the following:
lex00:<console port determined during MinIO installation above>
<MinIO Access Key>
<MinIO Secret Key>
<MinIO Access Key>
<MinIO Secret Key>
For example:
lex00:30791 RwDiPu1z4F31mu1E 2pXcivwp7XkWKqHskGFLwk25DqU56eR8
Create the image (for arm64), by creating a docker file with the following contents.
nano Dockerfile
with the following:
FROM ubuntu:latest AS build RUN apt update -y RUN apt upgrade -y RUN apt install python-is-python3 -y RUN apt install python3-pip -y RUN pip3 install minio --break-system-packages RUN pip3 install hera --break-system-packages COPY public.crt /root/. COPY minio_access.txt /root/.
This will create an ubuntu docker image that includes the python package for MinIO that works on the Raspberry Pi (arm64)
Notes:
- The python package for Hera is included so that Hera workflows can be written to kickoff other Hera workflows.
- Use of #1 above is not covered in this tutorial, but is good to have installed for future projects that your Hera workflows kickoff other Hera workflows within Kubernetes.
Create image:
docker buildx rm --builder mybuilder docker buildx create --name mybuilder --use docker buildx build --platform linux/arm64 -t lexicon:v1.0.0 . -o type=docker,dest=./lexicon-arm64.tar
Sign on to each node and install the image
Copy the image to each node (lex00-03)
If you are replacing an existing image, remove the current one first:
sudo k3s crictl rmi lexicon:v1.0.0
Install the image:
sudo k3s ctr images import lexicon-arm64.tar
Check that the image is installed:
sudo ctr -a /run/k3s/containerd/containerd.sock images ls | grep lexicon
Testing the System with a Workflow
Create test workflow (on the remote computer):
nano workflow_test.py
and insert the following code:
Workflow Test
#! /usr/bin/env python # This example has one task that stores a file to MinIO which is retrieved by a second task from hera.shared import global_config from hera.workflows import DAG, Task, Workflow, service, script # define the container image to use global_config.image = "docker.io/library/lexicon:v1.0.0" # Create and upload a file to MinIO @script() def file_uploader(): from minio import Minio from minio.error import S3Error # this tells MinIO where to find the certificate os.environ['SSL_CERT_FILE'] = "/root/public.crt" file = open("/root/minio_access.txt", 'r') endpoint = file.readline().strip() accessKey = file.readline().strip() secretKey = file.readline().strip() file.close() # create a client with the MinIO server, its access key # and secret key. client = Minio( endpoint=endpoint, access_key=accessKey, secret_key=secretKey, secure=True ) try: # Make 'spirit' bucket if not exist. found = client.bucket_exists("spirit") if not found: client.make_bucket("spirit") else: print("Bucket 'spirit' already exists") # create a file to upload to the bucket file = open("/ghost.txt", "w") file.write("abcd\n") file.write("efgh\n") file.close() # upload '/ghost.txt' as object name # 'ghost.txt' to bucket 'spirit' client.fput_object( "spirit", "ghost.txt", "/ghost.txt", ) print( "'/ghost.txt' is successfully uploaded as " "object 'ghost' to bucket 'spirit'." ) except S3Error as exc: print("error occurred.", exc) exit(-1) # read file stored in MinIO by file_uploader() @script() def file_reader(): from minio import Minio from minio.error import S3Error # this tells MinIO where to find the certificate os.environ['SSL_CERT_FILE'] = "/root/public.crt" file = open("/root/minio_access.txt", 'r') endpoint = file.readline().strip() accessKey = file.readline().strip() secretKey = file.readline().strip() file.close() # create a client with the MinIO server, its access key # and secret key. client = Minio( endpoint=endpoint, access_key=accessKey, secret_key=secretKey, secure=True ) try: response = client.get_object( bucket_name = "spirit", object_name = "ghost.txt") print("Contents of file 'ghost.txt'") print(response.data.decode()) except S3Error as exc: print("error occurred.", exc) exit(-1) # define and execute the miniotest argo workflow namespace = "argo" service = service.WorkflowsService(host="https://localhost:2746", verify_ssl=False) with Workflow(generate_name="miniotest-", namespace=namespace, automount_service_account_token=True, workflows_service=service, entrypoint="m") as w: with DAG(name="m"): # define the tasks in the workflow t1: Task = file_uploader() t2: Task = file_reader() # execute the tasks sequentially t1 >> t2 # create and execute workflow w.create()
Make script executable and run:
chmod +x workflow_test.py ./workflow_test.py
You can review the workflow’s progress by reviewing the Argo Workflows page:
https://localhost:2746
The workflow executes two tasks: one for creating a file and uploading it to MinIO and the second for downloading the file from MinIO and reading it.
Made with Bullet