Files

medusa 116911ac8e Update tech_docs/edge-compute-k3s.md

2024-10-14 06:16:06 +00:00

21 KiB

Raw Blame History

Great! I’ll review and streamline the suggestions to ensure they are useful, actionable, and necessary, focusing on practical learning while setting up your system. Here’s a clean, highly focused version to guide your journey into learning the Linux CLI, bash, and shell scripting as you set up Rocky Linux for K3s and Python-based workloads:

1. Update and Secure the System

Start by ensuring your system is updated and secure. This is essential to establish a solid foundation.

System Updates

sudo dnf update -y && sudo dnf upgrade -y
sudo reboot

Purpose: Keep the system updated with the latest security patches and improvements.

SELinux Configuration

sestatus  # Check SELinux status

If not in enforcing mode, edit the config:

sudo nano /etc/selinux/config

Set SELINUX=enforcing, save, and reboot.
Purpose: Enforce security at the kernel level by keeping SELinux enabled.

2. Install Essential System Tools

Install basic tools for system management and development.

Install Development Tools and Utilities

sudo dnf groupinstall "Development Tools" -y
sudo dnf install wget curl vim nano net-tools git unzip -y

Purpose: Install necessary utilities and compilers to manage and build software.

Firewall Configuration

sudo firewall-cmd --permanent --add-port=6443/tcp  # K3s API server port
sudo firewall-cmd --permanent --add-port=10250/tcp # Kubelet port
sudo firewall-cmd --reload

Purpose: Allow required ports for K3s operations and secure the system by controlling network access.

3. Install Python and Set Up a Virtual Environment

Install Python 3 and Development Tools

sudo dnf install python3 python3-pip python3-devel -y
python3 --version
pip3 --version

Purpose: Install Python 3 and ensure it's set up for development.

Set Up a Python Virtual Environment

python3 -m venv ~/myenv
source ~/myenv/bin/activate

Purpose: Create and activate a virtual environment to manage Python packages independently.

4. Install and Configure K3s

Install K3s

curl -sfL https://get.k3s.io | sh -
sudo systemctl status k3s  # Check K3s status

Purpose: Install K3s and ensure the service is running.

Configure K3s CLI

sudo ln -s /usr/local/bin/kubectl /usr/bin/kubectl
kubectl get nodes  # Verify the node is running

Purpose: Set up kubectl to interact with your K3s cluster.

5. Learn Bash Scripting through Automation

Now that the basics are in place, you can learn shell scripting by automating common tasks. This reinforces your CLI skills while building useful scripts.

Automate System Updates

Create a script to automate updates and log the process.

Script: update-system.sh

#!/bin/bash
LOGFILE=~/update-log.txt
echo "Updating system..." | tee -a $LOGFILE
sudo dnf update -y | tee -a $LOGFILE
sudo dnf upgrade -y | tee -a $LOGFILE
echo "System updated on $(date)" | tee -a $LOGFILE

Purpose: Automate the update process and learn about logging and automation.

Automate Python Environment Setup

Script: setup-python-env.sh

#!/bin/bash
ENV_NAME=$1

if [ -z "$ENV_NAME" ]; then
    echo "Usage: $0 <env_name>"
    exit 1
fi

echo "Creating Python virtual environment '$ENV_NAME'..."
python3 -m venv ~/$ENV_NAME

echo "Activating virtual environment..."
source ~/$ENV_NAME/bin/activate

echo "Virtual environment '$ENV_NAME' created and activated."

Purpose: Create a flexible script to automate Python environment setup, using arguments and learning bash conditionals.

Automate K3s Installation and Configuration

Script: install-k3s.sh

#!/bin/bash

echo "Installing K3s..."
curl -sfL https://get.k3s.io | sh -

if systemctl status k3s | grep "active (running)" > /dev/null; then
    echo "K3s is installed and running!"
else
    echo "K3s installation failed."
fi

echo "Configuring firewall..."
sudo firewall-cmd --permanent --add-port=6443/tcp
sudo firewall-cmd --permanent --add-port=10250/tcp
sudo firewall-cmd --reload

echo "K3s and firewall configured successfully."

Purpose: Automate the installation and configuration of K3s while practicing bash scripting with conditionals.

6. Monitor the System

Create a script to monitor your system’s health. This teaches system commands and formatting output.

Script: monitor-resources.sh

#!/bin/bash
echo "System Resource Usage"
echo "---------------------"
echo "Disk Usage:"
df -h
echo "Memory Usage:"
free -m
echo "CPU Load:"
uptime

Purpose: Gather and display system metrics to monitor performance.

7. Learn Automation through Cron Jobs

Use cron to automate tasks like system backups or updates.

Example: Schedule a Daily Backup of K3s Data

Script: backup-k3s.sh

#!/bin/bash
BACKUP_DIR="/var/lib/rancher/k3s/server/db/backups"
BACKUP_FILE="k3s-backup-$(date +%Y-%m-%d-%H-%M-%S).db"

echo "Backing up K3s data..."
sudo cp $BACKUP_DIR/snapshot.db /backups/$BACKUP_FILE
echo "Backup completed: /backups/$BACKUP_FILE"

Automate with Cron:

crontab -e

Add the following line for daily backups at 2 am:

0 2 * * * /path/to/backup-k3s.sh

Purpose: Learn how to schedule recurring tasks with cron and automate backups.

8. Secure the K3s Environment

Focus on basic security configurations and policies as you learn.

Create a Pod Security Policy

Write a basic Pod Security Policy (PSP) for K3s to enforce non-root container execution.

Example YAML:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: restricted-psp
spec:
  privileged: false
  runAsUser:
    rule: MustRunAsNonRoot

Purpose: Practice creating Kubernetes manifests and learn how to secure your environment.

Final Thoughts and Next Steps

This clean, focused approach not only sets up your Rocky Linux system for K3s and Python-based workloads but also helps you gain hands-on experience with the Linux CLI, bash scripting, and automation through real-world tasks.

What’s next?

Continuously refine your scripts and make them more robust.
As you gain comfort, start incorporating more advanced Linux features (e.g., process management, networking).
Work towards automating the entire K3s environment, including monitoring, logging, and alerting.

You're absolutely right! Let’s include a few actionable examples to introduce more advanced Linux features, such as process management and networking, while keeping the learning process gradual and practical.

1. Process Management: Monitor and Control Running Processes

Understanding how to manage processes is fundamental in Linux. Let’s create a couple of bash scripts that help you monitor and control processes on your system.

Example: Monitor Active Processes

Script: monitor-processes.sh

#!/bin/bash
echo "Monitoring System Processes"
echo "---------------------------"

echo "Top 10 CPU-consuming processes:"
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu | head -n 10

echo
echo "Top 10 Memory-consuming processes:"
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head -n 10

Learning Outcomes:
- Use ps to view processes and understand the options for filtering and sorting by CPU or memory usage.
- Practice managing processes by identifying high-resource users.

Example: Kill a Process by Name

Script: kill-process.sh

#!/bin/bash

if [ -z "$1" ]; then
    echo "Usage: $0 <process_name>"
    exit 1
fi

PROCESS_NAME=$1
echo "Attempting to kill process: $PROCESS_NAME"

pkill $PROCESS_NAME && echo "Process '$PROCESS_NAME' terminated."

Learning Outcomes:
- Use pkill to terminate a process by name.
- Understand how to handle input arguments in bash scripts.

2. Advanced Networking: Network Configuration and Monitoring

Networking is key for edge compute workloads. Let’s dive into monitoring your network connections and controlling traffic with basic networking scripts.

Example: Monitor Network Usage

Script: monitor-network.sh

#!/bin/bash
echo "Network Usage Report"
echo "--------------------"

echo "Current network interfaces:"
ip link show

echo "Network traffic summary:"
ifconfig | grep -A 8 "eth0"

echo
echo "Routing table:"
netstat -rn

Learning Outcomes:
- Learn to use ifconfig and ip for network interface status.
- Display routing information using netstat.

Example: Block/Unblock IP Address Using Iptables

Script: block-ip.sh

#!/bin/bash

if [ -z "$1" ]; then
    echo "Usage: $0 <ip_address>"
    exit 1
fi

IP_ADDRESS=$1
echo "Blocking IP: $IP_ADDRESS"

sudo iptables -A INPUT -s $IP_ADDRESS -j DROP
echo "IP address $IP_ADDRESS has been blocked."

Unblock Script: unblock-ip.sh

#!/bin/bash

if [ -z "$1" ]; then
    echo "Usage: $0 <ip_address>"
    exit 1
fi

IP_ADDRESS=$1
echo "Unblocking IP: $IP_ADDRESS"

sudo iptables -D INPUT -s $IP_ADDRESS -j DROP
echo "IP address $IP_ADDRESS has been unblocked."

Learning Outcomes:
- Use iptables to block and unblock network traffic from specific IP addresses.
- Learn basic network security by controlling traffic at the firewall level.

3. Automate Network Health Checks

Automate network status checks by writing a script to ping external servers and display results. This is useful to test the connectivity from your edge devices.

Example: Ping Multiple Servers

Script: network-health.sh

#!/bin/bash
SERVERS=("8.8.8.8" "1.1.1.1" "google.com" "example.com")

echo "Pinging Servers..."

for SERVER in "${SERVERS[@]}"; do
    echo "Pinging $SERVER:"
    ping -c 4 $SERVER
    echo "------------------------------------"
done

Learning Outcomes:
- Use arrays in bash to handle multiple items (servers).
- Learn about network connectivity testing using ping.

4. Advanced Disk Usage and Resource Monitoring

As edge workloads may have limited resources, disk usage monitoring becomes crucial. Let’s script resource checks for disk space, memory, and CPU usage.

Example: Monitor Disk Space and Report Critical Usage

Script: disk-usage-report.sh

#!/bin/bash
THRESHOLD=80

echo "Checking Disk Usage..."
df -h | awk 'NR>1 {print $5 " " $1}' | while read output; do
  usep=$(echo $output | awk '{print $1}' | sed 's/%//')
  partition=$(echo $output | awk '{print $2}')
  if [ $usep -ge $THRESHOLD ]; then
    echo "CRITICAL: $partition is $usep% full."
  fi
done

Learning Outcomes:
- Use df to check disk usage.
- Apply basic conditional logic to report when usage crosses a defined threshold (80%).

5. Advanced Automation with Cron Jobs

After you’re comfortable with your scripts, you can automate more advanced monitoring and reporting using cron.

Example: Automate Disk Monitoring

Automate Disk Usage Check:
```
crontab -e
```
Add this cron job to run the disk usage check script every day at 6 am:
```
0 6 * * * /path/to/disk-usage-report.sh
```
- Learning Outcomes:
  - Learn how to use cron for recurring tasks.
  - Automate your monitoring and alerts, reducing manual checks.

Conclusion

This approach adds actionable examples to help you dive deeper into advanced Linux features such as process management, networking, and resource monitoring, all while integrating them into your current workflow. These tweaks provide you with hands-on practice in managing and automating system processes, monitoring network activity, and ensuring system health—all key skills for managing K3s and edge compute workloads effectively.

Absolutely! Here are additional CLI learning tips to enhance your mastery of Linux and shell scripting. These tips are designed to help you deepen your understanding of the command line, make your work more efficient, and improve your problem-solving skills.

Tip: Practice basic navigation commands until they become second nature:
- cd: Change directory.
- ls: List directory contents.
- pwd: Print current working directory.
- mkdir/rmdir: Create and remove directories.
Try this:
- Create a directory structure, navigate through it, and practice moving and renaming files:
```
mkdir -p ~/projects/k3s/python
cd ~/projects/k3s
mv python/ ~/projects/
```

2. Command History and Shortcuts

Tip: Use the command history to save time when repeating or modifying commands.
- Use the up and down arrow keys to scroll through previously executed commands.
- Use !! to re-run the last command or !<number> to run a specific command from history.
- Ctrl + r: Search through command history interactively.
Try this:
- Run multiple commands and practice recalling them using history and !<number>.
```
history
!45  # Re-run command number 45
```

3. Tab Completion and Wildcards

Tip: Use tab completion to autocomplete file names, paths, and commands, which will make you faster and reduce errors.
Wildcards: Use wildcards like * and ? to work with multiple files or patterns:
- *.txt: Match all files with .txt extension.
- file?.txt: Match files with one-character differences (e.g., file1.txt, file2.txt).

Try this:

Practice file matching and renaming using wildcards:

mv *.log ~/logs/  # Move all .log files to logs directory

4. Learn to Use `man` and `help`

Tip: Whenever you’re unsure how a command works, use man (manual pages) or --help:
- man ls: Get the manual for the ls command.
- ls --help: Get a quick overview of the command options.
Try this:
- Explore the options of a new command you haven’t used before, like find or tar:
```
man find
```

5. Chain Commands with Pipes and Redirection

Tip: Learn how to chain commands together to process data more effectively:
- Use | (pipe) to send the output of one command as input to another.
```
ps aux | grep python
```
- Use > to redirect output to a file, and >> to append.
```
df -h > disk-usage.txt  # Save disk usage output to a file
```
Try this:
- Combine pipes and redirection to filter and store information:
```
ps aux | grep k3s > k3s-processes.txt
```

6. Automate Tasks with Aliases

Tip: Define aliases for frequently used commands to save time.
- Open ~/.bashrc and add your aliases:
```
alias ll='ls -lah'
alias upd='sudo dnf update -y && sudo dnf upgrade -y'
```
- Reload the .bashrc:
```
source ~/.bashrc
```
Try this:
- Define your own alias for frequently run commands or scripts, like K3s or Python management tasks.

7. Learn to Use `grep`, `awk`, and `sed`

Tip: Master text processing tools like grep, awk, and sed for searching and manipulating text output:
- grep: Search for patterns in files or output.
```
grep "error" /var/log/syslog
```
- awk: Process and extract fields from structured data.
```
awk '{print $1, $5}' /path/to/file
```
- sed: Stream editor for text substitution.
```
sed 's/error/success/g' file.txt  # Replace 'error' with 'success' in the file
```
Try this:
- Extract useful information from logs or system commands using grep:
```
ps aux | grep k3s
```

8. Create More Complex Bash Scripts

Tip: Start combining loops, conditionals, and functions in your scripts to automate more complex tasks:
- For loops: Iterate over a list of items.
```
for file in *.txt; do
  echo "Processing $file"
done
```
- If statements: Add conditional logic.
```
if [ -f /path/to/file ]; then
  echo "File exists"
else
  echo "File not found"
fi
```

Try this:

Create a script that checks if a service is running and restarts it if necessary:

#!/bin/bash
if systemctl is-active --quiet k3s; then
  echo "K3s is running"
else
  echo "K3s is not running, restarting..."
  sudo systemctl restart k3s
fi

9. Explore `tmux` or `screen` for Multitasking

Tip: Use tmux or screen to manage multiple CLI sessions within a single terminal. This is especially useful when running long tasks or managing multiple remote connections.
Try this:
- Install and use tmux:
```
sudo dnf install tmux
tmux
```
- Split the screen and run different tasks in separate panes.

10. Learn Networking Commands

Tip: Gain familiarity with networking-related commands to troubleshoot connectivity:
- ping: Check connectivity to another host.
```
ping 8.8.8.8
```
- traceroute: See the path packets take to a destination.
```
traceroute google.com
```
- netstat: Check open network connections.
```
netstat -tuln
```
Try this:
- Run netstat or ss to inspect which ports are open and which services are listening:
```
netstat -tuln
```

11. Practice Permissions and Ownership

Tip: Understanding file permissions is critical for securing your system. Learn the basics of chmod, chown, and chgrp.
- chmod: Change file permissions.
```
chmod 755 script.sh
```
- chown: Change file ownership.
```
chown user:group file.txt
```
Try this:
- Modify file permissions on a script to make it executable only by you:
```
chmod 700 myscript.sh
```

12. Learn Job Control: Background Jobs and Process Management

Tip: Use job control features to manage processes in the foreground and background:
- Run a command in the background by appending &:
```
long_process &  # Runs long_process in the background
```
- Use jobs to list background jobs and fg to bring one to the foreground:
```
jobs
fg %1  # Bring job 1 to foreground
```
Try this:
- Start a long-running command in the background and monitor it using jobs.

Conclusion

By incorporating these CLI learning tips, you’ll not only increase your proficiency with Linux commands but also develop more robust skills in automation, scripting, and problem-solving. As you continue setting up and managing your system, these techniques will help you work more efficiently and tackle increasingly complex tasks with confidence.

Make sure to practice these techniques daily and gradually incorporate them into your workflows. The more you explore the Linux command line, the more intuitive it will become.

Learning Roadmap for K3s, Edge Computing, and Advanced Networking

Networking Basics

TCP/IP
- Understanding IP addressing
- Subnetting and Routing
DNS
- DNS Resolution Process
- Configuring DNS Servers
HTTP/S
- HTTP Request/Response Cycle
- SSL/TLS and Secure Communication
Network Protocols and Architecture
- OSI and TCP/IP Models
- Common Network Protocols (DHCP, SSH, FTP)

Kubernetes Core Concepts

Kubernetes Architecture
- Master and Node Components
- Control Plane and Worker Nodes
Pods, Deployments, Services
- Lifecycle of a Pod
- Creating and Managing Deployments
- Service Types and Load Balancing
Persistent Storage and Networking
- Volumes and Persistent Volumes
- Storage Classes and Dynamic Provisioning
- Kubernetes Networking Model

K3s Specific Learning

Installation and Configuration
- Setting Up a K3s Cluster
- Configuring K3s on Different Environments
Cluster Management
- Node Management
- Backup and Restore Strategies
Resource Optimization
- Resource Limits and Requests
- Autoscaling in K3s

Edge Computing and IoT with K3s

K3s in Edge Scenarios
- Deployment Strategies for Edge Computing
- Managing Low-Resource Environments
Integrating IoT Devices with Kubernetes
- Connecting IoT Devices to K3s
- Security Considerations in IoT and Kubernetes

Advanced Kubernetes Networking

Deep Dive into CNI (Container Network Interface)
- Understanding CNI Plugins
- Custom Network Solutions in Kubernetes
Network Policies
- Implementing Network Policies
- Securing Pod-to-Pod Communication
Service Meshes (Istio, Linkerd)
- Introduction to Istio and Linkerd
- Traffic Management and Observability

Hands-On and Practical Implementation

Setting up K3s Clusters
- Creating High-Availability Clusters
- Disaster Recovery and Failover
Real-World Projects and Scenarios
- Implementing a CI/CD Pipeline with K3s
- Deploying a Multi-Service Application

21 KiB Raw Blame History Unescape Escape

1. Update and Secure the System

System Updates

SELinux Configuration

2. Install Essential System Tools

Install Development Tools and Utilities

Firewall Configuration

3. Install Python and Set Up a Virtual Environment

Install Python 3 and Development Tools

Set Up a Python Virtual Environment

4. Install and Configure K3s

Install K3s

Configure K3s CLI

5. Learn Bash Scripting through Automation

Automate System Updates

Automate Python Environment Setup

Automate K3s Installation and Configuration

6. Monitor the System

7. Learn Automation through Cron Jobs

Example: Schedule a Daily Backup of K3s Data

8. Secure the K3s Environment

Create a Pod Security Policy

Final Thoughts and Next Steps

1. Process Management: Monitor and Control Running Processes

Example: Monitor Active Processes

Example: Kill a Process by Name

2. Advanced Networking: Network Configuration and Monitoring

Example: Monitor Network Usage

Example: Block/Unblock IP Address Using Iptables

3. Automate Network Health Checks

Example: Ping Multiple Servers

4. Advanced Disk Usage and Resource Monitoring

Example: Monitor Disk Space and Report Critical Usage

5. Advanced Automation with Cron Jobs

Example: Automate Disk Monitoring

Conclusion

1. Master Command-Line Navigation

2. Command History and Shortcuts

3. Tab Completion and Wildcards

4. Learn to Use man and help

5. Chain Commands with Pipes and Redirection

6. Automate Tasks with Aliases

7. Learn to Use grep, awk, and sed

8. Create More Complex Bash Scripts

9. Explore tmux or screen for Multitasking

10. Learn Networking Commands

11. Practice Permissions and Ownership

12. Learn Job Control: Background Jobs and Process Management

Conclusion

Learning Roadmap for K3s, Edge Computing, and Advanced Networking

Networking Basics

Kubernetes Core Concepts

K3s Specific Learning

Edge Computing and IoT with K3s

Advanced Kubernetes Networking

Hands-On and Practical Implementation

21 KiB

Raw Blame History

4. Learn to Use `man` and `help`

7. Learn to Use `grep`, `awk`, and `sed`

9. Explore `tmux` or `screen` for Multitasking