the_information_nexus/tech_docs/linux/linux-troubleshooting.md

Certainly! Here's a concise, outlined guide focusing on troubleshooting within network, storage, and user stacks on Linux systems, incorporating relevant terms, commands, log locations, and features for effective diagnostics.

Linux Troubleshooting Guide Outline

1. Network Stack Troubleshooting

• Initial Checks
  • ping localhost and ping google.com for basic connectivity.
  • traceroute google.com to trace packet routing.
• Network Configuration
  • ip addr show for interface statuses.
  • nslookup google.com for DNS resolution.
• Port and Service Availability
  • sudo netstat -tulnp for active listening ports and services.
  • sudo nmap -sT localhost to identify open ports on the local machine.
• Logs and Monitoring
  • General network errors: /var/log/syslog (grep for "network").
  • Service-specific issues: e.g., /var/log/apache2/error.log.

2. Storage Stack Troubleshooting

• Disk Space
  • df -h for filesystem disk usage.
  • du -h /var | sort -hr | head -10 for top disk space consumers.
• Disk Health
  • sudo smartctl -a /dev/sda for disk health (Smartmontools).
• I/O Performance
  • iostat -xm 2 for I/O stats.
  • vmstat 1 10 for memory, process, and I/O statistics.
• Filesystem Integrity
  • sudo fsck /dev/sdX1 (ensure unmounted) for filesystem checks.

3. User Stack Troubleshooting

• Login Issues
  • sudo grep 'Failed password' /var/log/auth.log for failed logins.
  • Check user permissions with ls -l /home/username/.
• Resource Utilization
  • top or htop for real-time process monitoring.
  • ulimit -a for user resource limits.
• User-Specific Logs
  • Application logs, e.g., sudo tail -f /path/to/app/log.log.
• Session Management
  • who and last for login sessions and activity.

4. Creating a Definitive Diagnosis

• Correlation and Baseline Comparison: Use monitoring tools to compare current states against known baselines.
• Advanced Diagnostics: Employ strace for syscall tracing, tcpdump for packet analysis, and perf for performance issues.

5. Tools and Commands for In-depth Analysis

• System and Service Status: systemctl status <service>.
• Performance Monitoring: atop, sar, and Grafana with Prometheus for historical data.
• Configuration Checks: Verify settings in /etc/sysconfig, /etc/network, and service-specific configuration files.
• Security and Permissions: Review /var/log/secure or use auditd for auditing access and changes.

This outline structures the troubleshooting process into distinct areas, providing a logical approach to diagnosing and resolving common Linux system issues. By following these steps and utilizing the outlined tools and commands, administrators can methodically identify and address problems within their systems.

---

Creating a focused reference guide for advanced log filtering and analysis, this guide will cover powerful and practical examples using grep, awk, sed, and tail. This guide is intended for experienced Linux users who are familiar with the command line and seek to refine their skills in parsing and analyzing log files for troubleshooting and monitoring purposes.

Log Filtering and Analysis Reference Guide

1. Using grep for Basic Searches

• Filter Logs by Date:

  grep "2024-03-16" /var/log/syslog
  

  This command filters entries from March 16, 2024, in the syslog.

• Search for Error Levels:

  grep -E "error|warn|critical" /var/log/syslog
  

  Use -E for extended regular expressions to match multiple patterns, useful for finding various error levels.

2. Advanced Text Processing with awk

• Extract Specific Fields:

  awk '/Failed password/ {print $1, $2, $3, $(NF-5), $(NF-3)}' /var/log/auth.log
  

  This example extracts the date, time, and IP address from failed SSH login attempts. NF represents the number of fields in a line, making $(NF-5) and $(NF-3) select fields relative to the end of the line.

• Summarize Access by IP Address:

  awk '{print $NF}' /var/log/apache2/access.log | sort | uniq -c | sort -nr
  

  Here, $NF extracts the last field (typically the IP address in access logs), uniq -c counts occurrences, and sort -nr sorts numerically in reverse for a descending list of IP addresses by access count.

3. Stream Editing with sed

• Remove Specific Lines:

  sed '/debug/d' /var/log/syslog
  

  This command deletes lines containing "debug" from the output, useful for excluding verbose log levels.

• Anonymize IP Addresses:

  sed -r 's/([0-9]{1,3}\.){3}[0-9]{1,3}/[REDACTED IP]/g' /var/log/apache2/access.log
  

  Using a regular expression, this replaces IP addresses with "[REDACTED IP]" for privacy in shared analysis.

4. Real-time Monitoring with tail -f and grep

• Watch for Specific Log Entries in Real-time:

  tail -f /var/log/syslog | grep "kernel"
  

  This monitors syslog in real-time for new entries containing "kernel", combining tail -f with grep for focused live logging.

Combining Tools for Enhanced Analysis

• Identify Frequent Access by IP with Timestamps:

  awk '{print $1, $2, $4, $NF}' /var/log/apache2/access.log | sort | uniq -c | sort -nr | head
  

  This command combines awk to extract date, time, and IP, then sort and uniq -c to count and sort access attempts, using head to display the top results.

• Extract and Sort Errors by Frequency:

  grep "error" /var/log/syslog | awk '{print $5}' | sort | uniq -c | sort -nr
  

  Filter for "error" messages, extract the application or process name (assuming it's the fifth field), count occurrences, and sort them by frequency.

This guide provides a foundation for powerful log analysis techniques. Experimentation and adaptation to specific log formats and requirements will further enhance your proficiency. For deeper exploration, consider the man pages (man grep, man awk, man sed, man tail) and other comprehensive resources available online.

---

Comprehensive Linux Troubleshooting Tools Guide

This guide provides an overview of key packages and their included tools for effective troubleshooting in Linux environments, specifically tailored for RHEL and Debian-based distributions.

Tools Commonly Included in Most Linux Distributions

• GNU Coreutils: A collection of basic file, shell, and text manipulation utilities. Key tools include:

  • df: Reports file system disk space usage.
  • du: Estimates file space usage.

• Util-linux: A suite of essential utilities for system administration. Key tools include:

  • dmesg: Examines or controls the kernel ring buffer.

• IPUtils: Provides tools for network diagnostics. Key tools include:

  • ping: Checks connectivity with hosts.
  • traceroute: Traces the route taken by packets to reach a network host.

RHEL (Red Hat Enterprise Linux) and Derivatives

• Procps-ng: Offers utilities that provide information about processes. Key tools include:

  • top: Displays real-time system summary and task list.
  • vmstat: Reports virtual memory statistics.

• Net-tools: A collection of programs for controlling the network subsystem of the Linux kernel. Includes:

  • netstat: Shows network connections, routing tables, and interface statistics.

• IPRoute: Modern replacement for net-tools. Key utility:

  • ss: Investigates sockets.

• Sysstat: Contains utilities to monitor system performance and usage. Notable tools:

  • iostat: Monitors system I/O device loading.
  • sar: Collects and reports system activity information.

• EPEL Repository (for tools not included by default):

  • htop: An interactive process viewer, enhanced version of top.

Debian and Derivatives

• Procps: Similar to procps-ng in RHEL, it provides process monitoring utilities. Key tools include:

  • top: For real-time process monitoring.
  • vmstat: For reporting virtual memory statistics.

• Net-tools: As with RHEL, includes essential networking tools like netstat.

• IPRoute2: A collection of utilities for controlling and monitoring various aspects of networking in the Linux kernel, featuring:

  • ss: A utility for inspecting sockets.

• Sysstat: Similar to its usage in RHEL, includes tools like iostat and sar for performance monitoring.

Conclusion

This guide emphasizes the importance of familiarizing oneself with the tools included in standard Linux packages. Whether you are operating in a RHEL or Debian-based environment, understanding the capabilities of these tools and their respective packages is crucial for effective troubleshooting and system monitoring.