the_information_nexus/tech_docs/linux/sed.md

# **The Sed Mastery Guide: From Basics to Advanced Text Alchemy**

## **Table of Contents**
1. [Sed Fundamentals](#1-sed-fundamentals)
2. [Pattern Addressing](#2-pattern-addressing)
3. [Advanced Substitutions](#3-advanced-substitutions)
4. [Multi-Command Scripts](#4-multi-command-scripts)
5. [Branching & Flow Control](#5-branching--flow-control)
6. [Real-World Recipes](#6-real-world-recipes)
7. [Performance & Limitations](#7-performance--limitations)

---

## **1. Sed Fundamentals**

### **Core Syntax**
```bash
sed [OPTIONS] 'COMMAND' file.txt
```

### **Essential Operations**
| Command | Description | Example |
|---------|-------------|---------|
| `s/old/new/` | Substitute first occurrence | `sed 's/foo/bar/'` |
| `s/old/new/g` | Substitute globally | `sed 's/foo/bar/g'` |
| `p` | Print pattern space | `sed -n '1,5p'` |
| `d` | Delete lines | `sed '/error/d'` |
| `i\text` | Insert before line | `sed '3i\New Line'` |
| `a\text` | Append after line | `sed '$a\END'` |

### **Common Flags**
```bash
sed -i '' file.txt       # In-place edit (BSD/macOS)
sed -i file.txt          # In-place edit (Linux)
sed -n '1,5p' file.txt   # Print only lines 1-5
sed -e 'cmd1' -e 'cmd2'  # Multiple commands
```

---

## **2. Pattern Addressing**

### **Line Number Targeting**
```bash
sed '5s/foo/bar/'        # Only line 5
sed '1,10d'              # Delete lines 1-10
sed '$d'                 # Delete last line
```

### **Regex Patterns**
```bash
sed '/^#/d'              # Delete comment lines
sed '/start/,/end/d'     # Delete range between patterns
sed '/pattern/!d'        # Keep only matching lines
```

### **Special Addresses**
```bash
sed -n '1~2p'            # Every 2nd line starting at 1
sed -n '2,${p;n}'        # Every other line from line 2
```

---

## **3. Advanced Substitutions**

### **Capture Groups**
```bash
echo "Hello World" | sed 's/\(Hello\) \(World\)/\2 \1/'
# Output: World Hello
```

### **Custom Delimiters**
```bash
sed 's|/usr/local|/opt|g'  # Use | instead of /
```

### **Case Conversion**
```bash
sed 's/.*/\L&/'           # Lowercase entire line
sed 's/.*/\U&/'           # Uppercase entire line
```

### **Conditional Replacement**
```bash
sed '/error/s/foo/bar/'   # Only substitute if line contains "error"
```

---

## **4. Multi-Command Scripts**

### **Command Files**
```bash
# script.sed
/error/ {
  s/foo/bar/g
  p
}

sed -f script.sed file.txt
```

### **Inline Multi-Commands**
```bash
sed -e 's/foo/bar/' -e '/baz/d' file.txt
```

### **Hold Space Operations**
```bash
sed -n '1h; 2,${H;g; s/\n/ /g; p}'  # Combine lines 1-N
```

---

## **5. Branching & Flow Control**

### **Basic Branching**
```bash
sed '/start/{:loop; N; /end/!b loop; s/start.*end/replacement/}'
```

### **Labels & Loops**
```bash
sed ':top; /pattern/{N; b top}'  # Accumulate until pattern fails
```

### **Conditional Branch**
```bash
sed '/error/{s/foo/bar/; t; s/foo/baz/}'  # If first sub worked, skip second
```

---

## **6. Real-World Recipes**

### **CSV Processing**
```bash
sed 's/^\([^,]*\),\([^,]*\)/\2,\1/' data.csv  # Swap first two columns
```

### **Log File Cleaning**
```bash
sed -E '/^(DEBUG|TRACE)/d; s/[0-9]{4}-[0-9]{2}-[0-9]{2}/DATE_REDACTED/g' app.log
```

### **HTML/XML Processing**
```bash
sed '/<script/,/<\/script>/d' page.html  # Remove script tags
```

### **Code Refactoring**
```bash
# Rename function and all calls
sed -i 's/old_func_name/new_func_name/g' *.py
```

---

## **7. Performance & Limitations**

### **Large File Handling**
```bash
sed -n '1,1000000p' huge.log > sample.log  # Extract first million lines
```

### **When Not to Use Sed**
- JSON/XML with nested structures (use `jq/xmllint`)
- Multi-line records spanning variable lines (use `awk`)
- Binary file editing (use `xxd + sed + xxd -r`)

### **Performance Tips**
```bash
LC_ALL=C sed ...        # 2-3x speedup for ASCII
sed ':a;N;$!ba;s/\n//g' # Slurp entire file (careful with memory)
```

---

## **Pro-Level Techniques**

### **In-Place Editing Backup**
```bash
sed -i.bak 's/foo/bar/' file.txt  # Creates file.txt.bak
```

### **Non-Greedy Matching**
```bash
sed 's/foo\(.*\?\)bar/\1/'  # Perl-style regex (requires -r/-E)
```

### **Random Line Selection**
```bash
sed -n "$RANDOM"' {p;q}'    # Get one random line
```

---

## **Sed + Awk Power Combo**
```bash
# Extract columns from lines matching pattern
sed -n '/pattern/p' file | awk '{print $2,$4}'
```

For complex text transformations, consider combining sed with:
- `awk` for columnar data
- `perl` for advanced regex
- `grep` for filtering

**Need a custom sed solution?** Provide your text transformation challenge!