Add tech_docs/linux/sed.md
This commit is contained in:
210
tech_docs/linux/sed.md
Normal file
210
tech_docs/linux/sed.md
Normal file
@@ -0,0 +1,210 @@
|
|||||||
|
# **The Sed Mastery Guide: From Basics to Advanced Text Alchemy**
|
||||||
|
|
||||||
|
## **Table of Contents**
|
||||||
|
1. [Sed Fundamentals](#1-sed-fundamentals)
|
||||||
|
2. [Pattern Addressing](#2-pattern-addressing)
|
||||||
|
3. [Advanced Substitutions](#3-advanced-substitutions)
|
||||||
|
4. [Multi-Command Scripts](#4-multi-command-scripts)
|
||||||
|
5. [Branching & Flow Control](#5-branching--flow-control)
|
||||||
|
6. [Real-World Recipes](#6-real-world-recipes)
|
||||||
|
7. [Performance & Limitations](#7-performance--limitations)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## **1. Sed Fundamentals**
|
||||||
|
|
||||||
|
### **Core Syntax**
|
||||||
|
```bash
|
||||||
|
sed [OPTIONS] 'COMMAND' file.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### **Essential Operations**
|
||||||
|
| Command | Description | Example |
|
||||||
|
|---------|-------------|---------|
|
||||||
|
| `s/old/new/` | Substitute first occurrence | `sed 's/foo/bar/'` |
|
||||||
|
| `s/old/new/g` | Substitute globally | `sed 's/foo/bar/g'` |
|
||||||
|
| `p` | Print pattern space | `sed -n '1,5p'` |
|
||||||
|
| `d` | Delete lines | `sed '/error/d'` |
|
||||||
|
| `i\text` | Insert before line | `sed '3i\New Line'` |
|
||||||
|
| `a\text` | Append after line | `sed '$a\END'` |
|
||||||
|
|
||||||
|
### **Common Flags**
|
||||||
|
```bash
|
||||||
|
sed -i '' file.txt # In-place edit (BSD/macOS)
|
||||||
|
sed -i file.txt # In-place edit (Linux)
|
||||||
|
sed -n '1,5p' file.txt # Print only lines 1-5
|
||||||
|
sed -e 'cmd1' -e 'cmd2' # Multiple commands
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## **2. Pattern Addressing**
|
||||||
|
|
||||||
|
### **Line Number Targeting**
|
||||||
|
```bash
|
||||||
|
sed '5s/foo/bar/' # Only line 5
|
||||||
|
sed '1,10d' # Delete lines 1-10
|
||||||
|
sed '$d' # Delete last line
|
||||||
|
```
|
||||||
|
|
||||||
|
### **Regex Patterns**
|
||||||
|
```bash
|
||||||
|
sed '/^#/d' # Delete comment lines
|
||||||
|
sed '/start/,/end/d' # Delete range between patterns
|
||||||
|
sed '/pattern/!d' # Keep only matching lines
|
||||||
|
```
|
||||||
|
|
||||||
|
### **Special Addresses**
|
||||||
|
```bash
|
||||||
|
sed -n '1~2p' # Every 2nd line starting at 1
|
||||||
|
sed -n '2,${p;n}' # Every other line from line 2
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## **3. Advanced Substitutions**
|
||||||
|
|
||||||
|
### **Capture Groups**
|
||||||
|
```bash
|
||||||
|
echo "Hello World" | sed 's/\(Hello\) \(World\)/\2 \1/'
|
||||||
|
# Output: World Hello
|
||||||
|
```
|
||||||
|
|
||||||
|
### **Custom Delimiters**
|
||||||
|
```bash
|
||||||
|
sed 's|/usr/local|/opt|g' # Use | instead of /
|
||||||
|
```
|
||||||
|
|
||||||
|
### **Case Conversion**
|
||||||
|
```bash
|
||||||
|
sed 's/.*/\L&/' # Lowercase entire line
|
||||||
|
sed 's/.*/\U&/' # Uppercase entire line
|
||||||
|
```
|
||||||
|
|
||||||
|
### **Conditional Replacement**
|
||||||
|
```bash
|
||||||
|
sed '/error/s/foo/bar/' # Only substitute if line contains "error"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## **4. Multi-Command Scripts**
|
||||||
|
|
||||||
|
### **Command Files**
|
||||||
|
```bash
|
||||||
|
# script.sed
|
||||||
|
/error/ {
|
||||||
|
s/foo/bar/g
|
||||||
|
p
|
||||||
|
}
|
||||||
|
|
||||||
|
sed -f script.sed file.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### **Inline Multi-Commands**
|
||||||
|
```bash
|
||||||
|
sed -e 's/foo/bar/' -e '/baz/d' file.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### **Hold Space Operations**
|
||||||
|
```bash
|
||||||
|
sed -n '1h; 2,${H;g; s/\n/ /g; p}' # Combine lines 1-N
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## **5. Branching & Flow Control**
|
||||||
|
|
||||||
|
### **Basic Branching**
|
||||||
|
```bash
|
||||||
|
sed '/start/{:loop; N; /end/!b loop; s/start.*end/replacement/}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### **Labels & Loops**
|
||||||
|
```bash
|
||||||
|
sed ':top; /pattern/{N; b top}' # Accumulate until pattern fails
|
||||||
|
```
|
||||||
|
|
||||||
|
### **Conditional Branch**
|
||||||
|
```bash
|
||||||
|
sed '/error/{s/foo/bar/; t; s/foo/baz/}' # If first sub worked, skip second
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## **6. Real-World Recipes**
|
||||||
|
|
||||||
|
### **CSV Processing**
|
||||||
|
```bash
|
||||||
|
sed 's/^\([^,]*\),\([^,]*\)/\2,\1/' data.csv # Swap first two columns
|
||||||
|
```
|
||||||
|
|
||||||
|
### **Log File Cleaning**
|
||||||
|
```bash
|
||||||
|
sed -E '/^(DEBUG|TRACE)/d; s/[0-9]{4}-[0-9]{2}-[0-9]{2}/DATE_REDACTED/g' app.log
|
||||||
|
```
|
||||||
|
|
||||||
|
### **HTML/XML Processing**
|
||||||
|
```bash
|
||||||
|
sed '/<script/,/<\/script>/d' page.html # Remove script tags
|
||||||
|
```
|
||||||
|
|
||||||
|
### **Code Refactoring**
|
||||||
|
```bash
|
||||||
|
# Rename function and all calls
|
||||||
|
sed -i 's/old_func_name/new_func_name/g' *.py
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## **7. Performance & Limitations**
|
||||||
|
|
||||||
|
### **Large File Handling**
|
||||||
|
```bash
|
||||||
|
sed -n '1,1000000p' huge.log > sample.log # Extract first million lines
|
||||||
|
```
|
||||||
|
|
||||||
|
### **When Not to Use Sed**
|
||||||
|
- JSON/XML with nested structures (use `jq/xmllint`)
|
||||||
|
- Multi-line records spanning variable lines (use `awk`)
|
||||||
|
- Binary file editing (use `xxd + sed + xxd -r`)
|
||||||
|
|
||||||
|
### **Performance Tips**
|
||||||
|
```bash
|
||||||
|
LC_ALL=C sed ... # 2-3x speedup for ASCII
|
||||||
|
sed ':a;N;$!ba;s/\n//g' # Slurp entire file (careful with memory)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## **Pro-Level Techniques**
|
||||||
|
|
||||||
|
### **In-Place Editing Backup**
|
||||||
|
```bash
|
||||||
|
sed -i.bak 's/foo/bar/' file.txt # Creates file.txt.bak
|
||||||
|
```
|
||||||
|
|
||||||
|
### **Non-Greedy Matching**
|
||||||
|
```bash
|
||||||
|
sed 's/foo\(.*\?\)bar/\1/' # Perl-style regex (requires -r/-E)
|
||||||
|
```
|
||||||
|
|
||||||
|
### **Random Line Selection**
|
||||||
|
```bash
|
||||||
|
sed -n "$RANDOM"' {p;q}' # Get one random line
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## **Sed + Awk Power Combo**
|
||||||
|
```bash
|
||||||
|
# Extract columns from lines matching pattern
|
||||||
|
sed -n '/pattern/p' file | awk '{print $2,$4}'
|
||||||
|
```
|
||||||
|
|
||||||
|
For complex text transformations, consider combining sed with:
|
||||||
|
- `awk` for columnar data
|
||||||
|
- `perl` for advanced regex
|
||||||
|
- `grep` for filtering
|
||||||
|
|
||||||
|
**Need a custom sed solution?** Provide your text transformation challenge!
|
||||||
Reference in New Issue
Block a user