Pipes and redirection

Written Explanation

Output goes somewhere

Every command has a place where its normal output goes. By default, that place is the screen. In Unix terms this is standard output. Errors travel on a separate channel, standard error, which also shows on screen by default. PowerShell has the same idea with named streams: the Success stream (like standard output) and the Error stream.

Because output is a stream and not a fixed picture on a screen, it can be pointed somewhere else.

Redirection with >

The operator > sends output to a file.

$ echo "first line" > notes.txt
$ cat notes.txt
first line

Nothing was printed when you ran the first command: the output went into notes.txt. If the file did not exist, it was created. Both shells work this way, and the same command works in PowerShell (Get-Content notes.txt shows the file; so does cat).

The same works for any command:

$ ls > list.txt
PS> Get-ChildItem > list.txt

Warning: > replaces the file's old contents, without asking. The Bash manual says the file is truncated to zero size if it already exists, and Microsoft's PowerShell documentation says > overwrites the current contents without warning. A second echo "second" > notes.txt leaves only second in the file.

Appending with >>

>> adds to the end of a file and keeps what was there. If the file does not exist, it is created.

$ echo "first line" > notes.txt
$ echo "second line" >> notes.txt
$ cat notes.txt
first line
second line

Rule of thumb: > for "start this file fresh", >> for "keep adding", as in a log.

Redirecting errors

Error messages go to the error channel, so ls missing > out.txt still prints the error on screen and out.txt stays empty. To capture errors, put 2 in front: ls missing 2> errors.txt. This works in Bash and also in PowerShell, where 2 is the number of the Error stream. You will use it rarely at first, but it explains why an error "ignored" your redirection.

Pipes with |

The pipe | connects the output of the command on its left to the input of the command on its right.

$ ls | sort

Read it as: "list the folder, then give that list to sort". The pipe does not use a file in between. A pipeline can have many steps: command1 | command2 | command3, processed left to right.

Pipes work in both shells. Microsoft's pipeline documentation defines a pipeline as commands connected by |, each sending its result to the next. The standard Unix idea is the same; I did not re-read the Bash manual's pipeline section in this session.

Filtering: grep and Select-String

The most useful partner for a pipe is a filter that keeps only the lines containing a word.

$ ls | grep log
app.log
log-notes.txt

PowerShell's equivalent is Select-String. A small catch: to give it plain lines of text, list names only with -Name:

PS> Get-ChildItem -Name | Select-String log

grep and Select-String also read files directly: grep error app.log and Select-String error app.log search app.log for lines containing error. Select-String prefixes each matching line with the file name and line number when it reads from a file. grep is case-sensitive by default; grep -i ignores case. Select-String is not case-sensitive by default.

Two shells, two kinds of pipe

In Bash, a pipe carries text. Each command in the chain reads lines of text and writes lines of text. In PowerShell, a pipe carries objects: structured things with named properties such as Name and Length. This is why PowerShell can sort files by size without parsing text:

PS> Get-ChildItem | Sort-Object Length

Microsoft's documentation shows exactly this pattern: get items, filter, sort by a property such as Length. In Bash you use options of the command instead: ls -lS lists a detailed view sorted by size, largest first. To reverse it in PowerShell, add -Descending: Sort-Object Length -Descending.

You do not need to master objects now. The practical rule is: in PowerShell, sort and filter by property name; in Bash, filter text with grep and use the command's own options to order.

Counting with a pipe

Pipes also count. In Bash, wc -l counts lines:

$ ls | wc -l
7

In PowerShell, Measure-Object counts the items it receives:

PS> Get-ChildItem -Name | Measure-Object

and its output includes a Count line.

Combining a pipe with redirection

Pipes and > work together. The pipe goes first and redirection last, because redirection captures the final output:

$ ls -lS | grep txt > big-text-files.txt
PS> Get-ChildItem | Sort-Object Length -Descending > sizes.txt

A word about file encoding in PowerShell

PowerShell 7 writes redirected output as UTF-8 without a byte-order mark, according to Microsoft's documentation. Older Windows PowerShell may use a different default encoding, so a file written with > might look garbled in another tool. If that happens, Out-File -Encoding utf8 file.txt lets you choose. I did not verify the exact default of every older version.