The awk command can make it easy to remove duplicate characters from a string even when those characters aren’t sequential, especially when the process is turned into a script. First, the awk command ...

awk does a lot of common work for you when you use it to process text files. It reads files a record at a time. Normally, a record is a single line. Then it splits the line on fields using whitespace, ...

A key point to these tools (from a practical point ) is that they are very memory efficient and allow work on just a "line".