Hack 24. Uniq Command Examples

Uniq command is mostly used in combination with sort command, as uniq removes duplicates only from a sorted file. i.e In order for uniq to work, all the duplicate entries should be in the adjacent lines. Following are some common examples.

1. Remove Duplicates from a file using uniq command

When you have an employee file with duplicate entries, you can do the following to remove duplicates.

$ sort namesd.txt | uniq

$ sort –u namesd.txt

2. Display Duplicate Row Count Using Uniq

If you want to know how many lines are duplicates, do the following. The first field in the following examples indicates how many duplicates where found for that particular line. So, in this example the lines beginning with Alex and Emma were found twice in the namesd.txt file.

$ sort namesd.txt | uniq –c

      2 Alex Jason:200:Sales
      2 Emma Thomas:100:Marketing
      1 Madison Randy:300:Product Development
      1 Nisha Singh:500:Sales
      1 Sanjay Gupta:400:Support

3. Display Only Duplicate Rows Using Uniq

The following displays only the entries that are duplicates.

$ sort namesd.txt | uniq –cd

      2 Alex Jason:200:Sales
      2 Emma Thomas:100:Marketing

Comments on this entry are closed.

  • steve shiflett October 12, 2012, 10:54 am

    …and suppose you are interested in only the first field of a | delimited file – as that is where the key is.

  • Kunal July 24, 2013, 8:00 am

    Hi All,

    i want to merge two files kun3.txt & kun1.txt and i am able achive this using paste command.
    But i want the will contained same account name in column 2 and 4

    OUTPUT SEE 4th line is not match with account name some one can help me it fix this.
    i want column 2 and column 4 will contained same account name

    Finale out put

    37009 prod 29449 prod
    4929 wamcopnt 4920 wamcopnt
    3182 bcrshpnt 2687 pimcopnt
    2687 pimcopnt 1402 baram1

    Out put of file kun3.txt

    XXXXXXXX # head kun3.txt
    37009 prod
    4929 wamcopnt
    3182 bcrshpnt
    2687 pimcopnt
    2372 brsindex
    1955 jpmorganpnt
    1578 putnampnt
    1402 baram1
    1356 imsi3
    1191 nlipnt

    Out put of file kun1.txt

    XXXXXXXX # head kun1.txt
    29449 prod
    4920 wamcopnt
    2687 pimcopnt
    1402 baram1
    1191 nlipnt
    1058 putnampnt
    1002 deshawnylct
    970 campbscs
    916 dbnypnt
    912 trwprpnt
    XXXXXX#

    Out put of file paste command

    xxxxxxxxx # paste /tmp/kun3.txt /tmp/kun1.txt | pr -t -e10 | head
    37009 prod 29449 prod
    4929 wamcopnt 4920 wamcopnt
    3182 bcrshpnt 2687 pimcopnt
    2687 pimcopnt 1402 baram1
    2372 brsindex 1191 nlipnt
    1955 jpmorganpnt 1058 putnampnt
    1578 putnampnt 1002 deshawnylct
    1402 baram1 970 campbscs
    1356 imsi3 916 dbnypnt
    1191 nlipnt 912 trwprpnt