awk

Removes duplication from file or string, must not be near to each other

$ awk '!a[$0]++' ./filename

Sample

$ cat sedtest
asdf bla fu aaaaa
asdf bla fu qwerqwer
asdf bla fu aasdfaaaa
asdf bla fu asdf
asdf bla fu aaaaa
asdf bla fu aaaaa
asdf bla fu aaaaa

$ cat sedtest | awk '!a[$0]++'
asdf bla fu aaaaa
asdf bla fu qwerqwer
asdf bla fu aasdfaaaa
asdf bla fu asdf

Duplicates are only removed on stdou and not on file base

$ awk '!a[$0]++' sedtest
asdf bla fu aaaaa
asdf bla fu qwerqwer
asdf bla fu aasdfaaaa
asdf bla fu asdf