find duplicate record in Text file

I have text file with this kind of structure

mv /XXX/20000/XXX-18245 /XXX/20000/XXX-28042
mv /XXX/10000/XXX-9942 /XXX/10000/XXX-18166
mv /XXX/10000/XXX-9962 /XXX/10000/XXX-18189
mv /XXX/20000/XXX-10007 /XXX/20000/XXX-18245

I would like to find the duplicate XXX-18245 record in first and fourth rows

1

3 Answers

Using Notepad++


  • Ctrl+F
  • Find what: XXX-(\d+)[\s\S]+?\K\b\1\b
  • CHECK Wrap around
  • CHECK Regular expression
  • UNCHECK . matches newline
  • Find next

Explanation:

XXX- # literally XXX-
(\d+) # group 1, 1 or more digits
[\s\S]+? # 1 or more any character including linebreaks, not greedy
\K # forget all we have seen until this position
\b # word boundary, make sure to match the exact same number
\1 # backreference to group 1
\b # word boundary, make sure to match the exact same number

Screenshot:

enter image description here

2

In notepad++ you can use a regular expression like below, it'll highlight the whole text from beginning of first match to the end of second one.

  • press CTRL+F
  • for find what type (\/XXX[^ ]+)( .*)\1
  • make sure to check both "regular expression" and ". matches newline"
  • press "find next"

enter image description here

4

Will you consider using PowerShell? If you don't know how to run PowerShell, press Win+R to bring up Run menu and type PowerShell then Enter, I would recommend pinning PowerShell to taskbar for easy access. If you are using Linux, just search PowerShell 7.1 and download and install and run, it is foss and cross-platform.

Then, once PowerShell is running(and accepting commands), copy and paste the following code:

Get-Content "$txt" | Select-String -Pattern "XXX-18245" -AllMatches | Foreach-Object {$_.Matches}

You have to either replace the $txt with the full path of the txt file (to do that, shift+rmb then select copy as path on Windows, if you are using Linux, just ignore this tutorial) or assign the path to the $txt first, by running $txt="path"(use the actual path, not "path", and it has to be enclosed in double quotation marks... ) Then Press Enter(sorry, but most people don't know how to do all of these)

It should only display lines containing a match(in this case XXX-18245)

The first section gets the content of the file then passes it down the pipeline, the second section reads the result of first section and finds all lines containing a given string, you don't need the third section to make it run, but that would display a lot irrelevant informations, the third section makes the command only display the matching lines.

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password

Post as a guest

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

You Might Also Like