Concatenate multiple files based on filename similarity : Windows Bash(Using Windows 10),Powershell, or Command Line)

I'm currently stuck in what seemed to be an easy task. I've been working on a python script that finds certain files in my Downloads folder and dumps them into another sub-folder within a different folder. Then, it converts the PDFs in that designated folder into text files since I will be extracting info from the files(and it's just way easier to work with TXT).

Now, I'm stuck in this next part : I have ~ 600 files (and will continue to have more) and want to combine files based on filename. This is how the filenames are formatted...

• Txt_BI_ProfilesBI_Profile_Export_BB+Generic_August+2016_GGP_20170316.pdf
• Txt_BI_BrofilesBI_Profile_Export_BB+Generic_August+2016_GGP_20170316.pdf (1)
• Txt_BI_ProfilesBI_Profile_Export_Search_20170228.pdf
• Txt_BI_ProfilesBI_Profile_Export_Search_20170228.pdf (1)

Note how there are files that share filename but are distinguishable with .pdf(1) . I need a powershell or bash script that groups and merges files into a consolidated text files based on filename similarity.

So given the filenames above, I want to merge all the content in

'TxtBI_ProfilesBI_Profile_Export_BB+Generic_August+2016_GGP_20170316.pdf' + 'Txt_BI_ProfilesBI_Profile_Export_BB+Generic_August+2016_GGP_20170316.pdf (1)' into a file = 'BB+Generic_August+2016_GGP'

and same goes for 'Search'. For 'BB+Generic_August+2016_GGP' and 'Search' group there are up to about '.pdf(40)' different files, and there are about 10 or more different group of files that share similar filenames. I have written a python script that attempts to do this here but all it's doing is splitting the filenames[33:] and filenames.rsplit('',1)[0] (splits after the 33 character and the last '_') and not combining them into consolidated files as I mentioned above. Can I do this with powershell or bash? Such that it splits filenames(as abovementioned) and then combines them If they hold the same text filenname?

I thought this was going to be super easy but it's not working for me. If anyone has any insight/ideas/suggestions on how to approach this, I would truly appreciate it! Have used bash in the past before for unix, but it's been a while. I'll go with whatever is most practical! Still a bit of a novice when it comes to programming....

1 Answer

Edit: better solution that produces the desired output file names

Making use of ls, awk and cat:

ls | awk '! /\([0-9]+\)/ {match($0, /Txt_BI_ProfilesBI_Profile_Export_([^.]+)/,matches); system("cat " $0 "* >"matches[1] ".txt")}'

Prompted by Matthew's display of Powershell's powers, I had to show the same can be achieved in a bash environment. This is just one of the many ways, as is common in the UNIX world.

Explanation:

ls lists all elements in the current directory

awk programming language designed for text processing

! /\([0-9]+\)/ exclude filenames that contain a number in parentheses

match($0, /Txt_BI_ProfilesBI_Profile_Export_([^.]+)/,matches) perform a regular expression match on the filename, capturing the part between the common prefix and the first dot

system(" execute a system command

cat " $0 "* concatenate files that start with the filename

>"matches[1] ".txt" output to a file named like the captured expression and having a .txt extension

First answer:You can use find, xargs, bash and cat:

find . -type f -regextype sed ! -regex "\./.*([0-9]\+)" | xargs -I{} bash -c 'cat {}* > {}.txt'

This is more of a proof-of-concept and could be refined, as the output filenames are not exactly what you sought, but should be enough to solve your problem in the immediate.

Explanation:

find . search the current directory

-type f look for files

-regextype sed use the regular expression engine with sed-compatible syntax

! -regex "\./.*([0-9]\+)" exclude results that match the specified regular expression, i.e. those that include a number in parentheses at the end

| xargs use each result to construct a command

-I{} in the following command, substitute {} with each find result

bash -c pass the following string as a command to bash

'cat {}* > {}.txt' concatenate the files that start with the found filename into a file that has a name made up from the find result and the .txt extension