[Solved] Merge Files

[Pecu1iar]

First, let me say, "Thank you all." for your help.

I need a bash script to achieve this goal.

I have a csv file (User Info) that contains 3 columns of information. Column A is the desired output file name. Column B is a Username. Column C is a password.

I have several rows of User information contained in this file.

I want to take a master file (Master) and search the text of the file and replace all instances of the word username and password with the information from the respective column from the csv file. Once finished editing the Master file, I want to "save as" with the filename from column A that corresponds with the username and password from column B and C. I have to use this same Master file for each user so that when finished, I still have the Master file and a duplicate file for each user with specific User Names and Passwords.

If needed, I can reorder the columns in the User Info file. The User Info file and the Master file are both located in the same folder, /home/tony/Desktop/Test.

If any further information is needed to solve this problem, please let me know.

Again, Thank You all for all that you do. You guys are awesome.

[xenopeek]

This should do that, assuming fields are separated by comma and the master file is called Master abd the user info file userinfo.csv:

Code: Select all

#!/bin/bash

masterfile="Master"
infofile="userinfo.csv"

while IFS=',' read -ra infoline; do
	outputfile=${infoline[0]}
	username=${infoline[1]}
	password=${infoline[2]}
	echo "Writing $outputfile for $username"
	sed -r "s/username/${username}/g; s/password/${password}/g" "$masterfile" > "$outputfile"
done < "$infofile"

At the top change the name of the Master file and of the userinfo.csv file if the are different.

Save this in a file in your /home/tony/Desktop/Test directory and mark it as executable. Then open a terminal on the /home/tony/Desktop/Test directory and run the file as ./filename and it should do it.

[Pecu1iar]

Thank You for your help.

This script almost works. The desired output file name consists of a first name, last name, and employee number so it looks something like
John Doe Jr. 12565

If I edit the desired file name to just a first name, the script works. If I use first, last, and employee number, I get an $outputfile: ambiguous redirect and not edited files. I need to use a first, last, and number because we have some employees with the same name.

Can this be fixed?

[xenopeek]

I've edited this line the script above:
sed -r "s/username/${username}/g; s/password/${password}/g" "$masterfile" > "$outputfile"
I put double quotes around $masterfile and around $outputfile. See if that makes a difference.

And mind that as written the script assumes there are exactly, no less no more, two commas on each line. Separating the three fields outputfile, username, password. If there can be commas in the outputfile name, a further change is needed (you'll need to use a different character to as field separator).

[Pecu1iar]

Thank you, the script that you created works perfectly.
I'm still trying to perfect the process.
The first issue:
The userinfo list is downloaded from a google sheets file. There are tons of empty fields that are included in the file. As a result, There is a file that is created with no name, with no username and password. This file is created and overwritten nearly a thousand times. Its not that big of a deal, it just prolongs the process. I'm trying to figure out how to either delete the empty lines or edit my google sheets formula so that the empty cells are not imported. I've been doing lots of googing and trial and error trying to find a solution.


T̶h̶e̶ ̶s̶e̶c̶o̶n̶d̶ ̶i̶s̶s̶u̶e̶:̶ ̶I̶'̶m̶ ̶t̶r̶y̶i̶n̶g̶ ̶t̶o̶ ̶f̶i̶g̶u̶r̶e̶ ̶o̶u̶t̶ ̶h̶o̶w̶ ̶t̶o̶ ̶s̶a̶v̶e̶ ̶t̶h̶e̶ ̶n̶e̶w̶l̶y̶ ̶c̶r̶e̶a̶t̶e̶d̶ ̶f̶i̶l̶e̶s̶ ̶i̶n̶t̶o̶ ̶a̶ ̶d̶i̶f̶f̶e̶r̶e̶n̶t̶ ̶d̶i̶r̶e̶c̶t̶o̶r̶y̶.̶ ̶ ̶I̶'̶v̶e̶ ̶t̶r̶i̶e̶d̶ ̶a̶d̶d̶i̶n̶g̶ ̶a̶ ̶d̶i̶f̶f̶e̶r̶e̶n̶t̶ ̶d̶i̶r̶e̶c̶t̶o̶r̶y̶ ̶t̶o̶ ̶t̶h̶e̶ ̶s̶c̶r̶i̶p̶t̶,̶ ̶b̶u̶t̶ ̶I̶'̶m̶ ̶n̶o̶t̶ ̶g̶e̶t̶t̶i̶n̶g̶ ̶i̶t̶ ̶t̶o̶ ̶w̶o̶r̶k̶.̶ ̶ ̶I̶'̶m̶ ̶s̶t̶i̶l̶l̶ ̶s̶e̶a̶r̶c̶h̶i̶n̶g̶ ̶l̶o̶l̶.̶ I've figured this one out.

Thanks again for your help.

[xenopeek]

To save to a different directory change the below line with the addition at the end. Either use an absolute path, like this if you want to put the files in "/home/username/newfiles":
sed -r "s/username/${username}/g; s/password/${password}/g" "$masterfile" > "/home/username/newfiles/$outputfile"
Or use a relative path if say you want to put the files in subdirectory "otherfiles" on the current directory:
sed -r "s/username/${username}/g; s/password/${password}/g" "$masterfile" > "otherfiles/$outputfile"

As for the other issue, you'll have to look at what is happening in the text file you downloaded from Google. Are there lines with more than 3 comma separated fields? Then in your Google sheet select the extra (empty) columns to the right of the first 3 columns and delete those columns. I don't use Google sheets but that's what I'd do in LibreOffice Calc if there were too many fields -- means you at some point put something in those 4th, 5th, 6th... columns and the spreadsheet has remembered those columns. So delete those empty columns and it should forget about them.

If there are lines that just have commas and/or whitespace on them you can remove those with a grep before you run the script.
grep -v '^[,[:space:]]*$' inputfile > outputfile
That command will read the file "inputfile" and remove all the lines from it that ONLY have whitespace and/or commas on them, and save it to a new file "outputfile".

[Pecu1iar]

Thank you xenopeek. I figured out how to write the files to a different directory before I saw your post explaining how to.
I was trying to figure out how to remove the empty lines either by editing the file or by using a formula to disregard the empty cells.
Your script worked perfectly. Now the whole script is doing exactly what I want it to do and it's all due to your expertise.
Thank you for taking the time to show a noob. I'm trying to learn. I do a lot of google searching before asking and I'm thankful for people like you
who are willing to help.

[xenopeek]

Happy to help. Glad it's now doing what you need.

Knowing a bit of shell scripting and core utilities like sed and grep can be a handy tool to have in your back pocket.