[SOLVED] How can I do a massive find and replace using regular expression?

Forum rules
Before you post please read how to get help
fricco
Level 1
Level 1
Posts: 26
Joined: Thu Apr 20, 2017 9:55 am
Location: Chicago, IL, USA
Contact:

[SOLVED] How can I do a massive find and replace using regular expression?

Postby fricco » Fri May 12, 2017 4:48 pm

Hello, I need to find and remove in more than 400 files a specific test. This files are XML files and for example I have a file with the following text:

Code: Select all

    <fieldPermissions>
        <editable>true</editable>
        <field>Quote__c.System_Pin__c</field>
        <readable>true</readable>
    </fieldPermissions>
    <fieldPermissions>
        <editable>true</editable>
        <field>Quote__c.Tax_Type_Client_Fee__c</field>
        <readable>true</readable>
    </fieldPermissions>
    <fieldPermissions>
        <editable>true</editable>
        <field>Quote__c.Tax_Type_Marketing_Fee__c</field>
        <readable>true</readable>
    </fieldPermissions>
    <fieldPermissions>
        <editable>true</editable>
        <field>Quote__c.Tax_Type_Risk_Fee__c</field>
        <readable>true</readable>
    </fieldPermissions>

And remove the complete section that goes:

Code: Select all

    <fieldPermissions>
        <editable>true</editable>
        <field>Quote__c.Tax_Type_Client_Fee__c</field>
        <readable>true</readable>
    </fieldPermissions>


Nowadays I'm using Sublime Text doing a search with the following regular expression:

Code: Select all

\s*<fieldPermissions>\r?\n\s*<editable>(false|true)<\/editable>\r?\n\s*<field>Quote__c.Tax_Type_Client_Fee__c<\/field>\r?\n\s*<readable>(false|true)<\/readable>\r?\n\s*<\/fieldPermissions>\r?\n\s*


This basically freezes Sublime Text as there are lot of files that match. So I'm looking to see if there is a way to do this from command line, expecting to have it done quicker as it doesn't use a UI.
I tried with sed:

Code: Select all

sed "s/\s*<fieldPermissions>\r?\n\s*<editable>(false|true)<\/editable>\r?\n\s*<field>Quote__c.Tax_Type_Client_Fee__c<\/field>\r?\n\s*<readable>(false|true)<\/readable>\r?\n\s*<\/fieldPermissions>\r?\n\s*/something/g"

and perl:

Code: Select all

perl -p -e "s/\s*<fieldPermissions>\r?\n\s*<editable>(false|true)<\/editable>\r?\n\s*<field>Quote__c.Tax_Type_Client_Fee__c<\/field>\r?\n\s*<readable>(false|true)<\/readable>\r?\n\s*<\/fieldPermissions>\r?\n\s*/something/g"

with no luck.
Any clue on how to do this, if it's even possible?
Last edited by fricco on Sun May 21, 2017 5:30 pm, edited 1 time in total.

User avatar
JerryF
Level 7
Level 7
Posts: 1701
Joined: Mon Jun 08, 2015 1:23 pm
Location: Rhode Island, USA

Re: How can I do a massive find and replace using regular expression?

Postby JerryF » Fri May 12, 2017 5:37 pm

Don't forget to edit your original post and add [SOLVED] to the beginning of the Subject line if your problem has been fixed.
My main language is English. I speak very little Portuguese, and a whole lot of gibberish.

fricco
Level 1
Level 1
Posts: 26
Joined: Thu Apr 20, 2017 9:55 am
Location: Chicago, IL, USA
Contact:

Re: How can I do a massive find and replace using regular expression?

Postby fricco » Fri May 12, 2017 7:53 pm


Thanks, but in here are talking about UI text editors, which will hav the same issues I'm having with Sublime Text. What I'm looking for is using something from the command line to avoid the load and display of the 400 files

Lemongrass38
Level 4
Level 4
Posts: 344
Joined: Mon Dec 12, 2016 11:29 pm
Location: Central Europe

Re: How can I do a massive find and replace using regular expression?

Postby Lemongrass38 » Sat May 13, 2017 5:16 am

.
If your issue is solved, please be so kind and indicate that by editing the topic title in the first post. :)

FreedomTruth
Level 4
Level 4
Posts: 326
Joined: Fri Sep 23, 2016 10:19 am

Re: How can I do a massive find and replace using regular expression?

Postby FreedomTruth » Sat May 13, 2017 9:31 pm

use sed? http://www.grymoire.com/Unix/Sed.html#uh-51a

Code: Select all

sed '
/<fieldPermissions>/ {
   N
   N
   /Quote__c.Tax_Type_Client_Fee__c/ {
      N
      N
      /<\/fieldPermissions>/d
   }
}
'

You still need to handle input/output but the above code should do what you want, I hope.
If you abide in My word, you are My disciples indeed. And you shall know the truth, and the truth shall make you free... Most assuredly, I say to you, whoever commits sin is a slave of sin... Therefore if the Son makes you free, you shall be free indeed.

fricco
Level 1
Level 1
Posts: 26
Joined: Thu Apr 20, 2017 9:55 am
Location: Chicago, IL, USA
Contact:

Re: How can I do a massive find and replace using regular expression?

Postby fricco » Sat May 13, 2017 10:27 pm

FreedomTruth wrote:use sed? http://www.grymoire.com/Unix/Sed.html#uh-51a

Code: Select all

sed '
/<fieldPermissions>/ {
   N
   N
   /Quote__c.Tax_Type_Client_Fee__c/ {
      N
      N
      /<\/fieldPermissions>/d
   }
}
'

You still need to handle input/output but the above code should do what you want, I hope.

Wow, yes it did! Any idea now how can I use this script to update the same file? When I tried file.xml > file.xml the result was an empty file.xml

rene
Level 6
Level 6
Posts: 1361
Joined: Sun Mar 27, 2016 6:58 pm

Re: How can I do a massive find and replace using regular expression?

Postby rene » Sun May 14, 2017 8:16 am

Use sed -i -e "script" input.xml for in-place editing of input.xml. sed -i.orig -e "script" input.xml for the same while backing up the original input.xml to input.xml.orig.

User avatar
WharfRat
Level 20
Level 20
Posts: 10215
Joined: Thu Apr 07, 2011 8:15 pm

Re: How can I do a massive find and replace using regular expression?

Postby WharfRat » Sun May 14, 2017 8:40 am

You should be able to incorporate find to go through all of the files e.g., find /usr/share/dbus-1/ -iname *.xml -exec cat '{}' \; replacing cat with your sed directive.

I would copy your target files to another location and test it first.
Image ImageImage

lmuserx4849
Level 4
Level 4
Posts: 489
Joined: Wed Dec 17, 2014 2:55 am

Re: How can I do a massive find and replace using regular expression?

Postby lmuserx4849 » Mon May 15, 2017 3:20 am

Incorporate FreedomTruth's sed into bash.

Code: Select all

#!/bin/bash
declare -- file=''
declare -- dir="$HOME"                              # ** change **
declare -- backup='/tmp/backup/'                # ** change **
declare -- log="/tmp/backup/${0##*/}.log"   # ** change **
declare -- rc=0

if [[ ! -d "${backup}" ]]; then
  mkdir "${backup}"
  printf -- 'Created directory %s\n' "${log}" >> "${log}"
fi

while read -re -d $'\0' file; do

  printf -- 'Processing %s\n' "${file}" >> "${log}"
  cp -a "${file}" "${backup}"

  sed -i '
  /<fieldPermissions>/ {
    N
    N
    /Quote__c.Tax_Type_Client_Fee__c/ {
        N
        N
        /<\/fieldPermissions>/d
    }
  }
  ' "${file}"

  rc=$?
  if (( "${rc}" != 0 )); then
    printf 'sed exit status not 0 [%s]' "${rc}"
    exit 1
  fi

done < <(find "${dir}" -name '*.xml' -print0)   # ** change as needed **

fricco
Level 1
Level 1
Posts: 26
Joined: Thu Apr 20, 2017 9:55 am
Location: Chicago, IL, USA
Contact:

Re: How can I do a massive find and replace using regular expression?

Postby fricco » Mon May 15, 2017 10:22 am

Thanks everyone, great advices. I will try them today and will mark the post as SOLVED once I confirm is working. But again, THANKS!


Return to “Scripts & Bash”