Remove notes script

Forum rules
Before you post please read this

Remove notes script

Postby Freestyler on Sun Dec 15, 2013 9:04 am

Many text files contains notes or un-setting commands by starting text lines with this caracter: #

For instance /etc/bluetooth/main.conf is 2.4kb and looks like this:
Code: Select all
[General]

# List of plugins that should not be loaded on bluetoothd startup
#DisablePlugins = network,input

# Default adaper name
# %h - substituted for hostname
# %d - substituted for adapter id
Name = %h-%d

# Default device class. Only the major and minor device class bits are
# considered.
Class = 0x000100

# How long to stay in discoverable mode before going back to non-discoverable
# The value is in seconds. Default is 180, i.e. 3 minutes.
# 0 = disable timer, i.e. stay discoverable forever
DiscoverableTimeout = 0

# How long to stay in pairable mode before going back to non-discoverable
# The value is in seconds. Default is 0.
# 0 = disable timer, i.e. stay pairable forever
PairableTimeout = 0

# Use some other page timeout than the controller default one
# which is 16384 (10 seconds).
PageTimeout = 8192

# Automatic connection for bonded devices driven by platform/user events.
# If a platform plugin uses this mechanism, automatic connections will be
# enabled during the interval defined below. Initially, this feature
# intends to be used to establish connections to ATT channels.
AutoConnectTimeout = 60

# What value should be assumed for the adapter Powered property when
# SetProperty(Powered, ...) hasn't been called yet. Defaults to true
InitiallyPowered = true

# Remember the previously stored Powered state when initializing adapters
RememberPowered = true

# Use vendor id source (assigner), vendor, product and version information for
# DID profile support. The values are separated by ":" and assigner, VID, PID
# and version.
# Possible vendor id source values: bluetooth, usb (defaults to usb)
#DeviceID = bluetooth:1234:5678:abcd

# Do reverse service discovery for previously unknown devices that connect to
# us. This option is really only needed for qualification since the BITE tester
# doesn't like us doing reverse SDP for some test cases (though there could in
# theory be other useful purposes for this too). Defaults to true.
ReverseServiceDiscovery = true

# Enable name resolving after inquiry. Set it to 'false' if you don't need
# remote devices name and want shorter discovery cycle. Defaults to 'true'.
NameResolving = true

# Enable runtime persistency of debug link keys. Default is false which
# makes debug link keys valid only for the duration of the connection
# that they were created for.
DebugKeys = false

# Enable the GATT functionality. Default is false[b]
EnableGatt = false

and all its use is this 263 bytes.
Code: Select all
[General]
Name = %h-%d
Class = 0x000100
DiscoverableTimeout = 0
PairableTimeout = 0
PageTimeout = 8192
AutoConnectTimeout = 60
InitiallyPowered = true
RememberPowered = true
ReverseServiceDiscovery = true
NameResolving = true
DebugKeys = false
EnableGatt = false

So the idea is to get rid of all the notes. Can this be done safely by a script after installating the distro? I do see the need of the notes for testers and programmers, so in develop-distroes, they are extreme relevant but for people like me, a common user...I do not need the notes at all.

Maybe there are similar use of # or other character in xml files, java files, etc. Can a cleansing script removing notes be include in the distro, or maybe even better, clean the normal distros and have smaller image-files to download?
Last edited by xenopeek on Sun Dec 15, 2013 9:10 am, edited 1 time in total.
Reason: Put file contents in [code] tags for readability of the topic.
Freestyler
Level 1
Level 1
 
Posts: 4
Joined: Sat Oct 17, 2009 9:10 am

Linux Mint is funded by ads and donations.
 

Re: Remove notes script

Postby xenopeek on Sun Dec 15, 2013 9:56 am

Let's count how many bytes there are "wasted" with the comment lines in all 42,200 files in /etc (configuration files) and /usr (scripts) on Linux Mint 16 Cinnamon that have comment lines. Disregarding that Linux Mint can't remove most comment lines that I'm counting this way, as that would be a license violation (as most comment lines in /etc and /usr are in fact license statements). But for sake of argument, let's just count 'em all.
Code: Select all
grep -hRs ^# /etc /usr | grep -v ^#! | wc -c

Shows it's 52,310 KB. But wait... That's uncompressed! Let's compress it and see how big it is:
Code: Select all
grep -hRs ^# /etc /usr | grep -v ^#! | bzip2 | wc -c

That shows 7,923 KB. Okay, so how big is the Linux Mint 16 Cinnamon ISO? 1,252,147 KB. So what would this have saved in downloading?
    7,923 KB / 1,252,147 KB = 0.6% :|
But wait! How much disk space would it have saved on a 100 GB hard disk, which I think is safe to assume as an average for Linux Mint users?
    52,310 KB / 100,000,000 KB = 0.005% :|
I'm sorry, but removing these comment lines is just not worth the space saved. Especially considering that these numbers are way too high, as most comment lines will be license statements that can't be removed without violating the license. And for problem solving or fine-tuning even average users might need to occasionally edit configuration files by hand and so removing the comments would make it needlessly much harder for them :( And if we look only at configuration files in /etc instead of also at scripts in /usr, the numbers drop to insignificance even further (ISO would be 0.013% smaller and on the hard disk 0.0009% would have been saved).

But yes, if you want to hinder future you from being able to understand the configuration files should there arise a need for you to edit them, you could remove them with a script. Do you still want to? :)
Forum Rules | IRC Channel Rules
Image
Arch Linux / 64-bit / Gnome Shell
User avatar
xenopeek
Level 21
Level 21
 
Posts: 13679
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Re: Remove notes script

Postby Freestyler on Tue Dec 17, 2013 4:16 pm

Hi Xenopeek

Thank you for your comments, calculations and estimates, wich seems pretty hard to argue against :-). The licence topic, I had not nosiddered and also I had an idea of the amount of space occupied, might have been larger. If I dont distribute the files, a deletion of licence statements hardly will offend anyone, as long as I just do not distrubute the files.

But to your question, if I still want to do it, the answer is yes....why not? I still think it could be fun trying :-D
Freestyler
Level 1
Level 1
 
Posts: 4
Joined: Sat Oct 17, 2009 9:10 am

Re: Remove notes script

Postby xenopeek on Tue Dec 17, 2013 5:19 pm

Okay, well then I'll let you in on one more secret :) Most files occupy more space on your hard disk than their actual size. That's because files are stored on your hard disk in block of, usually, 4,096 bytes. Even if a file is only 1 byte, it still takes up 1 block. If a file is 1 byte larger than a multiple of the block size, it takes up 1 additional block. For example if a file has a size of 4,097 bytes (1 byte larger than the block size), it takes 8,192 bytes of disk space (2 blocks).

So, let's assume on average the last file system block used for a file is used only half. So 2,048 bytes in the last file system block are used. How much bytes are we actually going to save on average for each file? It needs to be more than this 2,048 bytes to make a dent. We can already calculate the average number of bytes saved per file by removing the comment lines: 52,310 KB / 42,200 files = 1,240 bytes per file saved on average. Not enough if we go by average :(

Well, anyway, I'd recommend trying this out in a directory in your home folder with some test files and not start doing this in /etc or /usr as I didn't test this. Taking hints from this topic http://stackoverflow.com/questions/1635 ... ot-shebang. Basically you could do:
Code: Select all
sed -i '/^(#[^!]|#$|$)/d' filename

Where you replace filename with the name of the file to affect. It will edit the file and save it in place (without a backup, see the sed manpage for how to make a backup of the file). It would remove all empty lines and all lines that start with #, but skip the shebang line (#!) that indicates a file is a script. I'd probably combine it with a find command to target all normal files in a directory. Except I wouldn't do this :) Anyway, I've moved your topic to the scripts & bash forum and others may perhaps help you further on this if you still want to. I'm done for the day here :wink:
Forum Rules | IRC Channel Rules
Image
Arch Linux / 64-bit / Gnome Shell
User avatar
xenopeek
Level 21
Level 21
 
Posts: 13679
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Re: Remove notes script

Postby Freestyler on Mon Dec 23, 2013 3:18 pm

Hi Xenopeek

Thanx again for your efforts of keeping me doin stupid things ;-)

I have started 2 identical virtual machines in Virtualbox, both running the new LinuxMint 16 Xfce. My intention is to have one untouched for eventually more klones, and then do all the mistakes in the klones. Like for a directory and sub's or a whole system at one time. Is that posible with a script like the last you posted? How would that look?
Freestyler
Level 1
Level 1
 
Posts: 4
Joined: Sat Oct 17, 2009 9:10 am

Re: Remove notes script

Postby xenopeek on Mon Dec 23, 2013 4:49 pm

I ended up with quite a complex command. I've only tested this a little bit on files in /etc, and didn't do a complete system check afterwards. So warranty voids once you run the command :wink: It probably needs some further fine-tuning as to on which files it works and which it skips (for example, this also edits the /etc/sudoers file which you should in fact never do like that, and get lock you out of using sudo).
Code: Select all
sudo find -L /etc -type f -exec bash -c 'if [[ $(file -b {}) = "ASCII text" ]] ; then echo "editing {}" ; sed -i -e "/^\#[^!]/d" -e "/^\#$/d" -e "/^\s*$/d" {} ; fi' \;

Let's go through the statement step by step so you understand what it's doing.
sudo find -L /etc -type f -exec bash -c 'if [[ $(file -b {}) = "ASCII text" ]] ; then echo "editing {}" ; sed -i -e "/^\#[^!]/d" -e "/^\#$/d" -e "/^\s*$/d" {} ; fi' \;

Runs the find command as root.
sudo find -L /etc -type f -exec bash -c 'if [[ $(file -b {}) = "ASCII text" ]] ; then echo "editing {}" ; sed -i -e "/^\#[^!]/d" -e "/^\#$/d" -e "/^\s*$/d" {} ; fi' \;

Makes find follow symbolic links, parse all files and folders in /etc, and match only on regular files.
sudo find -L /etc -type f -exec bash -c 'if [[ $(file -b {}) = "ASCII text" ]] ; then echo "editing {}" ; sed -i -e "/^\#[^!]/d" -e "/^\#$/d" -e "/^\s*$/d" {} ; fi' \;

For each match, executes a command. In this case, as it's quite complex, one command wouldn't suffice so as a command I launch bash to interpret a string of commands.
sudo find -L /etc -type f -exec bash -c 'if [[ $(file -b {}) = "ASCII text" ]] ; then echo "editing {}" ; sed -i -e "/^\#[^!]/d" -e "/^\#$/d" -e "/^\s*$/d" {} ; fi' \;

For each match, first check if the reported file type is "ASCII text"--we don't want to remove any characters from binary files! (I didn't have this check in place first, and that wrecked one virtual machine :)) If the file is an "ASCII text" file, next print that it is going to be edited (so you can see what it is doing). Note that inside the find command's exec option the sequence {} is replaced by the name of the file matched.
sudo find -L /etc -type f -exec bash -c 'if [[ $(file -b {}) = "ASCII text" ]] ; then echo "editing {}" ; sed -i -e "/^\#[^!]/d" -e "/^\#$/d" -e "/^\s*$/d" {} ; fi' \;

Next the actual editing of the matched file, of which it is confirmed that it is an "ASCII text" file. The -i parameter makes sed write changes directly back to the file. There are three edit actions for sed:
  • "/^\#[^!]/d" - deletes any line that starts with # and doesn't start with #!
  • "/^\#$/d" - deletes any line that start with # and is otherwise empty
  • "/^\s*$/d" - deletes any line that is empty or only contains whitespace characters
sudo find -L /etc -type f -exec bash -c 'if [[ $(file -b {}) = "ASCII text" ]] ; then echo "editing {}" ; sed -i -e "/^\#[^!]/d" -e "/^\#$/d" -e "/^\s*$/d" {} ; fi' \;

Final sequence \; is to end the exec option of the find command.
Forum Rules | IRC Channel Rules
Image
Arch Linux / 64-bit / Gnome Shell
User avatar
xenopeek
Level 21
Level 21
 
Posts: 13679
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands


Return to Scripts & Bash

Who is online

Users browsing this forum: No registered users and 2 guests