Order of Log Entries Changes Output

[Jator]

Trying to figure out how to use a Bash Script to create an entry for my Conky display.

I'm searching through my Channels DVR Log. For each channel that is opened, it creates a 'Opened Connection...' and when we stop watching the channel, it turns into 'Closed Connection....'. We have multiple tuners, so we could have 0-8 channels going at one time and I want to display which channels are actively being watched.

I can figure out how to pull the last entry of the opened channels from the below log, but the key here is determining if the last entry for each channel is "opened" vs "closed". Said another way, I need the script to be able to determine which channels have been opened but not yet closed.

Here's the lot format:

Code: Select all

2021/07/11 20:59:50.545461 [TNR] Opened connection to 13149111/0 for ch731 AMC HD
2021/07/11 21:33:57.085945 [TNR] Opened connection to 13149111/0 for ch847 Starz! Comedy H
2021/07/11 21:33:57.626407 [TNR] Opened connection to 13149111/1 for ch731 AMC HD
2021/07/11 21:45:10.003159 [TNR] Closed connection to 13149111/0 for ch847 Starz! Comedy H
2021/07/12 07:00:00.494440 [TNR] Opened connection to 1076456B/0 for ch8.1 WGAL-TV
2021/07/12 09:00:00.012414 [TNR] Closed connection to 1076456B/0 for ch8.1 WGAL-TV
2021/07/12 09:00:00.561826 [TNR] Opened connection to 1076456B/0 for ch8.1 WGAL-TV
2021/07/12 10:00:00.045563 [TNR] Closed connection to 1076456B/0 for ch8.1 WGAL-TV
2021/07/12 10:00:00.494161 [TNR] Opened connection to 1076456B/1 for ch8.1 WGAL-TV

Using the above code, WGAL and AMC HD would be shown as open (all others having been closed).

From programming in PHP over 15 years ago, I would imagine some form of a For i=n, do something; n++ until you run out of entries, but it's been too long and this isn't PHP so I'm not certain the best way to tackle this. Any suggestions.

I would access the logs using curl -s http://192.168.0.2:8089/logs

TIA

[xenopeek]

Assuming you have the channel names and the logs aren't huge (not gigabytes) you can do this by reversing the file (with tac logs) and then grepping the first occurrence (with grep -m 1) of each channel from it.

To show the idea for your example logs:

Code: Select all

$ tac logs | grep -m 1 'ch731 AMC HD'
2021/07/11 21:33:57.626407 [TNR] Opened connection to 13149111/1 for ch731 AMC HD
$ tac logs | grep -m 1 'ch847 Starz! Comedy H'
2021/07/11 21:45:10.003159 [TNR] Closed connection to 13149111/0 for ch847 Starz! Comedy H
$ tac logs | grep -m 1 'ch8.1 WGAL-TV'
2021/07/12 10:00:00.494161 [TNR] Opened connection to 1076456B/1 for ch8.1 WGAL-TV

I think that's right and the only channel closed in your example logs is ch847 Starz! Comedy H.

Or does the "13149111/0" on the logs line also have some relevance?

If you don't know the channels names up front you can first extract those from the logs. Assuming the "13149111/0" is always that length you can do it with cut and sort:

Code: Select all

$ cut -c70- logs | sort -u
ch731 AMC HD
ch8.1 WGAL-TV
ch847 Starz! Comedy H

In a bash script it is trivial to read that into an array:

Code: Select all

mapfile -t channels < <(cut -c70- logs | sort -u)

And you can loop over that array with:

Code: Select all

for channel in "${channels[@]}"; do
	echo $channel
done

If you need more help you'll need to explain the logs format. What parts are relevant of each line.

[1000]

There are many ways.
My examples. ( From last line, fourth word )

Code: Select all

$ tail -n 1 file.log |  awk '{ print $4 }'
Opened

Code: Select all

$ awk 'END{print}' file.log | awk '{ print $4 }'
Opened

Code: Select all

$ awk 'END{print $4}' file.log 
Opened

Code: Select all

$ sed -n '$p' file.log | sed 's/ /\n/g' | sed '1d;2d;3d;5d;6d;7d;8d;9d;$d'
Opened

Probably with "sed" I could have made smaller, but it's too difficult for me.

Code: Select all

$ tail -n 1 file.log | cut -d' ' -f4
Opened

Code: Select all

$ while read -r LINE; do LAST_LINE="$LINE" ; done < "file.log" ; COUNT=0 ; while IFS=' ' read -r LINE; do WORD="$LINE" ; COUNT=$[$COUNT + 1] ; ["$COUNT" -eq 4 ] && break ; done <<< $(printf %"s\n" $LAST_LINE) ; echo "$WORD"
Opened

Instead file from this examples, you can use the pipe " | ".

Code: Select all

curl do_something | next_do_something

P.S.
sed , awk , maybe also tail is not from bash
https://www.cyberciti.biz/faq/linux-uni ... -commands/

Code: Select all

$ dpkg -S $(locate tail | grep /bin/ | grep "/tail"$)
coreutils: /usr/bin/tail

Code: Select all

$ compgen -b | grep tail

From coreutils. But you can also try use.

[Jator]

Assuming you have the channel names and the logs aren't huge (not gigabytes) you can do this by reversing the file (with tac logs) and then grepping the first occurrence (with grep -m 1) of each channel from it.

To show the idea for your example logs:

Code: Select all

$ tac logs | grep -m 1 'ch731 AMC HD'
2021/07/11 21:33:57.626407 [TNR] Opened connection to 13149111/1 for ch731 AMC HD
$ tac logs | grep -m 1 'ch847 Starz! Comedy H'
2021/07/11 21:45:10.003159 [TNR] Closed connection to 13149111/0 for ch847 Starz! Comedy H
$ tac logs | grep -m 1 'ch8.1 WGAL-TV'
2021/07/12 10:00:00.494161 [TNR] Opened connection to 1076456B/1 for ch8.1 WGAL-TV

I think that's right and the only channel closed in your example logs is ch847 Starz! Comedy H.

Or does the "13149111/0" on the logs line also have some relevance?

If you don't know the channels names up front you can first extract those from the logs. Assuming the "13149111/0" is always that length you can do it with cut and sort:

Code: Select all

$ cut -c70- logs | sort -u
ch731 AMC HD
ch8.1 WGAL-TV
ch847 Starz! Comedy H

In a bash script it is trivial to read that into an array:

Code: Select all

mapfile -t channels < <(cut -c70- logs | sort -u)

And you can loop over that array with:

Code: Select all

for channel in "${channels[@]}"; do
	echo $channel
done

If you need more help you'll need to explain the logs format. What parts are relevant of each line.

Thanks for the reply. No, logs are only several hundred deep before they drop off the web display (FIFO format).

I "think" the numeric value is the identified for which HDHOMERUN tuning box is being used (I have 4 total, 2 prime, 1 duo and 1 quatro) and the number behind the backslash is which tuner is being used within the specific HDHOMERUN box. So although the number may change in different instances of watching the same channel, it will always be the same from the opening of the channel to the close of the channel.

Your code is fairly advanced for me, so I'll need to play with it and reverse engineer to see if it works, but many thanks for giving me this head start. I'll report progress and ask any clarifying questions as I make progress.

I like the flipping of the log, as the ultimate trigger of showing a value is if the channel is still opened and hasn't been closed. I

[Jator]

So played with a couple of the scrips above, very informative. Here's some additional details:

- The length of the log files is not always the same. As an example, I can also get my channel from a streaming service vs. the physical tuner in my house, thus in some instances the code looks like this;

2021/07/11 17:49:24.620248 [TNR] Opened connection to 13149111/0 for ch552 TBS HD
2021/07/11 17:50:31.983554 [TNR] Opened connection to 13149111/0 for ch551 TNT HD
2021/07/11 18:01:21.171851 [TNR] Opened connection to TVE-Verizon for ch6033 TBS
2021/07/11 18:01:39.656477 [TNR] Opened connection to TVE-Verizon for ch6036 TNTP
2021/07/11 18:02:20.925513 [TNR] Opened connection to 1076456B/0 for ch8.1 WGAL-TV

Also, I may be misinterpreting the code suggested, but I think I need to do a record compare to determine if a channel should be listed. In layperson's terminology, I think the code has to do the following:

Code: Select all

If "[TNR] Open.*.ch{number} {channel_name}" exists
   Look for Same Channel Closed Record AFTER the Open of the Channel, If Present Mark as FALSE // Identifying a closed record that occurred before the opening of the channel would throw off a false True
Else
   If Same Channel Closed Record does not exist AFTER the Open record, mark as TRUE
End If Statement
Repeat for All Open Channels to create an array of channels to be displayed as currently being watched (ie marked as TRUE)
Display each Channel Captured from above script 

Easiest way I believe is to use the "channel number" in search and compare values as each channel number is unique, whereas I could have duplicate WGAL-TV sources (channel 8.1 and 508).

Hopefully that makes some semblance of sense?

[xenopeek]

Something like this I think:

Code: Select all

#!/bin/bash

# get a temporary filename
tempfile="$(mktemp)"

# reverse the logs and store it in the temporary file
tac logs > "$tempfile"

# get list of (unique) channel names from the logs, store in array 'channels'
mapfile -t channels < <(cut -d' ' -f9- logs | sort -u)

# iterate over the channels and figure out if they are opened or closed
for channel in "${channels[@]}"; do
	# find the last status line of the channel
	status=$(grep -m 1 "$channel" "$tempfile")
	# only print the channel name if the last status is that it is opened
	if [[ $status == *" Opened connection "* ]]; then
		# print the channel name and for debug purposes also print the last status
		echo "$channel :: $status"
	fi
done

# remove the temporary file
rm "$tempfile"

Put that in a file and save it. Make it executable (chmod +x file or through file manager > right-click on file > properties > permissions). Then run it in the directory where you have a logs file. See if it correctly reports the opened channels.

[1000]

I can figure out how to pull the last entry of the opened channels from the below log, but the key here is determining if the last entry for each channel is "opened" vs "closed". Said another way, I need the script to be able to determine which channels have been opened but not yet closed.

8 channels of constant named ( ch1 , ch2 , ch3 , ch4 , ch5 , ch6 , ch7 , ch8 ).
You can extract the data in two ways.

1. Read only the last line of the log ,
Default value for each variable ( ch1 , ch2 , ch3 , ch4 , ch5 , ch6 , ch7 , ch8 ) will be "UNKNOWN".
Then from last line you can read "STATUS" and "CHANNEL" name.
And "STATUS_ch_number" will be replaced from "UNKNOWN" to "Opened" or "Closed".

Code: Select all

if [[ "$CHANNEL" == "ch1" ]] ; then
	STATUS_ch1="$STATUS"
elif  [[ "$CHANNEL" == "ch2" ]] ; then
	STATUS_ch2="$STATUS"
elif  [[ $"CHANNEL" == "ch3" ]] ; then
	STATUS_ch3="$STATUS"
elif  [[ "$CHANNEL" == "ch4" ]] ; then
	STATUS_ch4="$STATUS"
elif  [[ "$CHANNEL" == "ch5" ]] ; then
	STATUS_ch5="$STATUS"
elif  [[ "$CHANNEL" == "ch6" ]] ; then
	STATUS_ch6="$STATUS"
elif  [[ "$CHANNEL" == "ch7" ]] ; then
	STATUS_ch7="$STATUS"
elif  [[ "$CHANNEL" == "ch8" ]] ; then
	STATUS_ch8="$STATUS"
fi

In Bash, instead of " if " You can try use " case " https://bash.cyberciti.biz/guide/The_case_statement


2. Or search all channels from log, then for every channel read last line.

Code: Select all

$ cat file.log | grep ch8.1 | tail -n 1 |  awk '{ print $4 }'
Opened

3. Or read the output line by line
and from every line read "STATUS" and "CHANNEL" name. ( example with while above )
and add " if " or " case " like above

Code: Select all

#!/bin/bash

while IFS=' ' read -r DATE HOUR TNR STATUS C T NUMBER F CHANNEL NAME; do
	if [[ "$CHANNEL" == "ch1" ]] ; then
		STATUS_ch1="$STATUS"
	elif  [[ "$CHANNEL" == "ch2" ]] ; then
		STATUS_ch2="$STATUS"
	elif  [[ "$CHANNEL" == "ch3" ]] ; then
		STATUS_ch3="$STATUS"
	elif  [[ "$CHANNEL" == "ch4" ]] ; then
		STATUS_ch4="$STATUS"
	elif  [[ "$CHANNEL" == "ch5" ]] ; then
		STATUS_ch5="$STATUS"
	elif  [[ "$CHANNEL" == "ch6" ]] ; then
		STATUS_ch6="$STATUS"
	elif  [[ "$CHANNEL" == "ch7" ]] ; then
		STATUS_ch7="$STATUS"
	elif  [[ "$CHANNEL" == "ch8" ]] ; then
		STATUS_ch8="$STATUS"
fi
done < "file.log"

echo "CHANNEL7 = $STATUS_ch7"
echo "CHANNEL8 = $STATUS_ch8"

Where for example ch8 this can be channel name " ch8.1 " from log .
But the construction of this is extensive, so in one line it will not be very readable.
In the post before, I also used "while" loop, but twice. Here you have one "while" loop example.

[Jator]

Something like this I think:

Code: Select all

#!/bin/bash

# get a temporary filename
tempfile="$(mktemp)"

# reverse the logs and store it in the temporary file
tac logs > "$tempfile"

# get list of (unique) channel names from the logs, store in array 'channels'
mapfile -t channels < <(cut -d' ' -f9- logs | sort -u)

# iterate over the channels and figure out if they are opened or closed
for channel in "${channels[@]}"; do
	# find the last status line of the channel
	status=$(grep -m 1 "$channel" "$tempfile")
	# only print the channel name if the last status is that it is opened
	if [[ $status == *" Opened connection "* ]]; then
		# print the channel name and for debug purposes also print the last status
		echo "$channel :: $status"
	fi
done

# remove the temporary file
rm "$tempfile"

Put that in a file and save it. Make it executable (chmod +x file or through file manager > right-click on file > properties > permissions). Then run it in the directory where you have a logs file. See if it correctly reports the opened channels.

Thanks xenopeek. That seems to work. Have to work through the code and see how to integrate it into Conky, but I'll tackle that.

Also, thanks to 1000 for the suggestions, I'm going to see if I can use some of those concepts in some other pet projects I'm working on.

[Jator]

Something like this I think:

Code: Select all

#!/bin/bash

# get a temporary filename
tempfile="$(mktemp)"

# reverse the logs and store it in the temporary file
tac logs > "$tempfile"

# get list of (unique) channel names from the logs, store in array 'channels'
mapfile -t channels < <(cut -d' ' -f9- logs | sort -u)

# iterate over the channels and figure out if they are opened or closed
for channel in "${channels[@]}"; do
	# find the last status line of the channel
	status=$(grep -m 1 "$channel" "$tempfile")
	# only print the channel name if the last status is that it is opened
	if [[ $status == *" Opened connection "* ]]; then
		# print the channel name and for debug purposes also print the last status
		echo "$channel :: $status"
	fi
done

# remove the temporary file
rm "$tempfile"

Put that in a file and save it. Make it executable (chmod +x file or through file manager > right-click on file > properties > permissions). Then run it in the directory where you have a logs file. See if it correctly reports the opened channels.

K, if I wanted to use curl to access the log files off the webserver, the first entry accepts that;

Code: Select all

# reverse the logs and store it in the temporary file
tac http://192.168.0.2:8089/log > "$tempfile"

However, the second section doesn't like it, any suggestion on how to address, I've tried a couple different options and nothing has worked:

Code: Select all

[mapfile -t channels < <(cut -d' ' -f9- http;//192.168.0.2:8089/logs | sort -u)/code]

[xenopeek]

I would download the logs first. Something like this:

Code: Select all

#!/bin/bash

# where to download the logs from
LOG_URL="http://192.168.0.2:8089/log"

# get temporary filenames for the logs and the reversed logs
logsfile="$(mktemp)"
reversedfile="$(mktemp)"

# download the logs
wget -O "$logsfile" "$LOG_URL" 2>/dev/null

# reverse the logs and store it in the temporary file
tac "$logsfile > "$reversedfile"

# get list of (unique) channel names from the logs, store in array 'channels'
mapfile -t channels < <(cut -d' ' -f9- "$logsfile" | sort -u)

# iterate over the channels and figure out if they are opened or closed
for channel in "${channels[@]}"; do
	# find the last status line of the channel
	status=$(grep -m 1 "$channel" "$reversedfile")
	# only print the channel name if the last status is that it is opened
	if [[ $status == *" Opened connection "* ]]; then
		# print the channel name and for debug purposes also print the last status
		echo "$channel :: $status"
	fi
done

# remove the temporary files
rm "$logsfile" "$reversedfile"