Comskip/PlexComskip?

[ralplpcr]

Not sure if this belongs here or in the software forum, but I'll give this one a shot?

I've got a Plex media server set up with the beta DVR function enabled. While it seems to work pretty well, there are 2 big issues I've been trying to figure out:

1. The recorded file size is huge. It outputs into a .TS file, which is often 2GB for a 30 minute show @ 1080p. Does anyone have a method to automate recoding with Handbrake down to a smaller .MKV (or .MP4) file? I can do it manually, but since it by default saves to a show's specific folder I don't see how it can be scripted.

2. I've seen that there are some automatic commercial-skipping programs that will detect & edit out commercials. (Comskip & PlexComskip) I haven't been able to make them work, however - - most of the info I've found relates to Windows and creating batch files. Anybody have experience that can help get these working? Ideally, I'd like to automate that as well.

Also, I guess it would help to know which of these operations should be done first? Does it make sense to remove the commercials before converting to MKV, or should that be done afterwards?

My main Plex is on Cinnamon 18.1, though I also have an instance running on Cinnamon 17.3. My antenna & Plex DVR is currently running on the 17.3 system, but I'm open to moving it to the 18.1 system if it makes sense to do so.

Appreciate any guidance you can offer!

[ugly]

I've been using Comskip. It generally works well. I never bothered to automate it.

Comskip wasn't too hard to get working. But I did it a long time ago. I think I got it here: https://github.com/erikkaashoek/Comskip

I think I needed to install these packages (or at least some of them) to get it working:

Code: Select all

sudo apt-get install git build-essential libargtable2-dev libsdl1.2-dev
sudo apt-get install libargtable2-0
sudo apt-get install libavformat-ffmpeg-dev
sudo apt-get install autoconf automake libtool
sudo apt-get install Livavutil-dev Libavformat-dev Libavcodec-dev

Then you can get comskip by using:
(note - you can change ~/git to wherever you want)

Code: Select all

cd ~/git
git clone git://github.com/erikkaashoek/Comskip
cd Comskip
./autogen.sh
./configure
make

Then to actually use it:
~/git/Comskip/comskip /path/to/videofile

Although, it's probably best to make a bash file so you can right click on the video and run it:

Code: Select all

#!/bin/bash

	gnome-terminal -e "~/git/Comskip/comskip \"$1\" --ini='/path/to/inifile'"

	
#keeps shell open
	$SHELL
	

Using an .ini file is optional.

Obviously, change the paths to suit your setup.

[ralplpcr]

Thanks ugly - - I've sort of got it working, though not entirely how I want.
As a test, I took a 10 min video clip that contained 5 commercials about 1:30 in. After adjusting my paths, I ran the PlexComskip.py and it went to work. It stripped out 3 of the 5 commercials, so it was a partial success?

Any suggestions how to tweak it to get all 5 of the commercials detected & removed?

My ultimate goal is to automate using Plex's PostProcessing script. To create a bash file that will both strip out commercials and recode from the large .TS file to a smaller .MKV, put the output back in the correct directory, and then delete the .TS file. I've got a ways to go before I reach that goal.... but at least I'm making progress?

[ugly]

Comskip isn't perfect. I think it mainly looks for certain features in the video file (black frame transitions, the network logo disappearing/reappearing, etc.). So it isn't uncommon for it to mess up. Generally, I find it's good enough.

From what I know (which isn't very much) Comskip uses an ini file with a bunch of settings. The idea is you can tweak some of the settings for particular networks, but it is all sort of educated guesswork. I don't remember where I got them, I think there's a forum somewhere for Comskip, but I did have a few different ini files for a couple networks. I can't really say if any of the ini files work better than the others. But, the ini files can be used cross-platform, so if you're finding a lot of tutorials and things for Windows, those should apply for your usage too.

Sometimes it works perfectly, sometimes it misses things badly. Often the title sequence of a TV show messes up for me.

I think there is a setting in the ini file that you can use to prevent from skipping the very beginning or very end of a file (for example, so it doesn't skip when there's a very short scene at the end of an episode in between commercial breaks). So maybe the file you're testing is too short. But, I really don't know. I'd test it out on a few different files to see what kind of results you're getting before you do any serious digging.

I actually planned to try and get things automated myself. I use MythTV for my DVR and it has some post-processing capabilities as well. Probably similar to what you're doing. But I never got particularly far. My setup works good enough, and I feel like there are diminishing returns on dumping more time into it.

As for your idea of encoding to get the .TS file (I'm assuming it's MPEG-2 or something, my HDHomerun records to .ts files as well), have you looked into using ffmpeg? I think you can encode a video file and get a x264 output. And ffmpeg can all be done from the command line, so that would probably be easier to get into a bash script. Handbrake probably has a command line option as well, but I've never tried it. But there is probably more help available for ffmpeg. Someone smarter than me would know.

[ralplpcr]

I did try tweaking the .ini file, and I think I've gotten it pretty good. I was able to do a 1 hr clip, and other than a 2 second frozen scene after a transition, it did successfully remove the commercials.

As far as automation goes, I've got a bash file that I'm putting together which should do the trick... I hope. It's a work in progress, and I have not yet tested it completely. The individual pieces are working... but I suspect I'll need to make some adjustments to it before I can claim success. I haven't yet determined if it's better to strip out the commercials before recoding or vice-versa, and I may need to adjust some permissions.... we shall see?

I should have a chance to test the script later today, so we'll see how it works!

Code: Select all

#!/bin/sh
#
#
#create variable, full path with extension stripped out
filename="$1"
mkvname="${filename%.*}"
#recode file, convert from .TS to .MKV, set to 840 wide
HandBrakeCLI -i "$1" -o "$mkvname".mkv -e x264 -f mkv -q 22 -r 29.97 -B 64 -X 840 -O
#wait a few seconds to ensure file is fully written/closed
sleep 7
# Change audio language from Unknown to English
/usr/bin/mkvpropedit "$mkvname".mkv --edit track:a1 --set language=eng --edit track:v1 --set language=eng
#
sleep 7
#
#execute the commercial skip script
/usr/bin/python /opt/PlexComskip/PlexComskip.py "$mkvname".mkv
#
rm "$1"
echo "Done!"

Edit: Tested, and made some adjustments to the script.
It is now functioning on my Mint 17.3 Cinnamon box when manually run.
I've named the script "plexify.sh", and I've run it as follows:
./plexify.sh "/path/to/file/filename.ts"

Just running this has taken a 938MB .TS file and converted it to a 239MB .MKV file with commercials removed. Took about 15 minutes overall - not too shabby! Going to make a few more adjustments before trying it directly as a postprocessing command... but so far, it looks promising?

[ralplpcr]

Just an update - have not yet gotten it to work as a PostProcessing script. I'm not entirely clear as to how it should be called, or what privileges/user is active when Plex finishes the recording. It's working well when I run it from the terminal, just not from within Plex. Still not perfect on the commercials, but on average it's catching all but 1 or 2 for an hour-long show... I can live with it.

I'll have to do some digging & research into trying to figure this out.

FYI - The command I'm entering in is /home/ralplpcr/Desktop/plexify.sh %1 - - which I think is the correct format, but can't be positive.

Another issue I've thought of: assuming I do get the PostProcessing script to run directly from Plex, what will happen if I've got 2 shows recording at the same time? Ideally, it would queue the programs and run the script sequentially... but I suspect that may not be true. I may have to see if I can come up with a way to do that myself.... probably will require using the PostProcessing command to put them into a queue, and then have a cron job run to sequentially run that queue?

[ralplpcr]

I've finally gotten the post processing command to work now. The issue turned out to be two-fold: one of permissions, and the format of the command.

It took a lot of trial & error, but it seems that PLEX stores it's recordings in a hidden temp directory while the DVR is active, and will automatically pass that parameter to whatever post processing script you give it. This means that I *don't* need to pass the path & file to my script, and I need to operate it as if I were the user "plex" instead of my normal logged-in account.

The postprocessing command I call now looks like this:
/home/ralplpcr/Desktop/plexify.sh.

The script for plexify.sh is as follows:

Code: Select all

#!/bin/sh
# [Script must be run as user PLEX]
#
#create variable, full path with extension stripped out
filename="$1"
mkvname="${filename%.*}"
#recode file, convert from .TS to .MKV, set to 840 wide
HandBrakeCLI -i "$1" -o "$mkvname".mkv -e x264 -f mkv -q 22 -r 15 -B 64 -X 840 -O
#Not sure if delay is needed, but give it a moment
sleep 7
# Change audio language from Unknown to English
/usr/bin/mkvpropedit "$mkvname".mkv --edit track:a1 --set language=eng --edit track:v1 --set language=eng
#
sleep 7
#
#execute the commercial skip script
/usr/bin/python /opt/PlexComskip/PlexComskip.py "$mkvname".mkv
#delete the original .TS recording
rm "$1"

The other piece of the puzzle that was missing was how to get the permissions correct. Since it was the built-in plex user account that was performing the recording, the directories in which it does it's work are owned by that account. In order to take any action on those files, I had to either do a sudo, or change the permissions on the folder/file for it to work. I solved this by adding my normal user account to the plex group. This was done simply by going into "Users & Groups", and checking the plex group to add to my own user's permissions.

I'm still playing around with it, and hope to adjust things to be a little more accurate. I'm at about 85% accuracy in auto-detecting commercials to remove them, and would like to improve that if possible. I've also got the concern over how to queue 2 simultaneous recordings, since I don't know whether simultaneous Handbrake/comskip instances will be possible. The compression is quite good now - an hour-long 3GB recording can be compressed down to about 200MB and still be perfectly watchable.

FYI - I used guidance from these instructions and this guide as my base to create my comskip & plexcomskip code, as well as to tweak the HandBrakeCLI program. There were lots of other sites from which I pulled ideas/tweaks, but those were the starting points.

Hopefully, this info will help others who may want to try something similar?

[ugly]

Nice work. I remember permissions being an issue when I attempted to use Comskip with MythTV. But my big issue with MythTV is how it names and stores the recordings. I had to use another script called mythlink.pl to get readable file names but that uses symlinks and I wouldn't have the comskip output files symlinked without making a bunch of other modifications. I was thinking of switching to TVHeadend to see if it would be easier to work with, but maybe Plex is worth a try.

You can run multiple instances of comskip. But it does take a decent amount of processing resources. I'm on a Skylake i5-6600k and I can comfortably run 3 instances of comskip and still use my web browser and watch videos. If I run a 4th instance of comskip, I tend to get some stuttering. So, it's really going to depend on the amount of resources you have.

[ralplpcr]

The only "drawback" to using Plex is that it's not free to use the DVR function. It's only available if you use the "Plexpass" option, which is roughly $5/mo, or you can pay yearly/lifetime. Since I already had a Plexpass, it made sense for me to try it out.

I'm not so worried about the Comskip program running multiple instances, but I suspect there may be some trouble with Handbrake. I know you can queue up multiple conversions in the GUI, but I haven't found a way to do the same in the CLI. And I've never been able to run 2 distinct GUI compressions simultaneously. Guess I'll find out in a couple of days when I've got 2 programs recording at the same time what happens?

Edit: Decided to try this manually, rather than wait. Looks like it may just work?
I copied 2 existing MKV files I already had onto the desktop. I then opened 2 separate terminals, and launched HandBrakeCLI in each one :

Code: Select all

(Terminal 1)
HandBrakeCLI -i "BBC.Life.s01e01.Challenges.of.Life.2009.HDTV.720p.x264.AC3.mkv" -o BB1.mkv -e x264 -f mkv -q 22 -r 15 -B 64 -X 840 -O
(Terminal 2)
HandBrakeCLI -i "BBC.Life.s01e02.Reptiles.And.Amphibians.2009.HDTV.720p.x264.AC3.mkv" -o BB2.mkv -e x264 -f mkv -q 22 -r 15 -B 64 -X 840 -O

They're both running. It appears they *may* be running a little slower than if it were a single instance, but as long as they finish, I'm ok with that. This is a test box, so it's a bit slower anyhow - - my main box has dual Xeons with loads of RAM, so I suspect it'll be a bit more fluid.

DualHndbrake.jpg

Edit 2: Both instances did finish successfully.