Add prefix and suffix to curl's stdout

Questions about applications and software
Forum rules
Before you post please read how to get help
Hyena
Level 1
Level 1
Posts: 8
Joined: Tue Dec 11, 2012 10:29 am

Add prefix and suffix to curl's stdout

Postby Hyena » Sat Apr 01, 2017 9:09 am

I've been trying really hard to construct a reliable pipeline of Linux commands that would make a curl request and add a textual prefix and suffix to the whatever data it writes to its stdout.

I cannot use printf / xargs combination because curl's output may exceed the maximum command line argument length.

I also cannot use subshell { printf 'prefix' ; curl ... ; printf 'suffix' ; } | netcat ... because it does not work reliably. I don't know why this does not work and it is hard to describe how it breaks but it just does not work and I don't know how to debug it. In some situations I get curl error 23 "error writing body" and in other cases the curl output never reaches netcat. Everything works OK when I simply pipe curl's output directly to netcat but whenever I use a subshell it stops working when the curl output is real big. Adding -N parameter to curl (disabling buffering) does not help. Adding stdbuf commands in front of netcat also didn't help. The closest to success I got with the unbuffer command in front of netcat like this:
{ printf 'prefix' ; curl ... ; printf 'suffix' ; } | unbuffer -p netcat ...
but this also stopped working when curl's stdout exceeded a certain size, because then netcat only got partial data from curl.

This problem really seems like a very easy one because the needed wrapper program would be a very simple one. Is it really possible that no one has ever needed such a program so it isn't included in any tools packages?

If I can't find a neat solution then I will have to use named pipes but that's like the last resort for me.

edit:
Just to add more context to the problem. I need to call this command line with popen system call from my C++ program. The netcat has to connect to the very process itself that calls popen feeding it the curls' output wrapped by a custom prefix and suffix.

Ideally, the needed program would work like this:
curl -s ... | wrap "PREFIX" "SUFFIX" | netcat ...

User avatar
xenopeek
Level 24
Level 24
Posts: 21386
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Re: Add prefix and suffix to curl's stdout

Postby xenopeek » Sat Apr 01, 2017 12:32 pm

I don't quite follow; you want a prefix and postfix on each line of curl's output or just one prefix before curl's entire output and one postfix after?

For the former perhaps something like:
curl | sed -r 's/^(.*)$/prefix\1postfix/' | netcat
For the latter perhaps something like:
bash -c "echo prefix; curl; echo postfix" | netcat
If you need to buffer the output of curl before it is passed on to the next pipe command, you can use the pv command for that.
Image

Hyena
Level 1
Level 1
Posts: 8
Joined: Tue Dec 11, 2012 10:29 am

Re: Add prefix and suffix to curl's stdout

Postby Hyena » Sun Apr 02, 2017 6:33 am

xenopeek wrote:I don't quite follow; you want a prefix and postfix on each line of curl's output or just one prefix before curl's entire output and one postfix after?

For the former perhaps something like:
curl | sed -r 's/^(.*)$/prefix\1postfix/' | netcat
For the latter perhaps something like:
bash -c "echo prefix; curl; echo postfix" | netcat
If you need to buffer the output of curl before it is passed on to the next pipe command, you can use the pv command for that.


I need a prefix and a suffix to the whole entire output of curl not each line.

Thanks for the bash -c idea! I haven't tried it. So far I have used curly brackets to spawn a subshell and it was causing me problems. I will definitely try the bash -c option. Also the pv command is indeed a neat tool.

Hyena
Level 1
Level 1
Posts: 8
Joined: Tue Dec 11, 2012 10:29 am

Re: Add prefix and suffix to curl's stdout

Postby Hyena » Sun Apr 02, 2017 6:50 am

Okey I am now using bash -c and it fails the same way { ; ; } subshell does. :(

So below is the problematic pipeline:

Code: Select all

stdbuf -i0 -o0 -e0 bash -c "printf '%s\n%s\n%s' 'su' 'log start' 'log ' ; printf '%s' '2d2d75726c2068747470733a2f2f626c6f636b636861696e2e696e666f2f726177626c6f636b2f3237373737373f666f726d61743d6865780a2d2d6d61782d74696d652031300a2d482022636f6e74656e742d747970653a20746578742f706c61696e3b220a' | xxd -p -r  | curl -s --config - | tail -c 10  ; printf '\nlog end\n%s\n%s\n' 'exit' 'exit'" | stdbuf -i0 -e0 -o0 netcat -q 10 -w 10 127.0.0.1 4000 &


It works perfectly if I execute it from a terminal window, below is what my daemon logs then:

Code: Select all

Sun Apr  2 13:43:29 2017 :: start
Sun Apr  2 13:43:31 2017 :: ac00000000
Sun Apr  2 13:43:31 2017 :: end


However, whenever I give that command line as an argument to the popen system call from my daemon it will only log the following:

Code: Select all

Sun Apr  2 13:43:17 2017 :: start


So only the prefix reaches the daemon. The command line breaks after that, and it is the same without stdbuf, and when I added | pv -q | in front of netcat it also didn't make any difference.

For the record, the above command line is given to popen like that in my code:

Code: Select all

    FILE *fp = popen(command.c_str(), "re"); // Open the command for reading.
    if (!fp) {
        manager->bug("Unable to execute '%s'.\n", command.c_str());
        return;
    }
    if (pclose(fp) == -1) {
        manager->bug("%s: pclose: %s", __FUNCTION__, strerror(errno));
        return;
    }


EDIT
Victory! :D :D :D
This is really weird (possibly a bug in Linux?)
When I changed the popen call to this:

Code: Select all

FILE *fp = popen(command.c_str(), "we");

everything started to work properly. So the issue is with popen called in read mode. When using write mode it did what was expected.
I didn't read anything from that pipe so it doesn't matter whether I call popen in read or write mode. I guess the system(3) would have also worked. But that's just a guess right now. I think it would be more proper to use system instead of popen because I don't write to nor read from the pipe I open with popen. I thought that the ampersand & in the end made my intentions clear for the system. Turns out not. I suppose the output of my command was partially consumed by the parent process (the daemon) but only if the netcat in the end sent the data to the daemon over TCP. For testing purposes I forwarded the curl's output with prefix and suffix to a separate netcat that was running in listening mode and then everything also worked well.

If this is a bug in Linux, where should I report it?

EDIT 2
I just got popen to function properly in read mode too. All I had to do is to redirect the stdout and stderr to /dev/null just before the & ampersand in the end of the command. :?

Code: Select all

{ printf '%s\n%s\n%s' 'su' 'log start' 'log ' ; printf "%s" "2d2d75726c2068747470733a2f2f626c6f636b636861696e2e696e666f2f726177626c6f636b2f3237373737373f666f726d61743d6865780a2d2d6d61782d74696d652031300a2d482022636f6e74656e742d747970653a20746578742f706c61696e3b220a" | xxd -p -r  | curl -s --config - | tail -c 10  ; printf '\nlog end\n%s\n%s\n' 'exit' 'exit' ; } | netcat -q 10 -w 10 127.0.0.1 4000 > /dev/null 2>/dev/null &


Return to “Software & Applications”