Redirecting stdout from a process invoked from a plugin

[ubu]

I'm trying to invoke a series of small external console apps from inside a plugin. I want them to run without a window appearing and I'd like to redirect the console output to a file. Ideally, having the option of running them in the background would be nice. I'd also like to keep the apps very simple - without requiring the passing of multiple arguments, etc. This thread, which describes a proposed architecture for "mini-plugins" to feed the UbuStream plugin, will explain why I need to do this.

I have tried two approaches but am still unable to come up with a solution that fits the above criteria. One approach:

Code:

String myArgs = "> log.txt";
            ProcessStartInfo startInfo = new ProcessStartInfo(dynSourceApp, myArgs);
            startInfo.WindowStyle = ProcessWindowStyle.Minimized;
            Process myProcess = Process.Start(startInfo);
            myProcess.WaitForExit();

runs the app OK and, by removing the WaitForExit() line, I can run it as a background process. But the myArgs string is passed to the app as an argument instead of being interpreted by the shell, so the stdout is not redirected and no file is created. (Also, running in a minimized command shell window causes the apps to show up in the Windows task bar while they run. Irritating but not a show stopper.) The app completes execution fairly quickly, however.

So I tried this approach:

Code:

ProcessStartInfo startInfo = new ProcessStartInfo(dynSourceApp, "");
            startInfo.UseShellExecute = false;
            startInfo.RedirectStandardOutput = true;
            startInfo.CreateNoWindow = true;
            Process myProcess = Process.Start(startInfo);
            String logLines = myProcess.StandardOutput.ReadToEnd();
            myProcess.WaitForExit();
            File.WriteAllText(log.txt, logLines);

This redirects the output to the file OK and the app runs invisibly (nothing in the task bar even) but it can't be run in the background since I'd need to get rid of the WaitForExit() line but then I wouldn't know when to write the captured output to the file. The command line ">" approach in the first example would have solved this by sticking the file output functionality into the shell process itself but the .Net "shell" doesn't seem to work like a real shell and do the right thing.

Interestingly, using the second approach, the app takes forever to execute. I mean waaaay longer than when run using ShellExecute. Which is wierd because you'd think the shell would add some overhead so running without it would be faster. (And, of course, you can't use RedirectStandardOutput unless UseShellExecute is false - Catch 22!)

I know that I could solve this with brute force and ignorance by having a switch in the target app, triggered by passing an argument, that would explicitly write execution log lines to a log file instead of to the console. But, as I said, I want to keep the design of the apps as simple as possible and with a minimum of confusing argument options.

Any ideas? Sympathy, even?

[Ton]

Well... know is isn't so nice, but you could always write a "wrapper" program that would execute the other program.

The wrapper program could then be fired and forgotten from your main app.

The wrapper would then start the actual "plugin" app (redicting the output as per your second example), and execute the "WaitFor"....

It would require another executable to be deployed, but would accomplish the objectives you've defined...

[ubu]

Well... know is isn't so nice, but you could always write a "wrapper" program that would execute the other program.

The wrapper program could then be fired and forgotten from your main app.

The wrapper would then start the actual "plugin" app (redicting the output as per your second example), and execute the "WaitFor"....

It would require another executable to be deployed, but would accomplish the objectives you've defined...

Actually that's not a bad idea. It creates another .exe file to keep track of but it certainly would solve the problem. Thanks for the suggestion. If nobody comes up with a simpler solution I may well go that route. I just wish M$ would provide a straightforward way to do this. It's not like this is a wildly unusual requirement.

BTW - I forgot to mention that I also tried executing "cmd.exe" and passing the entire app execution command as the argument (in the hopes that a "real" shell would respect the ">" redirection symbol) but I couldn't get that to work either (it was 3am by then, however, and my curses of frustration were upsetting my dog - maybe I should give that approach another shot).

[NiteShdw]

I've been working on doing this in VB.net 2005 as well. I've figured out how to use the RedirectStandardOutput, but the event that captures the output only fires at the very beginning and the very end, missing everything in between. It seems to work fine for applications like cmd.exe, but not where screen contents are being changed.

I've been thinking that it might be better to use a buffer to collect the stream and dump it when the buffer is full, but I'm not sure how to go about doing that.

[ubu]

I've been working on doing this in VB.net 2005 as well. I've figured out how to use the RedirectStandardOutput, but the event that captures the output only fires at the very beginning and the very end, missing everything in between. It seems to work fine for applications like cmd.exe, but not where screen contents are being changed.

The RedirectStandardOutput thing works OK for me. I'm executing a simple console app that writes all its output to the console so all the output gets captured fine. Not sure what you mean by "screen contents are being changed" but, if you are executing a non-console app, the output won't be captured.

I've been thinking that it might be better to use a buffer to collect the stream and dump it when the buffer is full, but I'm not sure how to go about doing that.

That's what my second example does. It continuously captures the output to a string while the program is running and then dumps it to the "log.txt" file when the process has completed.

BTW - I finally solved my original problem by using a .Net BackgroundWorker object. This lets you run code on a separate thread, so you can fire off the method that executes the process (including the WaitForExit line, so you can capture the output) but, since it's running in a separate thread, the user can continue using your application without experiencing any delay.

[NiteShdw]

Using RedirectStandardOutput on a Process does run a separate thread. It uses callback functions for asyncronous communication.

What I meant is that some programs, like BeSweet or Comskip, don't just write lines to the output, they keep writing data to the same line on the screen to show progress (think BeSweet showing the current timestamp). That's the kind of data that I can't get to get redirected.

[ubu]

Using RedirectStandardOutput on a Process does run a separate thread. It uses callback functions for asyncronous communication.

But (if you specify WaitForExit) your original thread has to wait for the process to complete before continuing. And you have to use WaitForExit to ensure getting all the output from the program before writing it out to a file. That's why Background Worker is so neat. It executes the whole darn thing asynchronously.

What I meant is that some programs, like BeSweet or Comskip, don't just write lines to the output, they keep writing data to the same line on the screen to show progress (think BeSweet showing the current timestamp). That's the kind of data that I can't get to get redirected.

Ah! Very interesting problem. I guess regular stdout redirection ain't going to work for that (I assume running it with a ">" symbol from the command line has the same problem?). I wonder what technique those apps are using to re-use the same line? If, for instance, they write a string terminated by a carriage return but no line-feed, you would think that all the original strings would be available in the output but, when it is written to the file, they disappear because the \r and \n characters are respected and applied to the file output. So, in my original example, does the string logLines actually contain everything that was written but the file log.txt has filtered out the lines with no \n at the end? If so, you could use regular expressions (or something) to insert the \n character after each \r in the string before writing it to a file.

[acheyne]

Maybe I'm missing something, but why wouldn't this work (basically #1 with one extra step):

Code:

String myArgs = "> log.txt";
ProcessStartInfo startInfo = new ProcessStartInfo(dynSourceApp, myArgs);
[B]startInfo.CreateNoWindow = true;[/B]
Process myProcess = Process.Start(startInfo);

The only thing I can see is that if you want to capture stdout as well, you'll need to update myArgs for that as well.

[ubu]

Maybe I'm missing something, but why wouldn't this work (basically #1 with one extra step):

Code:

String myArgs = "> log.txt";
ProcessStartInfo startInfo = new ProcessStartInfo(dynSourceApp, myArgs);
[B]startInfo.CreateNoWindow = true;[/B]
Process myProcess = Process.Start(startInfo);

The only thing I can see is that if you want to capture stdout as well, you'll need to update myArgs for that as well.

Yes, indeed, you are missing something. If you had actually run the code you're suggesting you would have found that dynSourceApp receives two arguments: ">" and "log.txt" and that no redirection occurs at all. The file "log.txt" would not be created. That's the problem I was describing in my first post.

If you run something like "MyApp parm1 > log.txt" in a regular Windows cmd shell, the shell first sees the ">" symbol (and interprets the redirection command as meaning "send stout to log.txt") and then it runs Myapp, passing parm1 as an argument. The .Net "shell", however, seems to do no interpretation of shell commands at all, simply passing "parm1", ">" and "log.txt" as three arguments to Myapp.

The correct way to redirect standard output (or standard error) is shown in my second example: setting RedirectStandardOutput to true and, because this won't work when run using a shell, setting UseShellExecute to false.

The problem I was trying to describe was that the second example (which works) forces you to wait for the process to complete before you can grab the output and write it to a file. If the first example (which is essentially identical to the code you suggest) had worked (which it doesn't) it would have meant that the shell would have written the file for you, removing the neccessity of waiting for the end of the process. In the event, I was able to produce an identical result by creating a separate thread (using a BackgroundWorker object) and then using that to run the code that invokes the application and writes its output to a file (see my earlier post).

[NiteShdw]

Could you post the code that you are using now to redirect standard output while waiting for the application to finish?