I am missing "tee"
oderwat opened this issue · 21 comments
I know we can write a filter for it, but having tee in the set of filter would be nice.
Great idea, @oderwat! How do you envisage this looking from the user's perspective?
script.Echo("hello").Tee(some_io_writer).Stdout()Something like this?
I actually missed it the other way around.
script.Exec("date").Tee().WriteFile("date.txt")
Maybe Tee is what you suggested and Probe is what I was missing? But Tee(os.Stdout) is probably good enough anyway.
Let's look at this from the point of view of some real-world program that we'd like to write. Can you think of a situation where someone would want output to go to two places in a script? What kind of data would this be, where would it come from, and where would it need to go to?
Well, I needed it today to see what is getting written to the files after using Exec and replacing some of the strings.
Basically, to show what the script does. I do the same with tee all the time, like:
python somescript.py ./some/dir | tee logfile.log
So, when using "script" the harder and possibly error causing part is the writing of the file and this is easiest done as:
script.Exec("python somescript.py ./some/dir").WriteFile("logfile.log")
But that means I can't see the output. I would need to re-open the file or store everything in a string first. Or use a filter that copies it from in to out and prints it to stdout. So from my point of view, the usual thing is probing. You also may want to write out the current data at different places in the chain to debug or log the process in the chain.
I think your initial understanding about what Tee should do is good. One could implement it with variable argument function like Tee(w ...io.Writer) and default it to os.Stdout if it gets an empty slice. If there is a slice, it could send it to all writers (which could be io.Stderr too).
Right! For the time being, as you say, you can copy the data to the terminal using:
script.Echo(data).FilterScan(func(line string, w io.Writer) {
fmt.Println(line)
}).AppendFile("logfile.log")Those filter commands are powerful. I needed something like this but for each line I found in my /etc/hosts that matched an sdn system. What was cool is that in the function run by 'filterline' I could do another script.Exec. This also shows that the function doesn't have to be inline, but could be a separate function, could be in a different file -- may be 100+ lines long.
func main() {
re := regexp.MustCompile("prod[\t\r\n\f]")
script.File("/etc/hosts").Match("sdn").MatchRegexp(re).Column(2).FilterLine(isPortOpen).AppendFile("Output.txt")
}
func isPortOpen(s string) string {
str, _ := script.Exec("tcpscan -O text " + s + " -p 22").String()
str = strings.TrimSpace(str)
fmt.Println(str)
return str
}
Output to both screen and Output.txt:
sdn-east-host1-prod 22 Open 143.80ms
sdn-west-host1-prod 22 Open 103.58ms
sdn-east-host2-prod 22 Open 143.80ms
sdn-west-host2-prod 22 Open 103.58ms
...and so on...
sdn-west-host1-prod 22 Open 103.58ms
Hi Bitfield,
"Tee" would definitely be a simple but extremely useful feature to have access to.
It could be implemented by rewiring the WriteFile command so that it returns the pipe.
To ensure that the pipe reader isn't emptied, the built in io.TeeReader method could be used to temporary store the readers' contents.
Something like this...
func (p *Pipe) Tee(path string) *Pipe {
// Opening file to write pipe contents to
out, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0666)
if err != nil {
p.SetError(err)
return p
}
// create buffer to hold reader contents
var buf bytes.Buffer
tee := io.TeeReader(p, &buf)
// write to file
_, err = io.Copy(out, tee)
if err != nil {
p.SetError(err)
return p
}
// reset reader
p = p.WithReader(&buf)
return p
}
What are your thoughts?
@Meandi-n, can you show an example of how this would be used in a program?
Hi,
For instance (as the commenter above stated): if you wanted to WriteFile a result, while still displaying it to the user terminal...
script.Echo("hello world").Tee("output.txt").Stdout()
Is this the direction you were thinking of going?
That sounds okay, but maybe it would be more flexible to have Tee take any io.Writer, instead of being restricted to files?
What if I don't want the file truncated? What if I want the results put to my main log file? Or I want one file, for several different script.Exec().Tee()? Passing a writer gives the programmer the flexibility of how the file is open and the ability to error before the pipeline if the file can't be open / written to.
That sounds okay, but maybe it would be more flexible to have
Teetake anyio.Writer, instead of being restricted to files?
That's a really good point, I'll try write something to that affect
I forked script into my repsitory, I added a Tee function to it, and added a test for it in script_test.go.
script] $ go test -v | grep -i tee
=== RUN TestTeeOutput
=== PAUSE TestTeeOutput
=== CONT TestTeeOutput
--- PASS: TestTeeOutput (0.01s)
If Bitfield could review it I could issue a pull request -- just want to make sure that's ok with him before I do a pull request.
https://github.com/rmasci/script
I wrote an implementation exactly the same as rmasci earlier today.
used as follows:
data := "Hello World"
path := "test.txt"
out, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0666)
if err != nil {
panic(err)
}
script.Echo(data).Tee(out).Stdout()
I like this implementation. its simple and flexible, I think it will make some scripts much cleaner.
I hope Bitfield feels the same.
Something like TeeFile might be a useful shorthand, mightn't it?
Tee(<io.Writer>) and TeeFile(<path>)
Would TeeFile amend the file (useful for updating logs) or write the file?
The end user could select via passing the mode as an input, or alternatively by calling TeeFileAmend() or TeeFileWrite().
Would this be too complex?
The append behaviour sounds more useful, doesn't it? This sort of design problem is always a case of drawing a fine line between what's useful, and what inflates the API more than it's worth.
That sounds okay, but maybe it would be more flexible to have
Teetake anyio.Writer, instead of being restricted to files?
Agree
Tee(<io.Writer>)andTeeFile(<path>)Would TeeFile amend the file (useful for updating logs) or write the file?
The end user could select via passing the mode as an input, or alternatively by calling
TeeFileAmend()orTeeFileWrite().Would this be too complex?
Quite like this . Can be extended further by other too I think . Good pattern
I could add patching on top of this pattern using the suggested
TeeFileAmend()
I think just leaving the user to open the file and pass the io.writer to Tee will allow greater functionality -- the user may not want to append the file at the start. Lets say you wanted the tee file to only hold the information from the current run, but you wanted to use it multiple times and append each. instance of Tee to that file:
logFile,_ := os.OpenFile("logFile.log", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
script.Exec("curl -s https://myserver.com/fruits.json").Tee(logFile).JQ(".fruits[].name").Tee(logFile).ExecForEach(`echo "I found a {{.}}"`).Tee(logfile).Stdout()
In this example the user opens logfile and has the the option to os.O_TRUNC or os.O_APPEND when it's opened, but will append to that file the result of curl, JQ query, and the ExecForEach.
ah i see your point.
Maybe we can provide both options ?? Or is that making the APi too confusing... Not sure...
What's really cool is that you could turn it on and off -- let's say you only wanted it for debugging, -v is a bool for verbose in your program, when in verbose mode it logs to a file, if not. Don't write anything.
if verbose {
logFile, err = os.OpenFile("mylog.log", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
errHandle(err, "open file", true)
} else {
logFile, err = os.OpenFile("/dev/null", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
errHandle(err, "dev/null", true)
}
Leave this in your final code and if something starts going wrong, you can pass a -v and see what it's doing.
I was also thinking of the debugging aspects too when i saw this :)
I have a few ideas around this too, and will prob make a repo showing doing patching, appending, etc. I want to put a produced / consumer model around this. So like your the consumer can be for scp, or it could be some other protocol.
os.OpenFile("/dev/null"...
If you just want a writer to nowhere, you can use io.Discard.