Object oriented code?

[clarkfitzg]

Base R methods such as aggregate, split, mean, [ have corresponding functions distribagg, distribsplit, distribmeans, distribgetrows in partools.

It might be nice to have a distributed object, like a distributed data frame, and write the partools versions as methods.

Thoughts?

[matloff]

Yes, a subclass of 'data.frame', interesting. What is your assessment of the cost/benefit ratio (cost being development time)? Norm

…

On Mon, Apr 17, 2017 at 12:08:54PM -0700, Clark Fitzgerald wrote: Base R methods such as `aggregate, split, mean, [` have corresponding functions `distribagg, distribsplit, distribmeans, distribgetrows` in partools. It might be nice to have a distributed object, like a distributed data frame, and write the partools versions as methods. Thoughts? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #11

[clarkfitzg]

Benefits

• Initially more familiar to users, which will help with adoption
• Well defined examples in base R to follow make it natural to extend this approach to other methods
• May simplify some partools code and make it more flexible if we can pass arguments directly through to methods on worker nodes using mechanism such as ...

Costs

• Likely a few weeks of initial coding
• Might end up creating and maintaining two versions of code which do the same thing

I've been thinking about this as I read Chambers' Extending R book (very good by the way).

I am just raising the issue to discuss and see if this is a possibility, or something that was considered and rejected.

[matloff]

I considered it briefly, but decided that there were too many technical problems. For example, compare the arguments of aggregate to distribagg(). Norm

…

On Mon, Apr 17, 2017 at 05:31:53PM -0700, Clark Fitzgerald wrote: ### Benefits - Initially more familiar to users, which will help with adoption - Well defined examples in base R to follow make it natural to extend this approach to other methods - May simplify some partools code and make it more flexible if we can pass arguments directly through to methods on worker nodes using mechanism such as `...` ### Costs - Likely a few weeks of initial coding - Might end up creating and maintaining two versions of code which do the same thing I've been thinking about this as I read Chambers' Extending R book (very good by the way). I am just raising the issue to discuss and see if this is a possibility, or something that was considered and rejected. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #11 (comment)

[clarkfitzg]

For example, compare the arguments of aggregate to distribagg().

Could you explain further? Here's the relevant signatures:

## S3 method for class 'data.frame'
aggregate(x, by, FUN, ..., simplify = TRUE, drop = TRUE)

## S3 method for class 'formula'
aggregate(formula, data, FUN, ...,
           subset, na.action = na.omit)

## Partools:
distribagg(cls,ynames,xnames,dataname,FUN,FUNdim=1,FUN1=FUN)

[matloff]

Well, there you are. The partools function uses strings, e.g. ynames, and it requires two functions rather than one. On top of that, some features of aggregate() are not implemented and would be difficult to add. It would be useful if you were to add some distrib*() function yourself, to see the problems. Norm

…

On Mon, Apr 17, 2017 at 05:56:54PM -0700, Clark Fitzgerald wrote: > For example, compare the arguments of aggregate to distribagg(). Could you explain further? Here's the relevant signatures: ``` ## S3 method for class 'data.frame' aggregate(x, by, FUN, ..., simplify = TRUE, drop = TRUE) ## S3 method for class 'formula' aggregate(formula, data, FUN, ..., subset, na.action = na.omit) ## Partools: distribagg(cls,ynames,xnames,dataname,FUN,FUNdim=1,FUN1=FUN) ``` -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #11 (comment)

[clarkfitzg]

It would be useful if you were to add some distrib*() function yourself, to see the problems.

I just had a go at writing a function to add a new column to a dataframe which is already distributed, say az. The call was looking something like this:

distribsetcol(cls, dataname = az, colname = LETTER, FUN = function(x) toupper(x$letter))

where FUN is a function expecting the chunk of az in that node. Then I realized that it's cleaner and easier to instead use the stuff already in parallel:

    clusterEvalQ(cls, {
        az$LETTER <- toupper(az$letter)
        NULL  # Necessary to avoid returning LETTER
    })

Maybe I'll add something like this into the docs.

[matloff]

Yes, a lot of useful things can be done directly using clusterEvalQ(). This should be emphasized in the vignette. It would be nice if you put your example into the vignette. Norm

…

On Mon, Apr 17, 2017 at 09:36:31PM -0700, Clark Fitzgerald wrote: > It would be useful if you were to add some distrib*() function yourself, to see the problems. I just had a go at writing a function to add a new column to a dataframe which is already distributed, say `az`. The call was looking something like this: ``` distribsetcol(cls, dataname = az, colname = LETTER, FUN = function(x) toupper(x$letter)) ``` where `FUN` is a function expecting the chunk of `az` in that node. Then I realized that it's cleaner and easier to instead use the stuff already in parallel: ``` clusterEvalQ(cls, { az$LETTER <- toupper(az$letter) NULL # Necessary to avoid returning LETTER }) ``` Maybe I'll add something like this into the docs. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #11 (comment)

[clarkfitzg]

Example is in the vignette now.

[matloff]

Thanks. Norm

…

On Thu, Apr 20, 2017 at 02:52:07PM -0700, Clark Fitzgerald wrote: Example is in the vignette now. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #11 (comment)