MPI cluster mode
Closed this issue · 6 comments
angerhang commented
I don't know whether this is a feature that should be merged into the master or should stay in the fork I have. The cluster mode constructs the models for all subjects in parallel when one is to use hold-one-out testing approach, the cluster mode decreases the computation time by a factor of number of subjects with just a little overhead.
However this use case is rather specific to the MPI computing infrastructure, and thus it's @bibliolytic 's decision on whether or not we should include this feature. If so, I would just do a pull request, otherwise I will just let the others use my fork.
vinay-jayaram commented
As in you use parfor? That's already an option within MT_baseclass, I don't
see how it helps to have two levels of parallelism...
…On Aug 15, 2017 11:27, "angerhang" ***@***.***> wrote:
I don't know whether this is a feature that should be merged into the
master or should stay in the fork I have. The cluster mode constructs the
models for all subjects in parallel when one is to use hold-one-out testing
approach, the cluster mode decreases the computation time by a factor of number
of subjects with just a little overhead.
However this use case is rather specific to the MPI computing
infrastructure, and thus it's @bibliolytic
<https://github.com/bibliolytic> 's decision on whether or not we should
include this feature. If so, I would just do a pull request, otherwise I
will just let the others use my fork.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEGAFoFsIUOG7Fo825l_69u7uqkCWVBfks5sYWR4gaJpZM4O3YDC>
.
angerhang commented
You are right, parfor can already do part of the job, however,
the current parallelism can only be done in a stand-alone mode without leveraging
the power of computing cluster.
… On Aug 15, 2017, at 1:35 PM, bibliolytic ***@***.***> wrote:
As in you use parfor? That's already an option within MT_baseclass, I don't
see how it helps to have two levels of parallelism...
On Aug 15, 2017 11:27, "angerhang" ***@***.***> wrote:
> I don't know whether this is a feature that should be merged into the
> master or should stay in the fork I have. The cluster mode constructs the
> models for all subjects in parallel when one is to use hold-one-out testing
> approach, the cluster mode decreases the computation time by a factor of number
> of subjects with just a little overhead.
>
> However this use case is rather specific to the MPI computing
> infrastructure, and thus it's @bibliolytic
> <https://github.com/bibliolytic> 's decision on whether or not we should
> include this feature. If so, I would just do a pull request, otherwise I
> will just let the others use my fork.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#3>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AEGAFoFsIUOG7Fo825l_69u7uqkCWVBfks5sYWR4gaJpZM4O3YDC>
> .
>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHTQHt3rVBLEW3xsIMtHASbHlMWhpjpKks5sYYKDgaJpZM4O3YDC>.
vinay-jayaram commented
Ah it uses the htcondor bindings for Matlab?
…On Aug 15, 2017 16:13, "angerhang" ***@***.***> wrote:
You are right, parfor can already do part of the job, however,
the current parallelism can only be done in a stand-alone mode without
leveraging
the power of computing cluster.
> On Aug 15, 2017, at 1:35 PM, bibliolytic ***@***.***>
wrote:
>
> As in you use parfor? That's already an option within MT_baseclass, I
don't
> see how it helps to have two levels of parallelism...
>
> On Aug 15, 2017 11:27, "angerhang" ***@***.***> wrote:
>
> > I don't know whether this is a feature that should be merged into the
> > master or should stay in the fork I have. The cluster mode constructs
the
> > models for all subjects in parallel when one is to use hold-one-out
testing
> > approach, the cluster mode decreases the computation time by a factor
of number
> > of subjects with just a little overhead.
> >
> > However this use case is rather specific to the MPI computing
> > infrastructure, and thus it's @bibliolytic
> > <https://github.com/bibliolytic> 's decision on whether or not we
should
> > include this feature. If so, I would just do a pull request, otherwise
I
> > will just let the others use my fork.
> >
> > —
> > You are receiving this because you were mentioned.
> > Reply to this email directly, view it on GitHub
> > <#3>, or mute the
thread
> > <https://github.com/notifications/unsubscribe-
auth/AEGAFoFsIUOG7Fo825l_69u7uqkCWVBfks5sYWR4gaJpZM4O3YDC>
> > .
> >
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub <https://github.com/
vinay-jayaram/MTlearning#3#issuecomment-322444288>, or mute the
thread <https://github.com/notifications/unsubscribe-auth/
AHTQHt3rVBLEW3xsIMtHASbHlMWhpjpKks5sYYKDgaJpZM4O3YDC>.
>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEGAFjb-91VBwp871vFiqKCGBi44Brd7ks5sYaeagaJpZM4O3YDC>
.
angerhang commented
yeah
vinay-jayaram commented
Then I think it doesn't make sense to publicize it outside the lab
…On Aug 15, 2017 16:25, "angerhang" ***@***.***> wrote:
yeah
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEGAFog5NHR89kx4sDOn6Ngn4lqjhKYNks5sYapVgaJpZM4O3YDC>
.
angerhang commented
ok, then I will just let the others use my fork :D