pipeline xxxx.py config default behaviour change

[snsansom]

I see that now "general_path" has replaced "pipeline_path" for the conf.py location:

def writeConfigFiles(pipeline_path, general_path): '''create default configuration files inpath`.
'''

for src in [
        os.path.join(pipeline_path, "pipeline.ini"),
        os.path.join(general_path, "conf.py")]:`

Thus breaking all existing non-CGATPipeline pipelines that rely on Pipeline/Control.py - is this really intentional?

[AndreasHeger]

Hi @snsansom, this might be a side-effect of cleaning up conf.py's in the repository. Previously, these were all supposed to be links to the standard conf.py in the configuration directory, but had become instantiated and thus created >20 copies of the same file. I removed the extra files and instead of collecting through a link I set the location of the directory within the repository directly.

My thinking was that conf.py should be something that comes from the repository and any variables in it should be set automatically from the pipeline.ini config files. Do you have custom conf.py files?

It could be possible to check for a conf.py in pipeline_path and only fall back to the copy in general_path.

Best wishes,
Andreas

[snsansom]

Hi Andreas, thanks for the comments and consideration #209 helps a lot.

The case where I still have issues is where I am using the Pipeline module essentially independently of the rest of the CGATPipelines in projects which do not have a conf.py.

In those instances it's easy enough to touch a conf.py and the issue goes away.

In general I find the Pipeline module super-useful - I could see value in separating out this brilliant "engine" from the the CGATPipelines collection into a separate module. This would increase its visibility and probably it's use: together with ruffus it forms an excellent pipeline system for omic analysis (could even be a standalone publication there). (From such a view the cloning of the configuration files could be generalised such to accommodate cases where the .ini or .py is unnecessary or where other formats might be used.)

[AndreasHeger]

@snsansom , thanks, I think the fellows are planning to do this as the next project.

I am closing the issue for now.