[Suggestion] Rewrite Skink

[fsouza]

I think it will be a good idea to rewrite Skink, or make some changes, make the dependencies inside lib dir external dependencies.

I would like to also suggest use of Tornado instead of cherrypy and ion.

What about it?

[heynemann]

I'd like that. Would you like to help?

[fsouza]

For sure.

I'm using Skink for some personal projects (http://skink.franciscosouza.com) and I am really liking it, but I turned very confused when tried to customize some stuffs.

Next days I'll start a new branch on my fork :)

[gabrielfalcao]

+1

[heynemann]

No need to fork. Just added you as collaborator.

Just create a branch with a complete rewrite and we can work together... Just a couple of things first. I agree with tornado completely. We have other tools to look for, though:

a) A performant and efficient daemonizer. Skink keeps two services running - a build service and a git service. They are responsible for verifying if a project needs building and then actually building it. They need to be interactive, meaning I need to be able to query them for what they are doing.
b) A library to interact with GIT properly. The current implementation has serious problems (calling bash to do the work).
c) A proper plugin architecture. I actually like the current one, but I'd rather use events than what we have. I have built a very simple library for an event bus (https://github.com/heynemann/cyrusbus). Not sure on whether it's useful or there's a better solution.
d) A test infrastructure that stimulates us to write tests. It needs to be easy to write unit tests that test little pieces of code isolated and FAST. I also need to be able to easily write tests that ACTUALLY spawn a skink server and test some aspect of it. Just something we need to think about.
e) I actually like cleese's implementation of running a bash script. If we can find a better one (that reports progress) I'm all for it, though. This is the lowest priority of the tools, though.

With that in mind I'm all for building a new tool. I actually would LOVE to add a new requirement to skink, which is building ALL branches of your project. If a branch has a new commit, build it. This would be a HUGE help to teams working with branches. The overhead is not that significant, since if you have many branches that you are not working on, they won't get any builds. The first time around we can build ordered by the last commit (most recent branches get built first). We could even set a cut date (DO NOT BUILD anything that is older than a week). Just throwing ideas around!

Thank you very much for the interest, both of you! :)

[fsouza]

Wow, nice! I am happy to be a collaborator \o/

About your points:

a) +1. I agree completely.

b) Yes. Sometimes I get a strange git error building my project, so I start a new build task and everything works fine. That's really strange. I also think Skink can provide an URL for service hooks

c) +++++++. I've just took a look at cyrusbus, but nothing really deep. I hope play with that next weekend. ;)

d) Yeah. It should be fun to write test.

e) Agreed. cleese is very cool, and this is the lowest priority now.

I liked so far the multibranch building feature, and I loved so far the Skink simplicity. I think we should keep it simple, light, powerful and pluggable, by writing a powerful, easy to use plugin API (as you pointed above).

I'm thinking about four principles on Skink:

1. It should have a nice code;
2. That code should be "overtested", and the tests code should also be nice;
3. Skink should have a fucking awesome documentation;
4. The plugin API should be wonderful =P

Now, some problems that I found:

1. Skink works only with MySQL. I don't know if I've done something wrong, but it isn't working with SQLite and PostgreSQL (I tested both). IMHO, the problem is not that Skink doesn't support SQLite and PostgreSQL, the problem is: Skink supports only MySQL, but the documentation says that it also supports PostgreSQL and SQLite.
2. Skink doesn't supports authentication. Ion provides an @authenticated decorator. I hard coded my personal Skink server by decorating each action (it was ugly). A requires_login option on config.ini would be more nice. Tornado provides a nice user authentication tool.
3. For developments, it is hard to know which are Skink dependencies. Using of pip requirements file or something like that would be cool.

Last, I would like to suggest some features, tools and plugins:

###Features and tools

• Tests:
  1. Writing, discovering and running: nose and specloud - specloud is really nice :D
  2. Mocking: I see that you are using fudge, it looks like JMock. I like mocker so much. Python has a lot of mocking frameworks, and sometimes it is not so good.
• Database:
  1. Support for MongoDB would be nice \o/
• SCM:
  1. Skink should not support only Git forever (but now it is enough ;D)

###Plugins

• Email plugin, for notifications about steps on building processes, with SMTP authentication support.
• Remote building by providing an API and a key. Useful for Github service hooks.

Please let me know what do you think about it.

[heynemann]

AWESOME!
About your principles, I couldn't agree more with all of them!

Now on the problems:

1. Lately I've been thinking that skink shouldn't have to use a DB at all. Not sure on how to do that either. Maybe redis or mongo, if we have to go with a db. I'd say that maybe users would be best served with xml or json reports on the filesystem. Not sure on that anyway.
2. Yeah, big flaw. We could use google auth that's baked in tornado. Who doesn't have a google account these days?
3. Totally agree.

Tools:

1. Agreed
2. Not sure yet on the mocking tool. Will check mocker.
   DB: Explained above
   SCM: Agreed

Plugins: Agreed.

What do you say you suggest iterations for us to build it and what would be the product delivered in each iteration?

[gabrielfalcao]

Hudson doesn't use DB at all, instead, it saves metadata files within each build workspace.

Build workspaces are another nice idea from hudson. Instead of "git resetting + pulling" always in the same workspace, it creates incremental build folders in each project workspaces, and for every single build, a new git clone is issued.

just a few ideas :)

[gabrielfalcao]

example, let's say there is a build named cleese-unit

in skink/projects:

skink
  |_  projects
        |_    cleese-unit
                |_  metadata.xml (or whatever)
                |_  stderr.log
                |_  stdout.log
                |_  builds
                       |_  2010-11-18_23-51-20 (something datetime-ish)
                              |_ code clonned from cleese's repository

[heynemann]

Hmm sounds really promising! I like the workspaces idea. HDD space is VERY cheap these days.

That would also allow us to keep all the build artifacts intact, so that plugins could interact with them, even with all builds artifacts (coverage comes to mind).

[fsouza]

Yeah, I also thought about coverage :)

Now I see and agree: we should not use any DB on Skink.

About the releases, we can plan features for 3.0.0 and work on this, getting a Github issue for each feature.

[heynemann]

Agreed!

[gabrielfalcao]

exciting!

[gabrielfalcao]

What you think about start from something like this:
https://github.com/gabrielfalcao/tornado-bdd-sample

?

[fsouza]

AWESOME!

I liked it, but I would like to see more stuffs on Tornado asynchronous way of life. I'll play with that and Lettuce today later :)

What are we going to release on version 3.0.0? :)

[heynemann]

We should release same functionality as it exists today.

We can build from that, then.

Sounds good?

Cheers

[fsouza]

Yeah, let's do it! ;)