Allow interoperability with cucumber or allow extensibility
amareensaleh opened this issue · 20 comments
Karate defines a DSL using cucumber for REST APIs. But is limited to things defined in Karate only.
In our use case, we are trying to build a cucumber DSL for JMS which we hope to make work in conjunction with Karate. This is only possible via eval with java types as of now and we'd like to have a DSL instead.
The main problem is that Karate prevents defining custom cucumber steps by throwing a syntax error. Such behavior is evident in the following classes: KarateBackend, KarateClassFinder and KarateObjectFactory. I was able to override those in my own project locally; and hence call out to custom cucumber steps
Now, If this ticket does not get declined (because the assumption is, it's Karate the Karate way or nothing) then I'd have two propositions which I can chip in and help with:
- A simple work around to, at least, allow calling out to plain user-defined cucumber steps from a karate feature.
- A formal solution to allow classpath-based registration/extension of step definition such that it's not limited to just the ones defined in
StepDefs. That way Karate becomes an extensible framework with pluggable DSLs, which just so happens to have a DSL for REST APIs.
yep you've probably seen that we've been against this from the start. but appreciate your PoV.
I'd like to see examples. the karate-netty project already has an ActiveMQ demo built in and IMO it works fine. so I'll need some convincing about any proposed DSL steps.
what would be interesting is related to #395 if we can arrive at a generic way to handle async flows. I admit that the ActiveMQ demo could possibly be improved.
so while I don't see this as a priority, if you can come up with a proposal + code, I'll be happy to look / consider.
If I understand correctly, Karate is built to help test REST services and via a DSL (less boiler plate, no code type of deal). If the DSL was not the selling point, then I don't know what else would.
Now, REST services have external dependencies such JMS, SSEs, DBs, Distributed Caches, other services...etc. The karate-netty only truly addresses the "HTTP service" dependency with a DSL (via test doubles). The Karate-JMS integration they provide is... "ok" since it's a little ugly to define java in a language (Karate) that ultimately translates to java, but that's a little subjective so let's move on.
The key point that I want to get to is for use cases where you need to probe something which karate does not implement... its a dead end unless you do a "hackery", especially that you have no way of creating your own cucumber steps either... Example: A situation where you'd want to assert certain things in a database after an HTTP POST which are not necessarily exposed via http?
Now, even though, I believe Karate can become a DSL-driven testing framework, I think my simplest proposition for the time being would just be the following. Why not allow an indirect way to use cucumber steps for those who know exactly what they want to achieve. Maybe via the CucumberOptions(glue="mypackage"). That way, kill two birds with one stone: you guarantee that Karate is the de-facto, but also provide an option to those who want to fallback to cucumber or in other words extend Karate
I have overwritten certain classes in Karate to allow calling out to cucumber, if you think this maybe an "ok" change, then I can start working on the code...
I'd rather see some code instead of engaging in walls of text, but here goes, for arguments sake.
If the DSL was not the selling point, then I don't know what else would.
I actually see Karate as more of a JSON / XML payload manipulation and assertion framework. with mechanisms for code re-use, e.g. call. not to mention test-running, parallel execution, tags, reporting and the very unique karate-config.js approach for test-environment switching and variable injection. I would go so far as to say if you take away the HTTP client from Karate , it will still have value :)
And as we will see in a moment, one of the big deals of Karate is how easy it is to call into the JVM from your test, and I like to think of this as a far more effective "glue" than what Cucumber has to offer.
The karate-netty only truly addresses the "HTTP service" dependency with a DSL (via test doubles). The Karate-JMS integration they provide is... "ok" since it's a little ugly to define java in a language (Karate) that ultimately translates to java, but that's a little subjective so let's move on.
No idea what you are talking about here. Not sure if you have even gone through the examples.
The key point that I want to get to is for use cases where you need to probe something which karate does not implement... its a dead end unless you do a "hackery"
Nope, I disagree. Karate's approach is that if you need something extra, you write JS code or Java code. Have you looked at the example for calling JDBC, see here: https://github.com/intuit/karate/blob/master/karate-demo/src/test/java/demo/dogs/dogs.feature#L36 And This is the same approach used for JMS. IMHO this is far more easier and cleaner that the horrible overhead of Cucumber step definitions and what-not. If you move the set-up of the DbUtils into a global call - it becomes very close to a "DSL".
Why not allow an indirect way to use cucumber steps for those who know exactly what they want to achieve.
As I said, I'll need convincing and need to see examples. There are many reasons why I'm not in favor of this. You've probably already seen the mega-hackery involved just to get Cucumber to work the way we want and I want Karate to be 'Cucumber-Free' as far as possible, I even see it as a worthy cause - to free people from the tyranny of Cucumber and BDD when not needed see link.
Also, if we go down this path CucumberOptions(glue="mypackage"), people will start expecting the Karate variable mechanism to be available to any custom steps and I haven't even thought about this.
I'm closing this issue, but you can always refer to this in case you want to create a PR etc.
Speaking personally, I would very much prefer the ability to write custom cucumber code. We have existing cucumber projects which I'd like to port over most of it to karate. Enforcing only the karate step definition requires us to either segregate into separate sub-modules (messy but technically correct) or to refactor all existing tests within a single PR (impractical).
I can appreciate that code is better than words. Still curious as to why it seems you're so adamant on prescriptive implementation. The current implementation of karate is the only cucumber allowed seems a violation of the robustness principle of "Be conservative in what you send, be liberal in what you accept". Since it's clearly my itch and not yours, I guess it's up to one of us for the PR.
I can appreciate that code is better than words.
@60secs - Yes. I was about to respond to each of your points but then figured that it is a waste of time. This is one of those discussions where I sense folks have bought into cucumber and "keyword driven testing" to such an extent that they are unable to take a step back and think about what the actual need is - which is simply "to make it easier to write some code to test something".
Yes, Karate is opinionated where it does not solve for read-ability by a non-coder. I sincerely feel this is why Karate has been a success in teams where multiple "BDD based" attempts have failed in the past, and I plan to protect this as far as possible.
I fully accept that there could be lack of open-mindedness on my side which is precisely why I'd like to see code examples - the "old way" vs the proposed new way and a clear articulation of the benefits. I think I've proven that in many other instances (here and on StackOverflow) - that I've changed my mind when provided with data.
Strong Opinions. Weakly Held. Peace.
I too believe that Karate (or any such framework for that matter) MUST NOT attempt to solve for readability by a non-coder, I am against BDD unless done the right way (i.e., the three amigos).
All I have been trying to say all along has been "promote well-defined DSL" not BDD. And since Karate cannot accommodate for all possible technologies out there, allow others to provide their own well-defined DSL by "restricted extension" instead of the ugly verbose javascript/java approach which we're trying to run away from to begin with. As such, Karate will end up with a bunch of DSL extensions in no time.
Here are some quick & dirty examples for JMS:
- JMS Producer
Given queue 'queue/my-dummy-target-queue'
* def content = read('my-message.json')
Then send content
- JMS Consumer
Given queue 'queue/my-dummy-source-queue'
When receive content
Then match content.id = '1234'
The DSL above is as declarative as one could possibly wish for, anything extra is just boiler plate.
Let alone that it reads much better than the demo provided by Karate-netty (or at least that's what I see).
I have taken a good look at the code, If you'd like to provide the ability for extension, it could be re-architected as follows:
- Define a
BaseStepDefswhich provides the "context" at runtime for the "implementors". - You then divide the original
StepDefsto multiple implementor classes (e.g.,HttpStepDefs,MatchingStepDefs,AssertStepDefs,JmsStepDefs,DatabaseStepDefs...etc). - At StepDef registration, you scan for anything that implements
BaseStepDefs, construct it and inject the "context".
Or you could stick to what you have a live a happy life...
Thanks...
Oh, and if sharing the request and response language constructs, the examples above could be shortened even more:
- JMS Producer:
Given queue 'queue/my-dummy-target-queue'
And request = read('my-message.json')
Then send
- JMS Consumer
Given queue 'queue/my-dummy-source-queue'
When receive
Then match response.id = '1234'
Or you could stick to what you have a live a happy life...
LOL. was that really necessary ?
It is very easy to provide "drive by" examples for anything and conveniently ignore the details. For example where you have:
Given queue 'queue/my-dummy-target-queue'
* def content = read('my-message.json')
Then send contentWe are missing the configuration, is it ActiveMQ, RabbitMQ, MQSeries, where is the server / broker, what is the connection type, the delivery mode, acknowledgement type etc. Just passing a queue name will not cut it. I know that this is a hurriedly created example but send certainly doesn't sound like the right choice for syntax.
Anyway, hope it is clear why ideally we need a working example for considering a PR.
Adopting your style, If the above example were to be implemented in Karate:
Given def queue = getQueue('queue/my-dummy-target-queue')
And def content = read('my-message.json')
Then eval queue.send(content)Any questions ?
@amareen1988 will it be possible for you to share example of how you were able to call custom cucumber steps.
for the attention of everyone watching this conversation !
please note that we plan to move the karate internals OUT of cucumber, and I will be using #444 to track this.
for those who are so fond of step-definitions, this may actually be a good thing, just write your own step definitions and karate won't get in the way :)
once again for the attention of all those watching this thread !
there are some drastic changes happening the core cucumber project - which makes me all the more convinced that de-coupling from Cucumber is the right thing to do for Karate !
Check out this pull request that removes Gherkin parsing from Cucumber-JVM. The plan is apparently to use a Go binary (compiled for multiple platforms) which is controversial.
https COLON SLASH SLASH github.com/cucumber/common/pull/424
Also refer this discussion: https://groups.google.com/forum/#!topic/cukes/fBsz3lqgkRg
for those not yet convinced that Karate is okay the way it is without needing to introduce user-defined keywords, here is a gRPC example from another thread:
looks like the Cucumber fans are all silent ^_^
an update for those listening to this thread: we've just added some neat syntax to perform an async / await and we have 2 examples - one is a JMS listener and the other is a websocket send and async listen example: #395 (comment)
I made a point earlier in this thread that Karate would have value even if the HTTP client was removed, and here is proof - Karate as a unit testing framework: https://twitter.com/ptrthomas/status/1132515667310047233
it is worth mentioning that Karate has crossed Cucumber-JVM in terms of GitHub stars - which is ironic since Karate was once based on that code base. another framework that we have crossed is ThoughtWorks https://gauge.org
Karate also IMO is a superior alternative to the https://robotframework.org - and I encourage anyone listening to experiment.
one recent improvement very relevant to this thread is that the eval keyword can be omitted for method calls on variables in scope, this brings even closer to a "DSL like" experience, refer the 4th line below in the Background:
EDIT: for another more complex example, see: https://twitter.com/getkarate/status/1417023536082812935
its a me again, sounding like a broken record ! just wanted to post this "DSL like" example that is achievable with Karate (commit: bfec626)
- note how the JS engine serves as the perfect "glue"
- the
callkeyword provides some readability, function arguments look less "noisy" - features can be assigned to variables (with clever names) and used in a
call
for anyone reading this thread in the future, note that Karate caught up to the "original" [cucumber-ruby] project in just 4 years: https://twitter.com/ptrthomas/status/1424259563503587330
there will always be people who "feel" that cucumber step-definitions are needed. to the few folks who put "like emojis" on the original post, I just ask you to consider. leave aside BDD, what if cucumber's approach was not the right one for test-automation in the first place.
I leave you with a quote from a comment I made earlier in this thread:
what the actual need is - which is simply to make it easier to write some code to test something.
i have a different problem, i have acceptance test written in Cucumber BDD and i need to create Karate gatling for performance testing but if i could map step def in my karate feature then i can write gatling to run the Karate.
but can Karate make those Step definition as API to call in the script.
today I found an article from 2019 by a team that saw Karate, liked the idea, but decided to write a "Cucumber-like" way to test JMS / ActiveMQ - in exactly the way that the original commenter here proposed: https://tech-blog.lectra.com/article/763-testing-amqp-apis
I'll paste 2 images from that post at the bottom here for reference. I'll also put down my thoughts as to why Karate will never support this style - and also try my best to discourage others from trying to do this. in the post above, the developers claimed that it was working well and that they would shortly open-source their framework. but there was no news after that
the main problem is suddenly the syntax "explodes". you feel that you have done a "good thing" by making it readable, but you have ended up making your tests more brittle and your users now need to remember all the ways they can (and can not) write tests. let us study this excerpt for example:
When message is sent to usage on usage.command.create
Then sleep for 2s
And credit for $tenant$ has increased and equal to <creditpoint>
And receive 1 message in $queue$
And json $.payload.points == int <creditpoint>- Look at all the "keywords" you need to learn now:
message is sent tosleep forcredit forhas increased(and) equal toreceivemessage
- ambiguity and english grammar rules - for e.g. are the following also supported ?
messages are sentreceive 2 messages
one of the design decisions of Karate, which I think has worked wonderfully well is the first-class support for JSON. you don't need to invent keywords and worry about building support for it as a "language". you just design a schema that works for you. for example, this kind of JSON can convey all the information captured above:
{
on: 'usage.command.create',
sendTo: 'usage'
}The way I would encourage Karate users to implement the above flow will be like this. You will need to write one Java helper, that's it.
* def creditBefore = utils.creditFor(tenant)
* def queue = utils.listen('usage')
* queue.on('usage.command.create').sendTo('usage')
* utils.sleep(2000)
* def creditAfter = utils.creditFor(tenant)
* assert creditAfter > creditBefore
* match creditAfter == creditPoint
* def messages = queue.messages()
* assert messages.length == 1
* match messages[0].payload.points == creditpointyes, this is a few more lines of code, but it is far more readable. also you didn't have to re-invent the handling of variables, json and assertions. you extend the framework in ways that make sense to you, but you still stick to some rules. it is indeed a trade-off. my experience is that trying to make non-technical users write and maintain tests always ends in failure. a better strategy is to make the test-reports more friendly to a business-user. there are already ways to do that in karate which we are still improving.
and since karate supports re-use, which means you can call a feature from another feature. you can certainly wrap the logic above into one clean JSON like this:
* call creditFlow { tenant: '#(tenant)', queue: 'usage', on: 'usage.command.create', expected: 100 }
