Defining tests in multiple files?

Question

Defining tests in multiple files?

Closed this issue 9 years ago · 26 comments

Is there a way to use canopy's runner (or some other runner) to define tests in multiple files and execute them all from the Program.fs file?

I can do this:
Program.fs:
open canopy
open runner
open search_tests
def_search_tests()
run()

search_tests.fs:
open canopy
open runner
let def_search_tests () =
context "search tests"=
test( fun _ -> ...)

But I don't want to maintain all those opens and function calls in Program.fs. Seems like there should be a way to do this. But my F# isn't strong enough!

Answer 1 · 2012-11-30T00:20:41.000Z

The way you are doing it is fine, but like you said you will have 1 open and 1 call to the tests for that module for each set of tests you have. At my job I do the same thing. I create 1 .fs file per page that I want to test and put my urls/cssSelectors/tests in that file, then call the mini suites from Program.fs

Another way is to not wrap your tests in 'let def_search_tests () =' and that should have them all load up by just using the open statement. Then you can simply call run() in Program.fs.

A third option would be to create another file to host all your 'test calls' like tests.fs or testList.fs. You'd simply be moving the calls out of Program.fs, not a major win.

Ultimately you do have to open the module with your test definition and potentially run the function you defined that houses all of your tests.

Another option I guess would be to write nUnit (or similar) tests in F# and write canopy calls to control the browser. You could mix nUnit assertions with canopy like this:

Assert.AreEqual("lefthandedgoat", (read "#username"))

I believe you would get GUI test runner support that way too, but I have never tested it.

You may be able to put attributes on modules and then using reflection load/run them. That is outside of my current know-how but I can definitely look into it if you think that is useful/valuable.

Hope this helps. Let me know if you have any feedback/suggestions/questions or need any other help!

Thanks for using canopy!

Answer 2 · 2012-11-30T03:54:09.000Z

"Another way is to not wrap your tests in 'let def_search_tests () =' and that should have them all load up by just using the open statement. Then you can simply call run() in Program.fs."

I tried this approach, but it turns out that F# only runs non-bind statements when the module is initialized, and modules aren't initialized when you call open. They're not initialized until you call into them. I'm experimenting with the technique in this stackoverflow question: http://stackoverflow.com/a/6262578/4407. But no luck yet.

I may experiment with some reflection techniques to find the test functions without having to open and call into the modules (maybe decorating functions with [Test] attributes, and writing a runner to find all those and pass them to test(...)?). But I also like your idea of using NUnit as the runner...

Thanks so much for the quick reply. I'm really excited about canopy! And from what I've seen of the code so far, it's remarkably clean and intuitive! I'd love to find a way to contribute as well.

Answer 3 · 2012-11-30T16:25:16.000Z

Looks like [ ] attribute on your module will at least alleviate the need to explicitly open it.

I will look more into tonight and let you know what I figure out.

Answer 4 · 2012-11-30T18:15:20.000Z

I found a way to dynamically find and load contexts in many files: https://gist.github.com/4177403

It uses reflection to find all the modules in the assembly and find any let bindings decorated with a [Context] attribute, then it executes those functions to cause the test definitions inside them to run. It works!

I also got it working with NUnit. There were no surprises, just install-package nunit and write an F# type with member methods decorated with []. The syntax when using NUnit is not as pleasing, but it gives you the GUI runner, lets you run just the tests you specify, and is easy to wire up with the build since we're already using nunit for unit testing.

Thanks again!

Answer 5 · 2012-12-01T18:21:37.000Z

I accepted your pull request and built it + a few minor features (ability to pin the browser window to the left or right, auto pin right on start, and a bug with the canopy directory in appdata) in 0.6.1.

Later I think I will add a project with Nunit and provide overrides of the built in canopy assertions that wrap nUnit ones so you can keep the nice syntax like "#selector" == "value". That may make using nUnit more palatable.

One downside to not using the built in runner is that you can't use things like:
xtest (tests marked as xtest instead of test won't run)
wip (work in progress, if one or more tests in a context are marked as wip, only those tests will run, they run slower so you can watch them, and they highlight the html elements they are interacting with, great for debugging)
many 5 (or any number, will run that one test many times, helps with figuring out why a test passes sometimes but not consistently)

There is a built in Team City reporter that which allows you to get live reports/statistics when using team city as your CI build tool.

I also have plans for making an HTML report so you can zip up the failure screenshots with an interactive html report and attach that to your build report.

Lastly, I use rake to kick off my canopy runs. Rake has really good support for many different CI tools and is really simple to use. example: https://github.com/lefthandedgoat/canopy/blob/master/Rakefile.rb
Just let it build your F# console app and run it. I think many build server will pick up whats written to the console so you get your pass/fail stats etc. with the built in runner.

Thanks for the pull request! Good find!

Answer 6 · 2013-05-21T16:39:46.000Z

It is unclear where you ended up with this approach. Do either of you have a sample for how to organize and run a suite of tests. One written for a F# n00b?

Answer 7 · 2013-05-21T17:18:30.000Z

The best public example is probably here:

https://github.com/amirrajan/Oak/tree/master/Sample%20Apps/TaskRabbits/TaskRabbits.UITests

Its a little different because its 1 set of tests to test 3 different implementations of the same page written in 3 different js frameworks.

If you look at common.fs and program.fs you will see a structure that I use. Program.fs is your main launching area where you can call all your tests from. You can create one .fs file for each page you are testing. So maybe something like login, home, editUser, etc. Within each page's file I usually break it into a few sections, helper methods directly related to that page, selectors for the elements on the page, and the tests related to the page.

Here is an example of a helper method I use for our login page

type roles =
    | Manager
    | QA
    | Uer

//selectors and urls
let surveyUrl = configuration.server + "/Navigator/Surveys.aspx"
let loginUrl = configuration.server + "/Login.aspx"
let userName = "input[id $= '_UserName']"
let password = "input[id $= '_Password']"
let loginButton = "input[id $= '_LoginButton']"

let loginAs role =
    //this is a function within a function that is used by the below match code
    let login u p = 
        url loginUrl
        userName << u
        password << p
        click loginButton
        on surveyUrl

    match role with
    | Manager -> login "testMananger" "password123!"
    | QA -> login "testQA" "password123!"
    | User -> login "testUser" "password123!"

Then within login.fs you can define your tests

//tests
let all _ =
    context "login tests"

    "manger login tests" &&& (fun _ ->
        loginAs Manager
        //validate)

    "QA login tests" &&& (fun _ ->
        loginAs QA
        //validate)

Then in program.fs call login.all() to run all of your login related tests

You can literally structure it however you like, all in one file, in many small files, seperate files for tests/selectors/helpers, but I think the above structure of .fs file per page is pretty reasonable for starting off.

I find that ultimately you build a dsl for navigating and asserting your webpage as a bunch of helpers, then your tests are just a series of actions that your users would do + validations. You can see how amir tests his whole website with 4 small tests: https://github.com/amirrajan/Oak/blob/master/Sample%20Apps/BorrowedGames/BorrowedGames.UI.Tests/Program.fs

Hope this helps.

Edit: Also, if you need more help I'd be happy to setup a skype session some day and answer any of your questions.

Answer 8 · 2013-05-21T17:27:00.000Z

Here are the instructions to run the test suites.
http://amirrajan.github.io/Oak/#taskrabbits

Answer 9 · 2013-05-22T12:57:42.000Z

Thanks for the examples... it helps to let me grock the f#, It seems like
this framework is pretty lightweight and as a result I can implement my own
approaches to taking in commandline args to insert variables into my tests
in interesting ways.

Eric Hexter

blog | http://Hex.LosTechies.com
info | http://www.linkedin.com/in/erichexter

On Tue, May 21, 2013 at 12:27 PM, Amir Rajan notifications@github.comwrote:

Here are the instructions to run the test suites.
http://amirrajan.github.io/Oak/#taskrabbits

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/58#issuecomment-18223714
.

Answer 10 · 2013-05-22T15:14:22.000Z

Yes, its not overly opinionated and allows a great deal of flexibility. If there is anything that isn't directly support by canopy functions, 'browser' is an instance of a Selenium IWebDriver so you can use any snippet you find on the internet directly and extend canopy. The test runner is optional, you can simple write canopy commands in any xUnit framework. You can also implement your own reporter to hook into your CI tool. There is an example implementation for TeamCity here: https://github.com/lefthandedgoat/canopy/blob/master/canopy/reporters.fs#L83

See you at Pablos!

Answer 11 · 2013-05-22T15:25:52.000Z

Just a note on the flexibility (and the fact that canopy tests are just an exe) running headless is fairly straight forward: https://github.com/amirrajan/Oak/blob/master/Sample%20Apps/BorrowedGames/headless.ps1

Answer 12 · 2013-05-22T17:30:32.000Z

It's definitely open and flexible but I would also look at the other side
of the conversation as there are a number of elements you need to define
and work out yourself. The lack of these features is actually friction in
my opinion when I also need to get up to speed on f# as well. I will need
to spend some time define how I want my test runner to work.

I really like what you have done to cut out the pain of dealing with the
.net selenium abstractions. I just would like a little more guidance or
samples around how so you use this with any test runner. If I had more time
this looks like a framework I would explore but the last mile is missing
for me and I have other tools that can get me up and running pretty quickly.

On Wednesday, May 22, 2013, Amir Rajan wrote:

Just a note on the flexibility (and the fact that canopy tests are just an
exe) running headless is fairly straight forward:
https://github.com/amirrajan/Oak/blob/master/Sample%20Apps/BorrowedGames/headless.ps1

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/58#issuecomment-18286175
.

Eric Hexter

blog | http://Hex.LosTechies.com
info | http://www.linkedin.com/in/erichexter

Answer 13 · 2013-05-22T18:18:04.000Z

What test framework/runner do you use and I will create some examples for it.

The built in test runner does provide some niceties that wouldn't be provided in normal unit testing frameworks like WIP which runs the test slowly and highlights the elements that its trying to interact with, helping you debug tests that aren't behaving correctly.

http://lefthandedgoat.github.io/canopy/testModes.html

"... there are a number of elements you need to define and work out yourself."
What features do you feel are missing? I am always happy to add new features/ideas to make the framework more complete.

Thanks for the feedback!

Answer 14 · 2013-05-22T18:20:11.000Z

Any chance we may be able to Skype about this. Definitely want to make the framework as approachable as possible. What did you think of the canopy tests in Oak? Personally, I've leaned towards having all the tests in one file. I'll move helper methods into separate modules/files as need to help partition page level functionality.

The last UI automation suite tested an app with ~20k lines of javascript, for that app we had ~40 UI tests, and 600+ unit/integration tests at the class and http level. I've found that my UI tests are very coarse grained and sweeping (leading to a small collection of tests overall).

For the BorrowedGames reference implementation:
128 unit/integration tests
4 UI automation tests

Answer 15 · 2013-05-24T12:48:42.000Z

I have some time next week or maybe late Friday or sat night.

On Wednesday, May 22, 2013, Amir Rajan wrote:

Any chance we may be able to Skype about this. Definitely want to make the
framework as approachable as possible. What did you think of the canopy
tests in Oak? Personally, I've leaned towards having all the tests in one
file. I'll move helper methods into separate modules/files as need to help
partition page level functionality.

The last UI automation suite tested an app with ~20k lines of javascript,
for that app we had ~40 UI tests, and 600+ unit/integration tests at the
class and http level. I've found that my UI tests are very coarse grained
and sweeping (leading to a small collection of tests overall).

For the BorrowedGames reference implementation:
128 unit/integration tests
4 UI automation tests

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/58#issuecomment-18297400
.

Eric Hexter

blog | http://Hex.LosTechies.com
info | http://www.linkedin.com/in/erichexter

Answer 16 · 2013-05-24T15:41:06.000Z

Memorial day (and that week) will be pretty packed with family. How about the first week of next month. I'm free every day then. Saturday may be possible, just need to see how things play out next week. I can prepare something for discussion. Anything specific out side of tests in multiple files?

Answer 17 · 2013-05-24T15:43:27.000Z

The next week works for me to, no need to prepare anything. I can show you
some variations of what I have done using other frameworks and some of what
I have I'm some JavaScript based ui testing as well.

On Friday, May 24, 2013, Amir Rajan wrote:

Memorial day (and that week) will be pretty packed with family. How about
the first week of next month. I'm free every day then. Saturday may be
possible, just need to see how things play out next week. I can prepare
something for discussion. Anything specific out side of tests in multiple
files?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/58#issuecomment-18412617
.

Eric Hexter

blog | http://Hex.LosTechies.com
info | http://www.linkedin.com/in/erichexter

Answer 18 · 2013-05-24T15:58:23.000Z

Great! I'll follow up closer to next month. Thanks again for taking the time to meet. It is really appreciated.

Answer 19 · 2013-06-06T15:31:12.000Z

Wanted to follow up @erichexter , I'm still open to Skype if you want.

Answer 20 · 2013-08-28T19:23:12.000Z

Looking this issue and I too am wondering how to organize my tests and run them.

I want to allow other developers on the team to simultaneously create tests so one big test file feels wrong there.
Absent a GUI runner. I want to be able to run individual suites of tests without too much work, a command line argument would do. It looks like I need to roll my own not terribly hard it seems but something I have to do.

I'll take a look at using your let all _ = technique for grouping tests.

Answer 21 · 2013-08-28T20:47:40.000Z

I will come up with an example tonight and post it.

Answer 22 · 2013-08-29T05:16:13.000Z

You can get the latest master branch and I have added a new project that shows how I structure my current project. You can also view it here https://github.com/lefthandedgoat/canopy/tree/master/src/sampleprojectstructure

Its pretty simple:

config.fs for configuration type stuff
canopyExtensions.fs for added dsl bits that are specific to my page, overrides of existing canopy functions, remember F# process top down, so can replace a canopy function by simply creating a new function with the same signature and 'open' the file that includes the definition after you open canopy. Last definition wins!
common.fs for selectors/helpers that are common across many pages, like the selector for a modal, or the ability to log out
Pages/pagename.fs a file per page with selectors/helpers/actual contexts/tests
Program.fs all this really does it list out the test suites in each page that I want to run, since its a console app you can easily pass parameters to limit what you run etc. Be creative =)

Hope this helps. There are many possibilties, find one that works well for you.

Answer 23 · 2013-08-29T14:06:10.000Z

This is fantastic and helps quite a bit. I found a semblance of this on my own in the few hours I've spent but your example goes much deeper. ⛵

Answer 24 · 2014-06-12T21:29:43.000Z

Your link above is out of date. Looks like the new place is https://github.com/lefthandedgoat/canopy/tree/master/src/sampleprojectstructure

Thanks for Canopy... it is working great for us!

Has anyone had luck getting the tests to show up in VS2013 test explorer?

Answer 25 · 2014-06-12T22:44:15.000Z

Thanks @whitezel I have updated the link. I haven't personally upgraded to 2013 so I don't haven experience with test explorer. Sorry.

Answer 26 · 2016-01-20T04:29:35.000Z

no activity.