NUnit/FsCheck.NUnit does not work with mono #115

paulyoung · 2015-07-18T22:04:45Z

Is FsCheck.NUnit known to work in Xamarin Studio on OS X?

I've structured my tests as in the examples but none of them run and all produce the following message:

Test method has non-void return type, but no result is expected.

kurtschelfthout · 2015-07-19T07:25:43Z

No, not known to work to me at least - and that error message seems to indicate that the FsCheck NUnit addin is not being loaded. Try installing the vanilla NUnit 2.6.x runners and check if it works with that in the first instance.

By the way, this wouldn't surprise me - NUnit is extremely picky with loading addins. I hope their story improves a lot with NUnit 3. The structure is basically one big workaround to force the NUnit Visual Studio addin to actually load the FsCheck addin.

paulyoung · 2015-07-19T09:03:15Z

Try installing the vanilla NUnit 2.6.x runners and check if it works with that in the first instance.

I've already been using NUnit with [<TestFixture>] and [<Test>] attributes. Is that what you mean?

Sounds like I may need to look into running tests from the command line. Should that work?

kurtschelfthout · 2015-07-19T15:44:16Z

Yes, trying cmd line is what I meant.

paulyoung · 2015-07-19T18:52:47Z

Thanks, I'll give that a try.

I hope their story improves a lot with NUnit 3

Is there anything in the works for v3?

The beta release notes say:

We are planning one more beta next month before the final release.

kurtschelfthout · 2015-07-19T20:06:45Z

Yes, something is in the works, see #98.

kurtschelfthout · 2015-07-20T20:55:17Z

Can you try with FsCheck.NUnit 2.0.4 which I just released. See #117, basically I fucked up 2.0.3...

paulyoung · 2015-07-20T22:12:55Z

I'm having the same issue with 2.0.4 so I thought I'd try cloning the repo and running the FsCheck.NUnit tests in the examples.

I never got that far due to the following build errors:

FsCheck/src/FsCheck.NUnit.Addin/FsCheckTestMethod.fs(17,16): error FS0855: No abstract or interface member was found that corresponds to this override
FsCheck/src/FsCheck.NUnit.Addin/FsCheckTestMethod.fs(18,26): error FS0801: This type has no accessible object constructors
FsCheck/src/FsCheck.NUnit.Addin/FsCheckTestMethod.fs(19,9): error FS0039: The namespace or module 'TestExecutionContext' is not defined
FsCheck/src/FsCheck.NUnit.Addin/FsCheckTestMethod.fs(31,14): error FS0039: The field, constructor or member 'setUpMethods' is not defined
FsCheck/src/FsCheck.NUnit.Addin/FsCheckTestMethod.fs(32,15): error FS0039: The field, constructor or member 'setUpMethods' is not defined
FsCheck/src/FsCheck.NUnit.Addin/FsCheckTestMethod.fs(36,18): error FS0039: The field, constructor or member 'tearDownMethods' is not defined
FsCheck/src/FsCheck.NUnit.Addin/FsCheckTestMethod.fs(37,19): error FS0039: The field, constructor or member 'tearDownMethods' is not defined
FsCheck/src/FsCheck.NUnit.Addin/FsCheckTestMethod.fs(46,17): error FS0501: The member or object constructor 'RecordException' takes 2 argument(s) but is here given 3. The required signature is 'TestMethod.RecordException(ex: exn, testResult: TestResult) : unit'.
FsCheck/src/FsCheck.NUnit.Addin/FsCheckTestMethod.fs(100,9): error FS0072: Lookup on object of indeterminate type based on information prior to this program point. A type annotation may be needed prior to this program point to constrain the type of the object. This may allow the lookup to be resolved.
FsCheck/src/FsCheck.NUnit.Addin/FsCheckTestMethod.fs(106,9): error FS0501: The member or object constructor 'RecordException' takes 2 argument(s) but is here given 3. The required signature is 'TestMethod.RecordException(ex: exn, testResult: TestResult) : unit'.

paulyoung · 2015-07-20T22:33:50Z

FWIW, in my project's tests I added this at the end of the file:

[<assembly: RequiredAddin("FsCheck addin")>]
do()

The tests were all still ignored but this time the message was:

Required addin FsCheck addin not available.

kurtschelfthout · 2015-07-21T07:53:33Z

Ok, thanks for trying all that! I'm hopeful we'll get this to work but unfortunately there seem to be a bunch of issues.

Did it run with the command line runner?

To build FsCheck, I think maybe paket/package restore hasn't kicked in correctly. Can you try running ./build.sh from the root folder. If that doesn't work send the output of it.

For using FsCheck.NUnit NuGet in your project, please send:

the content of your project file (fsproj/csproj) - the one you're trying to install FsCheck.NUnit in
show the contents of the packages folder (just top-level should be fine)
contents of packages.config
build output

paulyoung · 2015-07-21T13:58:51Z

Did it run with the command line runner?

I had trouble with that too, but should have it figured out soon: https://twitter.com/dotb52/status/623383545973731328

paulyoung · 2015-07-21T14:16:23Z

To build FsCheck, I think maybe paket/package restore hasn't kicked in correctly. Can you try running ./build.sh from the root folder. If that doesn't work send the output of it.

I had to make the bash script executable first by doing chmod +x build.sh. I think this is probably the only relevant part of the output:

Program.cs(148,34): warning CS0219: The variable `chooseFromList' is assigned but its value is never used
Program.cs(152,17): warning CS0219: The variable `chooseBool' is assigned but its value is never used
Program.cs(154,17): warning CS0219: The variable `chooseBool2' is assigned but its value is never used
Program.cs(170,17): warning CS0219: The variable `gen' is assigned but its value is never used

 4 Warning(s)
 0 Error(s)

After that the projects would build but the NUnit example tests failed in Xamarin Studio with:

Internal error
    RemotingException: Unix transport error.

paulyoung · 2015-07-21T14:17:32Z

I tried:

$ mono packages/NUnit.Runners/tools/nunit-console.exe examples/FsCheck.NUnit.Examples/bin/Debug/FsCheck.NUnit.Examples.dll

but got the following output:

WARNING: The runtime version supported by this application is unavailable.
Using default runtime: v4.0.30319
NUnit-Console version 2.6.4.14350
Copyright (C) 2002-2012 Charlie Poole.
Copyright (C) 2002-2004 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov.
Copyright (C) 2000-2002 Philip Craig.
All Rights Reserved.

Runtime Environment -
   OS Version: Unix 14.4.0.0
  CLR Version: 4.0.30319.17020 ( Mono 4.0 ( 4.0.2 ((detached/c99aa0c Thu Jun 11 18:53:01 EDT 2015) ) )

ProcessModel: Default    DomainUsage: Single
Execution Runtime: mono-4.0
* Assertion at object.c:2248, condition `vt' not met

There was also a stack trace.

paulyoung · 2015-07-21T14:17:40Z

I managed to get something to run from the command line by running:

$ mono /Library/Frameworks/Mono.framework/Versions/Current/lib/mono/4.5/nunit-console.exe examples/FsCheck.NUnit.Examples/bin/Debug/FsCheck.NUnit.Examples.dll

but all I got was this:

NUnit version 2.4.8
Copyright (C) 2002-2007 Charlie Poole.
Copyright (C) 2002-2004 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov.
Copyright (C) 2000-2002 Philip Craig.
All Rights Reserved.

Runtime Environment -
   OS Version: Unix 14.4.0.0
  CLR Version: 4.0.30319.17020 ( 4.0.2 ((detached/c99aa0c Thu Jun 11 18:53:01 EDT 2015) )


Tests run: 0, Failures: 0, Not run: 0, Time: 0.000 seconds

paulyoung · 2015-07-21T14:18:35Z

As for providing an example project, I think if we can get the examples to work that would be a good start.

kurtschelfthout · 2015-07-21T14:55:50Z

See #113, although it's beyond me why that would work. But if it does, and it works for VS also, might as well use it going forward - we could get rid of the install.ps1 in that case.

With the command line runner, you probably have to put FsCheck.NUnit.Addin in bin/addins or some crazy shit like that. http://nunit.org/index.php?p=nunitAddins&r=2.6.4

kurtschelfthout · 2015-07-21T14:56:37Z

The compile warnings are just warnings, nothing to worry about. Have a look at the travis build if you want to compare.

paulyoung · 2015-07-21T15:04:25Z

See #113, although it's beyond me why that would work

I added the NUnitTestAdapter package and no longer receive an internal error, but I'm back to the original issue:

Test method has non-void return type, but no result is expected.

paulyoung · 2015-07-21T16:06:01Z

With the command line runner, you probably have to put FsCheck.NUnit.Addin in bin/addins

I believe this should be unnecessary since NUnit is supposed to load Addins it finds within test assemblies, but I did it anyway and got the same results.

I'm not totally sure it was even being found or loaded.

kurtschelfthout · 2015-07-23T20:57:41Z

Yeah, I see the exact same behavior with mono under windows. No idea what to do about it though. The xunit runner works fine. Also no idea why it says "runtime version not supported", bloody useless error message (at least it could us which version it needs, or something.)

paulyoung · 2015-07-23T21:24:39Z

I just managed to get my (non-FsCheck) tests running from the command line by adding the NUnit.Runners package and then running:

mono --runtime=v4.0 packages/NUnit.Runners.2.6.4/tools/nunit-console.exe MyProject.Tests/bin/Debug/MyProject.Tests.dll -noxml -nodots -labels

I then added FsCheck and wasn't able to build the test project due to errors in FsCheck.Addin.fs. I added the following references to fix being unable to find NUnit.Core.Extensibility and NUnitAddin, respectively:

<Reference Include="nunit.core">
  <HintPath>..\packages\NUnit.Runners.2.6.4\lib\nunit.core.dll</HintPath>
</Reference>
<Reference Include="nunit.core.interfaces">
  <HintPath>..\packages\NUnit.Runners.2.6.4\lib\nunit.core.interfaces.dll</HintPath>
</Reference>

The test suite builds again but unfortunately I'm seeing the same message in the console:

Tests Not Run:
1) NotRunnable : ...
   Test method has non-void return type, but no result is expected

paulyoung · 2015-07-23T21:40:31Z

I tried this approach with the examples project. I ran this:

mono --runtime=v4.0 packages/NUnit.Runners/tools/nunit-console.exe examples/FsCheck.NUnit.Examples/bin/Debug/FsCheck.NUnit.Examples.dll

and got this output:

NUnit-Console version 2.6.4.14350
Copyright (C) 2002-2012 Charlie Poole.
Copyright (C) 2002-2004 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov.
Copyright (C) 2000-2002 Philip Craig.
All Rights Reserved.

Runtime Environment -
   OS Version: Unix 14.4.0.0
  CLR Version: 4.0.30319.17020 ( Mono 4.0 ( 4.0.2 ((detached/c99aa0c Thu Jun 11 18:53:01 EDT 2015) ) )

ProcessModel: Default    DomainUsage: Single
Execution Runtime: mono-4.0
Unhandled Exception:
System.InvalidCastException: Cannot cast from source type to destination type.

Server stack trace:
  at (wrapper xdomain-dispatch) NUnit.Core.RemoteTestRunner:Load (object,byte[]&,byte[]&)

Exception rethrown at [0]:

  at (wrapper xdomain-invoke) NUnit.Core.RemoteTestRunner:Load (NUnit.Core.TestPackage)
  at NUnit.Util.TestDomain.Load (NUnit.Core.TestPackage package) [0x00000] in <filename unknown>:0

kurtschelfthout · 2015-07-24T04:41:33Z

It dawned on me that this is almost certainly the same Mono bug we're waiting for in the xunit 2 thread #86 #97.

This has to do with AppDomain support in mono, though I don't know specifics (the commandlines above all work with .NET, btw).

Anyway, try the following which did something for me:

mono --runtime=v4.0 packages/NUnit.Runners/tools/nunit-console.exe examples/FsCheck.NUnit.Examples/bin/Debug/FsCheck.NUnit.Examples.dll /domain=None

paulyoung · 2015-07-24T09:47:47Z

I get this:

NUnit-Console version 2.6.4.14350
Copyright (C) 2002-2012 Charlie Poole.
Copyright (C) 2002-2004 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov.
Copyright (C) 2000-2002 Philip Craig.
All Rights Reserved.

Runtime Environment -
   OS Version: Unix 14.4.0.0
  CLR Version: 4.0.30319.17020 ( Mono 4.0 ( 4.0.2 ((detached/c99aa0c Thu Jun 11 18:53:01 EDT 2015) ) )

File type not known: /domain=None

paulyoung · 2015-07-24T09:48:17Z

It works if I use -domain=None. I get one test failure, but things definitely work in the example project.

kurtschelfthout · 2015-07-24T09:51:56Z

Yes, one is expected to fail. Same Mono bug then, reportedly fixed in Mono 4.3.

paulyoung · 2015-07-24T10:13:50Z

The problem now is that everything I'm trying to test fails with System.MissingMethodException.

I think I may need to resort to using Xunit for now, but also happy to keep debugging.

kurtschelfthout · 2015-07-24T10:16:22Z

Sure, don't know what else to try really.

paulyoung · 2015-07-24T10:18:44Z

To clarify, those errors appear to be a result of using -domain=None and nothing specific to FsCheck.

kurtschelfthout · 2015-07-29T07:32:06Z

Marked as up-for-grabs since I neither use NUnit or mono (though the latter may change shortly-ish, the former will not) so I won't be actively debugging this. Happy to make changes to support this though.

kurtschelfthout · 2015-08-31T20:16:58Z

Could you try with NUnit 3 and FsCheck.NUnit beta, as explained in #98. I expect it to get rid of a bunch of these gnarly dependency issues.

kurtschelfthout · 2015-10-02T20:02:36Z

Going to close this as we're moving to NUnit 3 anyway, so I don't expect anyone to really look into this further. Please test #98 though,

mausch mentioned this issue Jul 20, 2015

NUnit integration doesnt work #117

Closed

kurtschelfthout changed the title ~~Issue using FsCheck.NUnit in Xamarin Studio~~ NUnit/FsCheck.NUnit does not work with mono Jul 23, 2015

kurtschelfthout added the bug label Jul 23, 2015

This was referenced Jul 24, 2015

Exception System.InvalidCastException thrown when running NUnit and FsCheck tests fsprojects/FAKE#855

Closed

Nunit Domain Fix per #855 fsprojects/FAKE#883

Merged

kurtschelfthout mentioned this issue Jul 29, 2015

FsCheck.Nunit should depend on NUnitTestAdapter #113

Closed

kurtschelfthout added the up-for-grabs label Jul 29, 2015

andersosthus mentioned this issue Aug 3, 2015

Align NUnitDomainModel with NUnit documentation fsprojects/FAKE#897

Merged

kurtschelfthout closed this as completed Oct 2, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NUnit/FsCheck.NUnit does not work with mono #115

NUnit/FsCheck.NUnit does not work with mono #115

paulyoung commented Jul 18, 2015

kurtschelfthout commented Jul 19, 2015

paulyoung commented Jul 19, 2015

kurtschelfthout commented Jul 19, 2015

paulyoung commented Jul 19, 2015

kurtschelfthout commented Jul 19, 2015

kurtschelfthout commented Jul 20, 2015

paulyoung commented Jul 20, 2015

paulyoung commented Jul 20, 2015

kurtschelfthout commented Jul 21, 2015

paulyoung commented Jul 21, 2015

paulyoung commented Jul 21, 2015

paulyoung commented Jul 21, 2015

paulyoung commented Jul 21, 2015

paulyoung commented Jul 21, 2015

kurtschelfthout commented Jul 21, 2015

kurtschelfthout commented Jul 21, 2015

paulyoung commented Jul 21, 2015

paulyoung commented Jul 21, 2015

kurtschelfthout commented Jul 23, 2015

paulyoung commented Jul 23, 2015

paulyoung commented Jul 23, 2015

kurtschelfthout commented Jul 24, 2015

paulyoung commented Jul 24, 2015

paulyoung commented Jul 24, 2015

kurtschelfthout commented Jul 24, 2015

paulyoung commented Jul 24, 2015

kurtschelfthout commented Jul 24, 2015

paulyoung commented Jul 24, 2015

kurtschelfthout commented Jul 29, 2015

kurtschelfthout commented Aug 31, 2015

kurtschelfthout commented Oct 2, 2015

NUnit/FsCheck.NUnit does not work with mono #115

NUnit/FsCheck.NUnit does not work with mono #115

Comments

paulyoung commented Jul 18, 2015

kurtschelfthout commented Jul 19, 2015

paulyoung commented Jul 19, 2015

kurtschelfthout commented Jul 19, 2015

paulyoung commented Jul 19, 2015

kurtschelfthout commented Jul 19, 2015

kurtschelfthout commented Jul 20, 2015

paulyoung commented Jul 20, 2015

paulyoung commented Jul 20, 2015

kurtschelfthout commented Jul 21, 2015

paulyoung commented Jul 21, 2015

paulyoung commented Jul 21, 2015

paulyoung commented Jul 21, 2015

paulyoung commented Jul 21, 2015

paulyoung commented Jul 21, 2015

kurtschelfthout commented Jul 21, 2015

kurtschelfthout commented Jul 21, 2015

paulyoung commented Jul 21, 2015

paulyoung commented Jul 21, 2015

kurtschelfthout commented Jul 23, 2015

paulyoung commented Jul 23, 2015

paulyoung commented Jul 23, 2015

kurtschelfthout commented Jul 24, 2015

paulyoung commented Jul 24, 2015

paulyoung commented Jul 24, 2015

kurtschelfthout commented Jul 24, 2015

paulyoung commented Jul 24, 2015

kurtschelfthout commented Jul 24, 2015

paulyoung commented Jul 24, 2015

kurtschelfthout commented Jul 29, 2015

kurtschelfthout commented Aug 31, 2015

kurtschelfthout commented Oct 2, 2015