Wierd errors after upgrading to 3.0.4

[sarjanen]

Yes too fast, emberjs/ember.js#14730

Don't know if my this issue belongs here, ember, ember-cli or it's just me.

After upgrading to ember/ember-cli 2.9 and going from ember-cli-qunit 2.1.0 to 3.0.4 i'm getting this weird error in almost all my acceptance tests:

Assertion after the final `assert.async` 
Source: 	
    at Assert.ok (http://localhost:7357/assets/test-support.js:4186:10)
    at http://localhost:7357/assets/tests.js:374:14
    at isolate (http://localhost:7357/assets/vendor.js:53599:17)
    at http://localhost:7357/assets/vendor.js:53555:16
    at tryCatch (http://localhost:7357/assets/vendor.js:73026:12)
    at invokeCallback (http://localhost:7357/assets/vendor.js:73038:13)
    at publish (http://localhost:7357/assets/vendor.js:73009:7)
    at http://localhost:7357/assets/vendor.js:52488:16

All my tests passed before the upgrade. I think it has something to to with the click helper but i'm not 100% sure.

[wmadden]

Why was this closed? This is the same error reported in ember-cli/ember-cli#6293 although I believe it's an issue with ember-cli-qunit rather than ember-cli.

[dknutsen]

Yeah, @Padchi can we reopen this? We just started running into this too with our app. I've tried downgrading but that has no effect, think it might be since upgrading to Ember 2.10+. We're not using the click helper in the way described in the issue you linked, so I don't think it has anything to do with that.

[wmadden]

@dknutsen I didn't read the issue too closely but it described a change in qunit's async semantics, which would explain failures / incompatibilities in Ember's async test helpers (e.g. andThen()).

[sarjanen]

Yeah of course @wmadden but isn't this issue related to ember (emberjs/ember.js#14730)?

[wmadden]

I'm no expert @padchi but I assumed it was due to the upgrade of qunit. I upgraded ember-cli-qunit without upgrading ember and experienced this problem.

[sarjanen]

Ah okey, I will open this then again

[carols10cents]

I'm able to reproduce this at will, on travis and locally on this PR running yarn run test in that repo, if it helps. I might have time to try a git bisect of ember-cli-qunit between 2.1.0 and 3.0.4 tomorrow...

[carols10cents]

Aaaaand it turns out i was trying to scope on an ID that didn't exist anymore: rust-lang/crates.io#562 I still might do the bisect, to see if this gave a better error message previously and if so, what made that change...

[trentmwillis]

Hi all, sorry for the delay in responding to this. The issue has to do with ember-qunit's test adapter and the recent QUnit 2.0 changes; both of which changed in [email protected].

Ember, in testing, "starts" and "stops" async portions of tests. Unfortunately, the only way to represent that in QUnit 2.0 is via assert.async() which has the caveat of not being able to continue calling assertions after calling the done() function returned by it.

So the error above is a result of Ember calling all of its asyncEnds before all of your assertions have run.

In general, all of your acceptance tests should continue working so long as your async helpers aren't nested/chained (two patterns that seem to run into the above issue). If you can't restructure your test you can try (1) returning the Promise from your last helper, or (2) wrapping your test in your own assert.async().

I've been thinking about a better solution, but have not come up with one. The current way Ember thinks about async tests (and provides "magic" async helpers) and the way QUnit thinks about async tests are different.

[dknutsen]

@trentmwillis thanks for the info, we use ember-cli-page-object and chain those helpers like so:

test('test filtering open orders', function(assert) {
  orders.visit()
        .status('Open')
        .account('All');
  andThen(function(){
    assert.equal(orders.orders().count, 4, 'Ensure there are 4 orders when filtered on Open status');
  });
});

is that what you mean when you're talking about chained async helpers? If so, how could we restructure our test in a suitable way? And if we were to wrap a test in the assert.async() would it look like this?

assert.async( test(..., function(){
  ...
}));

Sorry if these are noob questions, I've never dug into the specifics of the test framework before.

[trentmwillis]

I'm unsure how ember-cli-page-object works, so I can't comment about that chaining.

For wrapping tests with assert.async, it would mean something like:

test('some test', function(assert) {
  let done = assert.async();
  // do your testing stuff
  andThen(() => done());
});

Or, you can just return the last promise. In your example above for instance:

test('test filtering open orders', function(assert) {
  orders.visit()
        .status('Open')
        .account('All');

  return andThen(function(){
    assert.equal(orders.orders().count, 4, 'Ensure there are 4 orders when filtered on Open status');
  });
});

[dknutsen]

@trentmwillis that definitely helps and will actually show the reason a test is failing (which seems to be what causes that error to show up). Thanks!

[bcardarella]

I am seeing this error in Phantom only but not in Chrome test runs, any reason?

[dknutsen]

@bcardarella that's pretty weird, I'm seeing the exact opposite in our app. Phantom tests run fine but I have intermittent errors in the Chrome-run tests. I think in our case it has something to do with the position the container is being rendered on the screen vs Phantom where that isn't an issue.

Have you tried wrapping your tests in the assert.async per above? In our case that helped identify tests that were failing for legitimate reasons, but the failures were being hidden by the 'assert.async' error. In your case maybe the Phantom rendered tests are failing for some reason related to Phantom but the failure reasons are being obfuscated by the assert.async error?

[bcardarella]

@dknutsen I was able to resolve in Phantom by returning the last andThen's promise. But then I had to polyfill Symbol for Phantom: https://til.hashrocket.com/posts/b4c0e5d1a4-cant-find-variable-symbol

[tomwayson]

@trentmwillis

I'm developing an addon that is used to perform async operations, typically after the application route renders for the first time. I installed this addon to an app with some existing passing tests: one acceptance test and a few unit/integration tests. I then set up the app to use the addon in the application route's afterRender() hook, and I start to see the "Assertion after the final assert.async was resolved" when running tests. That error never shows for the acceptance test itself, instead for a different random integration or unit test each time. Happens in both in Chrome in and Phantom. I never see the error if I skip the acceptance tests.

So I changed the acceptance test from:

test('smoke-test', function(assert) {
  visit('/');

  andThen(function () {
    assert.equal(currentURL(), '/');
  });

  fillIn('form .input-group input', 'water');
  click('form .input-group button');

  andThen(function () {
    assert.equal(currentURL(), '/items?q=water');
    assert.equal(find('table tbody tr').length, 10);
  });
});

to:

test('smoke-test', function(assert) {
  let done = assert.async();
  visit('/');

  andThen(function () {
    assert.equal(currentURL(), '/');
  });

  fillIn('form .input-group input', 'water');
  click('form .input-group button');

  andThen(function () {
    assert.equal(currentURL(), '/items?q=water');
    assert.equal(find('table tbody tr').length, 10);
    done();
  });

Unfortunately I am still seeing the error. I also tried returning the last andThen()s promise instead, and commenting out the first andThen() (wasn't sure if having 2 andThen()s in a single test is what you meant by chaining) but none of that seemed to make any difference.

I even tried using assert.async() and done() in one of the (randomly) failing integration tests, but that didn't make any difference either.

Is there something I'm not doing right in the above?

I should mention that this is intermittent but usually happens on the first test run. So for example if I run ember t several times it might pass once. Or if run ember t -s and either phantom, or chrome, or both will fail the first time, but if I save any file (w/o changes) they pass consistently. So the tests themselves don't fail for any logical errors.

Adding versions:
[email protected]
[email protected]

[trentmwillis]

If you run into this again, please do a fresh npm install. The error has been removed upstream (qunitjs/qunit#1148), so as long as you're using [email protected] or newer you should be good.

[tomwayson]

@trentmwillis thanks for the update!

After a clean npm install w/: npm ls [email protected] and [email protected] I no longer see the "Assertion after the final assert.async errors.

I am seeing other errors like TypeError: null is not an object (evaluating 'a[c]') on random unit/integration tests (following the same pattern I was seeing above). It's unclear if this an issue w/ my own code or the testing stack. I will try and create a reproducible test case and if I can open a new issue.

[miguelcobain]

@trentmwillis I'm using [email protected]

➜  ember-paper git:(v1.1.0-update) ✗ npm ls qunitjs
[email protected] /Users/miguel/workspace/ember-paper
└─┬ [email protected]
  └── [email protected] 

but I still get a lot of Assertion occured after test had finished. errors.
Any idea of why this could be happening?

[trentmwillis]

@miguelcobain that is a different error. The above error is Assertion after the final 'assert.async'.

The error you're seeing should be considered a bug in your tests. As it indicates that an assertion occured outside the test it is associated with. This means the test had finished (and already been reported) when the assertion occurs.

[miguelcobain]

Ah! Sorry for the confusion.
The thing is that these tests were working before. They work when I specify [email protected], for example.

[bjornharrtell]

I think I've finally found the offending testcase in Ember Paper, fixed by adopted-ember-addons/ember-paper@0414824#diff-eb52e3198460432405aa4bacaf342c31. Not sure why this would cause these problems though.