[Discuss] Functional tests for NP client side #52768
Conversation
mshustov
added
discuss
Team:Operations
|
Pinging @elastic/kibana-platform (Team:Platform) |
|
Pinging @elastic/kibana-operations (Team:Operations) |
| expect(hasAccessToInjectedMetadata).to.equal(true); | ||
| expect( | ||
| await browser.execute(() => { | ||
| return window.np.setup.core.injectedMetadata.getKibanaBuildNumber(); |
There was a problem hiding this comment.
won't work on a NP app page, because we don't run LP
| let value; | ||
| window.np.setup.core.uiSettings.get$('ui_settings_plugin').subscribe(v => (value = v)); | ||
| callback(value); | ||
| return Promise.resolve(value); |
There was a problem hiding this comment.
this is here just to mute TS error. I didn't manage to make it work with a promise. @dmlemeshko FYI
There was a problem hiding this comment.
the node-browser communications are, per nature, limited. To my knowledge browser.executeAsync will always be using callbacks
There was a problem hiding this comment.
It looks like there is a wrong type on the our side
https:/elastic/kibana/blob/master/test/functional/services/browser.ts#L474
Because in Selenium types:
* Unlike executing synchronous JavaScript with {@link #executeScript},
* scripts executed with this function must explicitly signal they are
* finished by invoking the provided callback. This callback will always be
* injected into the executed function as the last argument, and thus may be
* referenced with {@code arguments[arguments.length - 1]}. The following
* steps will be taken for resolving this functions return value against the
* first argument to the script's callback function:
from https:/DefinitelyTyped/DefinitelyTyped/blob/master/types/selenium-webdriver/index.d.ts#L2316
|
|
||
| const settingsValueViaObservables = await browser.executeAsync((callback: Function) => { | ||
| let value; | ||
| window.np.setup.core.uiSettings.get$('ui_settings_plugin').subscribe(v => (value = v)); |
There was a problem hiding this comment.
we cannot rxjs in the test file
💚 Build SucceededTo update your PR or re-run it, just comment with: |
|
One alternative we may be able to create: build very basic karma-like functionality into the functional test runner. It could work something like this:
In theory we could write a functional test runner service that encapsulates all of this functionality and exposes it to functional tests. |
|
I want to discuss the high-level architecture before coming to implementation.
That would work as long as we set up all required Kibana & Elasticsearch states in the FTR and run this 'test bundle'. With this, each setup will require a separate 'test bundle'. |
By 'setup' do you mean each FTR suite (each config file) or each test file? I think we could do this in a way that can be done on the fly each time this service gets invoked. |
There was a problem hiding this comment.
Well, this is a big one.
in-browser testing for non-functional behaviour has always been a pain. To be honest I'm not even sure this is something that should be achieved.
I will start by asking the big question: Now that JS testing environnement properly achieves to emulates a browser environment, with current transpilers 'ensuring' compatibility with major browsers, and with the postulate that we still have proper in-browser functionnal test suites for actual functional application behaviours, what are the actual upside to test non-UI behaviour in real browser environment instead of running them in jest with jsdom?
I tried to reuse existing functional tests infrastructure because:
- we run the full Kibana & ES setup.
- we have control over elasticsearch. what is important for us: data preload, license type, roles, etc.
I know the terms differ from a project to another, and from a company to another, but imho there is a subtle difference between integration tests and end-to-end/functional tests. These needs seems to be for the latest, which I'm not sure makes a lot of sense regarding core testing?
Basically I think that anything that could not be human-tested is not an end to end or functional test, but an integration test. And integration tests should probably not be running in browser environment.
We cannot have access to the platform in runtime. It means we cannot test lifecycles properly.
These should be covered in unit/integ testing, not functional imho.
We cannot subscribe to any events in js runtime.
This is a pain. That can be achieved with proper tooling and browser-side injected code (either via 'test' plugin, bootstrap code or browser.executescript, but the boilerplate to have that kind of things working is heavy (and ugly most of the time)
I started all that effort to add some proper tests for the Licensing plugin.
Yes, licensing is exactly the kind of 'not at the end of the pipe' component that is incredibly hard to test in real-time browser environ, as if what the ONLY plugin enabled on the platform, an user/human test could not test anything.
What other alternatives we can consider?
All the limitations you listed will be present in any browser e2e testing framework. However imho, if today we have the opportunity to choose, cypress and cypress tooling is amazing.
| let value; | ||
| window.np.setup.core.uiSettings.get$('ui_settings_plugin').subscribe(v => (value = v)); | ||
| callback(value); | ||
| return Promise.resolve(value); |
There was a problem hiding this comment.
the node-browser communications are, per nature, limited. To my knowledge browser.executeAsync will always be using callbacks
| window.np = { | ||
| setup: npSetup, | ||
| start: npStart, | ||
| }; |
There was a problem hiding this comment.
I'm wondering if this will be sufficient or if we will need to find a way to expose some more internal core APIs for our tests.
An example from functional tests:
Some core API depends on the elasticsearch. From this point of view:
With this classification, I'd say all the tests from the current PR should be integration tests (except an expired license banner). I'd be happy to move them, but there are several problems:
|
|
After writing a few FT, I think the most blocking point is (at least for me)
For #50223 and now #54366 I have to write navigation test between apps. Some behaviours can only be achieved by navigating from a NP app to another NP app. Which mean that in that case, I can't use the legacy I'm forced to inject manually my NP plugin start/setup API to another I think that from a functional test perspective, having a way to do what @restrry has done in We could for exemple have an environment variable (or another way to enable) to have core expose it whole API to a property under the window object to be able to access it from FT when in test mode/env, as the core_provider_plugin currently do. WDYT? |
I didn't go this way because I had seen no other test specific env flags. Introducing one, we add another small difference of the test env from prod env. OTOH I don't see any other less hack-ish ways, because isolation and explicit dependency declaration are the platform principles. |
If we want to avoid the 'test-only' flag, we could decide to actually add this flag as a 'public' setting/flag, that would ask Not sure if preferable though |
Summary
I want to discuss with @elastic/kibana-platform how to write integration tests for the non-UI client logic.
We have karma tests currently but it seems that we are deprecating them without providing an alternative solution.
I tried to reuse existing functional tests infrastructure because:
My pain points with the current setup:
What other alternatives we can consider?
I believe
https:/puppeteer/puppeteer, https://www.cypress.io/
have the same limitations regarding js-runtime. Although Cypress has nice support for assertions
cypress-io/cypress-documentation#1109
Checklist
Use
strikethroughsto remove checklist items you don't feel are applicable to this PR.For maintainers