Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

example: using the x/exp/teatest package #352

Merged
merged 54 commits into from
May 5, 2023
Merged

example: using the x/exp/teatest package #352

merged 54 commits into from
May 5, 2023

Conversation

caarlos0
Copy link
Member

Introducing the teatest package - a very small package with testing helpers for bubbletea apps.

You basically test a model, giving the model itself, a set of interactions (tea.Msg's) and an assertion, which is basically asserting the output against a given golden file.

Its a very raw idea, any feedback welcome.

Signed-off-by: Carlos A Becker <[email protected]>
Signed-off-by: Carlos A Becker <[email protected]>
Signed-off-by: Carlos A Becker <[email protected]>
Signed-off-by: Carlos A Becker <[email protected]>
Signed-off-by: Carlos A Becker <[email protected]>
Signed-off-by: Carlos A Becker <[email protected]>
@caarlos0 caarlos0 added the enhancement New feature or request label Jun 21, 2022
@caarlos0 caarlos0 self-assigned this Jun 21, 2022
Signed-off-by: Carlos A Becker <[email protected]>
Signed-off-by: Carlos A Becker <[email protected]>
Signed-off-by: Carlos A Becker <[email protected]>
@purpleclay
Copy link
Contributor

Is this PR a candidate for merging anytime soon? I have been using bubbletea for a little while now, it is a fantastic library, but it does mean you end up with a program that is largely untested through automation. I have been thinking of writing my own library for testing bubbletea, but noticed this PR.

@caarlos0
Copy link
Member Author

@purpleclay we're trying it out in some places to see how it feels like... if you wanna give it a try and let us know what you think, it would be greatly appreciated as well (and you can help driving how it should look like).

You can play with it by getting this branch instead:

go get github.com/charmbracelet/bubbletea@test

@purpleclay
Copy link
Contributor

@caarlos0 sure I will try it out on my current project. I guess the only question I would have while trying to use the current flavour of teatest is how to generate the golden image files that are used for comparison. Especially if you have a complex TUI. It feels like validating the entire output is the safest option

@caarlos0
Copy link
Member Author

@purpleclay you can run go test ./yourpackage/... -update and it will generate them for you

@purpleclay
Copy link
Contributor

purpleclay commented Jul 18, 2022

Hi @caarlos0 I have attempted to use teatest by just writing a single test case. You can view my test here:

https:/purpleclay/dns53/blob/teatest/internal/tui/dashboard_test.go

So here are my observations (btw I am liking the fact I can test my TUI):

  1. My Update() method is driven by the initial Init() and needs it to return a tea.Msg containing EC2 metadata. This requires me to add an enforced sleep. Is this something most people will have to do? As I noticed it in your simple example that you do this also. Is there not a way for teatest to wait for all Init commands to complete automatically?
  2. The generation of golden files through the -update flag is a nice touch. Would it feel more intuitive to do this through a go:generate?
  3. Do you envisage people testing partial parts of the terminal output within teatest.TestModel(func(out []byte) {})? I imagine I would always use the golden file approach. Attempting to assert against fragments of text would require me to include special characters e.g. �[1;;mPHZ:�[0m AAAAAAAAAAAAAAAAAAAAAAAA [testing] . Partial matching would be useful if these characters could be stripped from the output
  4. It seems you cannot run any of the teatest test cases if you provide the -race flag to go test. Is that by design? This is the current output from my test:
  5. I am thinking of including a ticker in my TUI. I am wondering what impact this would have on the golden file generated. Would we get sporadic failures
data race details
==================
WARNING: DATA RACE
Write at 0x00c0003b7eb0 by goroutine 8:
  bytes.(*Buffer).Write()
      /usr/local/Cellar/go/1.18.3/libexec/src/bytes/buffer.go:169 +0x44
  fmt.Fprintf()
      /usr/local/Cellar/go/1.18.3/libexec/src/fmt/print.go:205 +0xb1
  github.com/charmbracelet/bubbletea.clearLine()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/screen.go:19 +0x85
  github.com/charmbracelet/bubbletea.(*standardRenderer).stop()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/standard_renderer.go:75 +0x31
  github.com/charmbracelet/bubbletea.(*Program).shutdown()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:588 +0xd1
  github.com/charmbracelet/bubbletea.(*Program).StartReturningModel()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:496 +0x1cd1
  github.com/charmbracelet/bubbletea.(*Program).Start()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:549 +0x52
  github.com/charmbracelet/bubbletea/teatest.TestModel.func1()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/teatest/teatest.go:35 +0x53

Previous write at 0x00c0003b7eb0 by goroutine 7:
  bytes.(*Buffer).Write()
      /usr/local/Cellar/go/1.18.3/libexec/src/bytes/buffer.go:169 +0x44
  fmt.Fprintf()
      /usr/local/Cellar/go/1.18.3/libexec/src/fmt/print.go:205 +0xb1
  github.com/charmbracelet/bubbletea.showCursor()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/screen.go:15 +0x4d
  github.com/charmbracelet/bubbletea.Program.restoreTerminalState()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/tty.go:30 +0x25
  github.com/charmbracelet/bubbletea.(*Program).ReleaseTerminal()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:687 +0x164
  github.com/charmbracelet/bubbletea/teatest.TestModel()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/teatest/teatest.go:44 +0x378
  github.com/purpleclay/dns53/internal/tui_test.TestDashboard()
      /Users/purpleclay/dev/github.com/purpleclay/dns53/internal/tui/dashboard_test.go:50 +0x1dc
  github.com/purpleclay/dns53/internal/tui_test.TestDashboard()
      /Users/purpleclay/dev/github.com/purpleclay/dns53/internal/tui/dashboard_test.go:42 +0x19b
  testing.tRunner()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1486 +0x47

Goroutine 8 (running) created at:
  github.com/charmbracelet/bubbletea/teatest.TestModel()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/teatest/teatest.go:34 +0x2fd
  github.com/purpleclay/dns53/internal/tui_test.TestDashboard()
      /Users/purpleclay/dev/github.com/purpleclay/dns53/internal/tui/dashboard_test.go:50 +0x1dc
  github.com/purpleclay/dns53/internal/tui_test.TestDashboard()
      /Users/purpleclay/dev/github.com/purpleclay/dns53/internal/tui/dashboard_test.go:42 +0x19b
  testing.tRunner()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1486 +0x47

Goroutine 7 (running) created at:
  testing.(*T).Run()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1486 +0x724
  testing.runTests.func1()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1839 +0x99
  testing.tRunner()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1439 +0x213
  testing.runTests()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1837 +0x7e4
  testing.(*M).Run()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1719 +0xa71
  main.main()
      _testmain.go:99 +0x3a9
==================
==================
WARNING: DATA RACE
Read at 0x00c0003b7e90 by goroutine 8:
  bytes.(*Buffer).tryGrowByReslice()
      /usr/local/Cellar/go/1.18.3/libexec/src/bytes/buffer.go:107 +0x52
  bytes.(*Buffer).Write()
      /usr/local/Cellar/go/1.18.3/libexec/src/bytes/buffer.go:170 +0x18
  fmt.Fprintf()
      /usr/local/Cellar/go/1.18.3/libexec/src/fmt/print.go:205 +0xb1
  github.com/charmbracelet/bubbletea.clearLine()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/screen.go:19 +0x85
  github.com/charmbracelet/bubbletea.(*standardRenderer).stop()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/standard_renderer.go:75 +0x31
  github.com/charmbracelet/bubbletea.(*Program).shutdown()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:588 +0xd1
  github.com/charmbracelet/bubbletea.(*Program).StartReturningModel()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:496 +0x1cd1
  github.com/charmbracelet/bubbletea.(*Program).Start()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:549 +0x52
  github.com/charmbracelet/bubbletea/teatest.TestModel.func1()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/teatest/teatest.go:35 +0x53

Previous write at 0x00c0003b7e90 by goroutine 7:
  bytes.(*Buffer).tryGrowByReslice()
      /usr/local/Cellar/go/1.18.3/libexec/src/bytes/buffer.go:108 +0xb3
  bytes.(*Buffer).Write()
      /usr/local/Cellar/go/1.18.3/libexec/src/bytes/buffer.go:170 +0x18
  fmt.Fprintf()
      /usr/local/Cellar/go/1.18.3/libexec/src/fmt/print.go:205 +0xb1
  github.com/charmbracelet/bubbletea.showCursor()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/screen.go:15 +0x4d
  github.com/charmbracelet/bubbletea.Program.restoreTerminalState()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/tty.go:30 +0x25
  github.com/charmbracelet/bubbletea.(*Program).ReleaseTerminal()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:687 +0x164
  github.com/charmbracelet/bubbletea/teatest.TestModel()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/teatest/teatest.go:44 +0x378
  github.com/purpleclay/dns53/internal/tui_test.TestDashboard()
      /Users/purpleclay/dev/github.com/purpleclay/dns53/internal/tui/dashboard_test.go:50 +0x1dc
  github.com/purpleclay/dns53/internal/tui_test.TestDashboard()
      /Users/purpleclay/dev/github.com/purpleclay/dns53/internal/tui/dashboard_test.go:42 +0x19b
  testing.tRunner()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1486 +0x47

Goroutine 8 (running) created at:
  github.com/charmbracelet/bubbletea/teatest.TestModel()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/teatest/teatest.go:34 +0x2fd
  github.com/purpleclay/dns53/internal/tui_test.TestDashboard()
      /Users/purpleclay/dev/github.com/purpleclay/dns53/internal/tui/dashboard_test.go:50 +0x1dc
  github.com/purpleclay/dns53/internal/tui_test.TestDashboard()
      /Users/purpleclay/dev/github.com/purpleclay/dns53/internal/tui/dashboard_test.go:42 +0x19b
  testing.tRunner()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1486 +0x47

Goroutine 7 (running) created at:
  testing.(*T).Run()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1486 +0x724
  testing.runTests.func1()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1839 +0x99
  testing.tRunner()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1439 +0x213
  testing.runTests()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1837 +0x7e4
  testing.(*M).Run()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1719 +0xa71
  main.main()
      _testmain.go:99 +0x3a9
==================
==================
WARNING: DATA RACE
Read at 0x00c000156ce8 by goroutine 8:
  runtime.racereadrange()
      <autogenerated>:1 +0x1b
  github.com/charmbracelet/bubbletea.(*Program).StartReturningModel()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:496 +0x1cd1
  github.com/charmbracelet/bubbletea.(*Program).Start()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:549 +0x52
  github.com/charmbracelet/bubbletea/teatest.TestModel.func1()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/teatest/teatest.go:35 +0x53

Previous write at 0x00c000156ce9 by goroutine 7:
  github.com/charmbracelet/bubbletea.(*Program).ReleaseTerminal()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/tea.go:682 +0xc4
  github.com/charmbracelet/bubbletea/teatest.TestModel()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/teatest/teatest.go:44 +0x378
  github.com/purpleclay/dns53/internal/tui_test.TestDashboard()
      /Users/purpleclay/dev/github.com/purpleclay/dns53/internal/tui/dashboard_test.go:50 +0x1dc
  github.com/purpleclay/dns53/internal/tui_test.TestDashboard()
      /Users/purpleclay/dev/github.com/purpleclay/dns53/internal/tui/dashboard_test.go:42 +0x19b
  testing.tRunner()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1486 +0x47

Goroutine 8 (running) created at:
  github.com/charmbracelet/bubbletea/teatest.TestModel()
      /Users/purpleclay/go/pkg/mod/github.com/charmbracelet/[email protected]/teatest/teatest.go:34 +0x2fd
  github.com/purpleclay/dns53/internal/tui_test.TestDashboard()
      /Users/purpleclay/dev/github.com/purpleclay/dns53/internal/tui/dashboard_test.go:50 +0x1dc
  github.com/purpleclay/dns53/internal/tui_test.TestDashboard()
      /Users/purpleclay/dev/github.com/purpleclay/dns53/internal/tui/dashboard_test.go:42 +0x19b
  testing.tRunner()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1486 +0x47

Goroutine 7 (running) created at:
  testing.(*T).Run()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1486 +0x724
  testing.runTests.func1()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1839 +0x99
  testing.tRunner()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1439 +0x213
  testing.runTests()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1837 +0x7e4
  testing.(*M).Run()
      /usr/local/Cellar/go/1.18.3/libexec/src/testing/testing.go:1719 +0xa71
  main.main()
      _testmain.go:99 +0x3a9
==================
--- FAIL: TestDashboard (0.21s)
    testing.go:1312: race detected during execution of test

Signed-off-by: Carlos A Becker <[email protected]>
@caarlos0
Copy link
Member Author

ahh, glad you got to test it @purpleclay!

  1. My Update() method is driven by the initial Init() and needs it to return a tea.Msg containing EC2 metadata. This requires me to add an enforced sleep. Is this something most people will have to do? As I noticed it in your simple example that you do this also. Is there not a way for teatest to wait for all Init commands to complete automatically?

in that example case, the sleep is to ensure some time passed so I can properly test the output... that said, since commands run in another goroutine, it might be needed to either sleep or sync with a chan or something like that...

  1. The generation of golden files through the -update flag is a nice touch. Would it feel more intuitive to do this through a go:generate?

I think you can put a //go:generate go test -update where you need...

  1. Do you envisage people testing partial parts of the terminal output within teatest.TestModel(func(out []byte) {})? I imagine I would always use the golden file approach. Attempting to assert against fragments of text would require me to include special characters e.g. �[1;;mPHZ:�[0m AAAAAAAAAAAAAAAAAAAAAAAA [testing]. Partial matching would be useful if these characters could be stripped from the output

I think some people might, for simpler outputs...

  1. It seems you cannot run any of the teatest test cases if you provide the -race flag to go test. Is that by design? This is the current output from my test:

investigating...

  1. I am thinking of including a ticker in my TUI. I am wondering what impact this would have on the golden file generated. Would we get sporadic failures

depending on your project, yes... unfortunately.. I don't have a good solution for that... but I'm open to hear any ideas you might have...

@caarlos0
Copy link
Member Author

@purpleclay the race should be fixed now!

@purpleclay
Copy link
Contributor

@purpleclay the race should be fixed now!

Awesome. I will give it another try

Copy link
Contributor

@twpayne twpayne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this!

I'm currently writing tests for a few simple models that build on github.com/charmbracelet/bubbles/textinput.Model (code in twpayne/chezmoi#2359).

I would like to be able to test for when tea.Model.Update() returns tea.Quit as the command, so that I know that my model has terminated correctly. This does not seem to be currently possible as tea.Cmd is a function. Go only allows for functions to be compared to nil, and tea.Quit uses an unexported struct value to signal its value. Would it be possible to add something like:

    func IsCmdQuit(cmd tea.Cmd) bool { /* ... *. }

to teatest so cmd.Quits can be tested?

examples/simple/main_test.go Outdated Show resolved Hide resolved
Runes: []rune{rune(c)},
Type: tea.KeyRunes,
})
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's would also be nice to be able to send tea.KeyMsgs of different types, e.g. for tea.KeyCtrlC, etc. There's an initial implementation at https:/twpayne/chezmoi/blob/4f4760f4911ff03fe431ce31bd847a05b213d6a9/pkg/chezmoibubbles/chezmoibubbles_test.go. This converts a string like "a\r" to the equivalent of the user pressing a then Enter.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems to make sense as well, will play with it later today 🙏

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry for the delay, I forgot about this...

nevertheless, for that you can use teatest.Send and send a KeyMsg directly:

	tm.Send(tea.KeyMsg{
		Type: tea.KeyEnter,
	})

@caarlos0
Copy link
Member Author

I think it's possible, yes. Will work on this later 🤘

@purpleclay
Copy link
Contributor

@caarlos0 is there a potential timeframe of when this teatest package will make it into the main branch?

Signed-off-by: Carlos A Becker <[email protected]>
Signed-off-by: Carlos A Becker <[email protected]>
@aschey
Copy link
Contributor

aschey commented Dec 2, 2022

This looks great; thanks so much for taking a look at that. I'll have to give these latest changes a spin!

@KarolosLykos
Copy link

Hello there!

I noticed this pull request and I'm wondering if it's still in development? Can you give us an update or any news on the progress?

Thank you!

@bashbunni bashbunni self-requested a review April 14, 2023 00:04
teatest/teatest.go Outdated Show resolved Hide resolved
@caarlos0
Copy link
Member Author

Okay, our idea is to do the following:

  • create a charmbracelet/x repo to keep our "experimental packages", very much like google does with their /x too
  • move this package there and let people use it
  • eventually, once its stable, move it back in here

thoughts?

caarlos0 added a commit to charmbracelet/x that referenced this pull request Apr 17, 2023
wait for the underlying context to finish.

extract from #352
@caarlos0 caarlos0 mentioned this pull request Apr 17, 2023
caarlos0 added a commit that referenced this pull request Apr 17, 2023
Allows to check if the given command/message will cause the application
to quit.

Extracted from #352
caarlos0 added a commit that referenced this pull request May 4, 2023
* feat: tea.Wait

wait for the underlying context to finish.

extract from #352

* fix: wait til the end of shutdown

Signed-off-by: Carlos Alexandro Becker <[email protected]>

---------

Signed-off-by: Carlos Alexandro Becker <[email protected]>
Signed-off-by: Carlos Alexandro Becker <[email protected]>
Signed-off-by: Carlos Alexandro Becker <[email protected]>
@caarlos0 caarlos0 changed the title feat: teatest package example: using the x/exp/teatest package May 5, 2023
@caarlos0
Copy link
Member Author

caarlos0 commented May 5, 2023

Hey everyone! 

After many months, we're finally releasing the teatest package!


It'll live under github.com/charmbracelet/x/exp/teatest, and will have a tagged release once we release bubbletea.


Please, do try it out in your projects, and open issues if you find any bugs or possible enhancements.

This PR right now contains only an example of teatest usage.

Thank you!

@caarlos0 caarlos0 merged commit 25022e9 into master May 5, 2023
@caarlos0 caarlos0 deleted the test branch May 5, 2023 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants