Streaming ingest #63

AsafMah · 2021-12-22T08:47:20Z

-Upgrade go version to 1.16 (1.13 is OOL for a while, and we need new features)
-Adding streaming ingest client with the same ability as ingest.
-Deprecating the Stream method inside ingest
-"Modernize" e2e testing -

Replaced "github.com/kylelemons/godebug/pretty" with testify because:
- pretty is subtly wrong - it will show an object and a pointer to the object as equal
- The output of testify is easier to read, for both humans and for IDEs (goland for example can understand the diff)
- pretty is pretty much dead
Making the tests run with Subtests (now you can know which test in the table failed easily)
Later we can go over the rest of the code and modernize other tests

Still needed:

Docs and samples

github-actions · 2021-12-29T07:22:42Z

Unit Test Results

    1 files ±  0   13 suites ±0 11m 26s ⏱️ + 7m 12s
108 tests +35 108 ✔️ +35 0 💤 ±0 0 ❌ ±0

Results for commit b1eeb26. ± Comparison against base commit b89f168.

♻️ This comment has been updated with latest results.

This reverts commit 26ee388.

This reverts commit dc6f8d0.

# Conflicts: # .github/workflows/go.yml

-Added tests

-Infer format for streaming ingest

-Fixed old reader tests -Fixed warnings -Added test files to gitignore

…use cleanup)

kusto/ingest/file_options.go

yogilad · 2022-02-03T12:49:36Z

kusto/ingest/file_options.go

+
+func (o option) Run(p *properties.All, scopes OptionScope) error {
+
+ for scope, scopeName := range scopesMap {


Why are you running on scopeMap?
Should it not be a simple x & y check?
I think the issue could be you put the source scope and protocol scope at the same enum

There's an assumption here that scopes include one Source scope and one Protocol scope.
I'm guessing the loop is for producing a specific error, but this can be resolved if you would pass them as separate elements and check the matching individually

Done
Let me know if that what you had in mind

kusto/ingest/file_options.go

kusto/ingest/ingest.go

kusto/ingest/internal/conn/conn.go

kusto/ingest/status.go

kusto/ingest/streaming_ingest.go

# Conflicts: # go.mod # go.sum # vendor/modules.txt

codecov-commenter · 2022-02-06T14:26:54Z

Codecov Report

Merging #63 (b1eeb26) into master (02c5c0e) will increase coverage by 2.12%.
The diff coverage is 37.69%.

@@            Coverage Diff             @@
##           master      #63      +/-   ##
==========================================
+ Coverage   50.29%   52.42%   +2.12%     
==========================================
  Files          28       30       +2     
  Lines        4096     4273     +177     
==========================================
+ Hits         2060     2240     +180     
+ Misses       1890     1871      -19     
- Partials      146      162      +16

Impacted Files	Coverage Δ
kusto/ingest/internal/filesystem/filesystem.go	`26.80% <0.00%> (-1.12%)`	⬇️
kusto/ingest/result.go	`6.95% <0.00%> (+6.95%)`	⬆️
kusto/ingest/status.go	`17.16% <0.00%> (+17.16%)`	⬆️
kusto/mock.go	`50.00% <0.00%> (-13.77%)`	⬇️
kusto/ingest/file_options.go	`25.89% <25.89%> (ø)`
kusto/ingest/ingest.go	`44.44% <56.00%> (+39.11%)`	⬆️
kusto/ingest/streaming.go	`63.21% <63.21%> (ø)`
kusto/ingest/internal/conn/conn.go	`66.10% <81.25%> (+0.36%)`	⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b89f168...b1eeb26. Read the comment docs.

# Conflicts: # kusto/ingest/ingest.go # kusto/test/etoe/etoe_test.go

kusto/ingest/file_options.go

# Conflicts: # README.md

AsafMah added 6 commits December 22, 2021 10:45

Fixed various warnings

c77f7f1

Fixed call

5b27030

More minor issues/warnings

f855eb1

Added streaming ingest - streamingingestprop

4bd3125

Streaming ingest - new design

abd57e9

merge fixes

8a6333c

AsafMah marked this pull request as draft December 22, 2021 08:47

Add test reporting

51e80f6

AsafMah added 19 commits December 29, 2021 09:23

Fix

6198ffe

Use go get

72a06b7

env

bda8b87

use new env and path

1527199

Fixed bug and get correct path

77f9023

Tee

20e167b

test failure

26ee388

Revert "test failure"

f20414e

This reverts commit 26ee388.

test failure again

dc6f8d0

Revert "test failure again"

890206d

This reverts commit dc6f8d0.

Flip DontCompress to compress by default

62f91e7

Fixed if

af2daf4

Merge branch 'master' into streaming-ingest

0cb0a4f

# Conflicts: # .github/workflows/go.yml

-Replaced bools with bit flags

850498e

-Added tests

Add e2e tests (wip)

a8264e1

-Fixed & modernized e2e tests

41b01e8

-Infer format for streaming ingest

-Added Reader tests

7d9d50b

-Fixed old reader tests -Fixed warnings -Added test files to gitignore

-Upgrade required version to min support version (1.16) (required to …

00ef051

…use cleanup)

go mod vendor

b0f4d4e

AsafMah marked this pull request as ready for review January 4, 2022 09:26

AsafMah requested a review from yogilad January 4, 2022 09:26

AsafMah requested review from element-of-surprise and danield137 January 4, 2022 09:26

element-of-surprise removed their request for review January 5, 2022 17:28

AsafMah added 2 commits January 12, 2022 13:38

Removed unsafe statements and duplicates whenever possible

697eaa6

merge master

3d93fad

yogilad reviewed Feb 3, 2022

View reviewed changes

AsafMah added 6 commits February 6, 2022 10:18

Renamed method for clarity

2b13e03

Cr fixes

727fbb4

Fix tests

8cb170f

Merge branch 'master' into streaming-ingest

1072773

# Conflicts: # go.mod # go.sum # vendor/modules.txt

Merged from master

7fce6b7

Remove vendor files

e086151

AsafMah added 2 commits February 7, 2022 09:53

Merge branch 'master' into streaming-ingest

8693fa1

# Conflicts: # kusto/ingest/ingest.go # kusto/test/etoe/etoe_test.go

update e2e tests

b763d20

yogilad approved these changes Feb 13, 2022

View reviewed changes

kusto/ingest/file_options.go Outdated Show resolved Hide resolved

kusto/ingest/file_options.go Outdated Show resolved Hide resolved

AsafMah added 3 commits February 14, 2022 08:36

Merge branch 'master' into streaming-ingest

af6fd57

# Conflicts: # README.md

PR fixes + e2e tests improvements

7032ae6

Docs + mark previous function as deprecated

b1eeb26

AsafMah merged commit be42011 into master Feb 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Streaming ingest #63

Streaming ingest #63

AsafMah commented Dec 22, 2021 •

edited

Loading

github-actions bot commented Dec 29, 2021 •

edited

Loading

yogilad Feb 3, 2022

yogilad Feb 3, 2022

AsafMah Feb 6, 2022

codecov-commenter commented Feb 6, 2022 •

edited

Loading


		func (o option) Run(p *properties.All, scopes OptionScope) error {

		for scope, scopeName := range scopesMap {

Streaming ingest #63

Streaming ingest #63

Conversation

AsafMah commented Dec 22, 2021 • edited Loading

github-actions bot commented Dec 29, 2021 • edited Loading

Unit Test Results

yogilad Feb 3, 2022

Choose a reason for hiding this comment

yogilad Feb 3, 2022

Choose a reason for hiding this comment

AsafMah Feb 6, 2022

Choose a reason for hiding this comment

codecov-commenter commented Feb 6, 2022 • edited Loading

Codecov Report

AsafMah commented Dec 22, 2021 •

edited

Loading

github-actions bot commented Dec 29, 2021 •

edited

Loading

codecov-commenter commented Feb 6, 2022 •

edited

Loading