Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Index page URL as ECS field #3857

Merged
merged 3 commits into from
Jun 10, 2020
Merged

Index page URL as ECS field #3857

merged 3 commits into from
Jun 10, 2020

Conversation

jalvz
Copy link
Contributor

@jalvz jalvz commented Jun 8, 2020

Motivation/summary

See:

Closes #3827

Checklist

I have considered changes for:

How to test these changes

Index a RUM event with something in context.page, and see it indexed as ECS fields in url and http.request.referrer.

Note: if agent sends context.request.url, that will be used for the url field, and not the page

@codecov-commenter
Copy link

codecov-commenter commented Jun 8, 2020

Codecov Report

Merging #3857 into master will increase coverage by 0.03%.
The diff coverage is 81.25%.

@@            Coverage Diff             @@
##           master    #3857      +/-   ##
==========================================
+ Coverage   79.73%   79.77%   +0.03%     
==========================================
  Files         135      135              
  Lines        6158     6175      +17     
==========================================
+ Hits         4910     4926      +16     
- Misses       1248     1249       +1     
Impacted Files Coverage Δ
model/context.go 78.99% <66.66%> (-5.72%) ⬇️
model/error.go 98.75% <100.00%> (+0.04%) ⬆️
model/modeldecoder/context.go 94.32% <100.00%> (+0.04%) ⬆️
model/modeldecoder/error.go 85.05% <100.00%> (ø)
model/modeldecoder/transaction.go 51.06% <100.00%> (ø)
model/transaction.go 100.00% <100.00%> (ø)
processor/otel/consumer.go 95.35% <100.00%> (+2.42%) ⬆️

@apmmachine
Copy link
Contributor

apmmachine commented Jun 8, 2020

💔 Build Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #3857 updated]

  • Start Time: 2020-06-09T13:27:32.213+0000

  • Duration: 9 min 15 sec

Steps errors

Expand to view the steps failures

  • Name: Run intake
    • Description: ./script/jenkins/intake.sh

    • Duration: 4 min 46 sec

    • Start Time: 2020-06-09T13:32:57.881+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-06-09T13:34:20.701Z] go: finding code.cloudfoundry.org/rfc5424 v0.0.0-20180905210152-236a6d29298a
[2020-06-09T13:34:20.701Z] go: finding code.cloudfoundry.org/go-diodes v0.0.0-20190809170250-f77fb823c7ee
[2020-06-09T13:34:20.701Z] go: finding github.com/cloudfoundry/sonde-go v0.0.0-20171206171820-b33733203bb4
[2020-06-09T13:34:20.701Z] go: finding github.com/cloudfoundry-community/go-cfclient v0.0.0-20190808214049-35bcce23fc5f
[2020-06-09T13:34:20.701Z] go: finding code.cloudfoundry.org/gofileutils v0.0.0-20170111115228-4d0c80011a0f
[2020-06-09T13:34:20.701Z] go: finding github.com/mailru/easyjson v0.7.1
[2020-06-09T13:34:20.701Z] go: finding github.com/Masterminds/semver v1.4.2
[2020-06-09T13:34:20.701Z] go: finding github.com/urso/ecslog v0.0.1
[2020-06-09T13:34:20.701Z] go: finding github.com/urso/diag v0.0.0-20200210123136-21b3cc8eb797
[2020-06-09T13:34:20.701Z] go: finding github.com/urso/diag-ecs v0.0.0-20200210114345-ab085841dcb9
[2020-06-09T13:34:20.701Z] go: finding github.com/urso/sderr v0.0.0-20200210124243-c2a16f3d43ec
[2020-06-09T13:35:57.261Z] exec: git update-index -q --refresh
[2020-06-09T13:35:57.261Z] exec: git diff-index HEAD -- .
[2020-06-09T13:35:57.261Z] go build -o build/linux/golint golang.org/x/lint/golint
[2020-06-09T13:35:57.261Z] go: downloading golang.org/x/lint v0.0.0-20200302205851-738671d3881b
[2020-06-09T13:35:57.261Z] go: extracting golang.org/x/lint v0.0.0-20200302205851-738671d3881b
[2020-06-09T13:35:57.261Z] go: finding golang.org/x/lint v0.0.0-20200302205851-738671d3881b
[2020-06-09T13:35:57.261Z] go build -o build/linux/reviewdog github.com/reviewdog/reviewdog/cmd/reviewdog
[2020-06-09T13:35:57.261Z] go: downloading github.com/reviewdog/reviewdog v0.9.17
[2020-06-09T13:35:57.261Z] go: extracting github.com/reviewdog/reviewdog v0.9.17
[2020-06-09T13:35:57.261Z] go: downloading github.com/xanzy/go-gitlab v0.22.3
[2020-06-09T13:35:57.261Z] go: downloading github.com/google/go-github/v29 v29.0.2
[2020-06-09T13:35:57.261Z] go: downloading github.com/reviewdog/errorformat v0.0.0-20200109134752-8983be9bc7dd
[2020-06-09T13:35:57.261Z] go: downloading github.com/haya14busa/go-actions-toolkit v0.0.0-20200105081403-ca0307860f01
[2020-06-09T13:35:57.261Z] go: downloading github.com/mattn/go-shellwords v1.0.7
[2020-06-09T13:35:57.261Z] go: downloading github.com/bradleyfalzon/ghinstallation v1.1.0
[2020-06-09T13:35:57.261Z] go: downloading cloud.google.com/go v0.51.0
[2020-06-09T13:35:57.261Z] go: extracting github.com/bradleyfalzon/ghinstallation v1.1.0
[2020-06-09T13:35:57.261Z] go: downloading github.com/google/go-github/v28 v28.1.1
[2020-06-09T13:35:57.261Z] go: downloading github.com/dgrijalva/jwt-go v3.2.1-0.20190620180102-5e25c22bd5d6+incompatible
[2020-06-09T13:35:57.261Z] go: extracting github.com/reviewdog/errorformat v0.0.0-20200109134752-8983be9bc7dd
[2020-06-09T13:35:57.261Z] go: extracting github.com/xanzy/go-gitlab v0.22.3
[2020-06-09T13:35:57.261Z] go: extracting github.com/dgrijalva/jwt-go v3.2.1-0.20190620180102-5e25c22bd5d6+incompatible
[2020-06-09T13:35:57.261Z] go: extracting github.com/mattn/go-shellwords v1.0.7
[2020-06-09T13:35:57.261Z] go: extracting github.com/haya14busa/go-actions-toolkit v0.0.0-20200105081403-ca0307860f01
[2020-06-09T13:35:57.261Z] go: extracting github.com/google/go-github/v29 v29.0.2
[2020-06-09T13:35:57.261Z] go: downloading github.com/google/go-querystring v1.0.0
[2020-06-09T13:35:57.261Z] go: extracting github.com/google/go-github/v28 v28.1.1
[2020-06-09T13:35:57.261Z] go: extracting github.com/google/go-querystring v1.0.0
[2020-06-09T13:35:57.261Z] go: extracting cloud.google.com/go v0.51.0
[2020-06-09T13:35:57.261Z] go: downloading cloud.google.com/go/datastore v1.0.0
[2020-06-09T13:35:57.261Z] go: extracting cloud.google.com/go/datastore v1.0.0
[2020-06-09T13:35:57.261Z] go: downloading github.com/googleapis/gax-go/v2 v2.0.5
[2020-06-09T13:35:57.261Z] go: downloading google.golang.org/api v0.15.0
[2020-06-09T13:35:57.261Z] go: downloading go.opencensus.io v0.22.2
[2020-06-09T13:35:57.261Z] go: extracting github.com/googleapis/gax-go/v2 v2.0.5
[2020-06-09T13:35:57.261Z] go: extracting go.opencensus.io v0.22.2
[2020-06-09T13:35:57.261Z] go: downloading github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7
[2020-06-09T13:35:57.261Z] go: extracting github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7
[2020-06-09T13:35:57.261Z] go: extracting google.golang.org/api v0.15.0
[2020-06-09T13:35:57.261Z] go: finding github.com/reviewdog/reviewdog v0.9.17
[2020-06-09T13:35:57.261Z] go: finding github.com/mattn/go-shellwords v1.0.7
[2020-06-09T13:35:57.261Z] go: finding github.com/reviewdog/errorformat v0.0.0-20200109134752-8983be9bc7dd
[2020-06-09T13:35:57.261Z] go: finding github.com/google/go-github/v29 v29.0.2
[2020-06-09T13:35:57.261Z] go: finding github.com/haya14busa/go-actions-toolkit v0.0.0-20200105081403-ca0307860f01
[2020-06-09T13:35:57.261Z] go: finding github.com/xanzy/go-gitlab v0.22.3
[2020-06-09T13:35:57.261Z] go: finding github.com/google/go-querystring v1.0.0
[2020-06-09T13:35:57.261Z] go: finding github.com/bradleyfalzon/ghinstallation v1.1.0
[2020-06-09T13:35:57.261Z] go: finding cloud.google.com/go v0.51.0
[2020-06-09T13:35:57.261Z] go: finding cloud.google.com/go/datastore v1.0.0
[2020-06-09T13:35:57.261Z] go: finding github.com/dgrijalva/jwt-go v3.2.1-0.20190620180102-5e25c22bd5d6+incompatible
[2020-06-09T13:35:57.261Z] go: finding github.com/googleapis/gax-go/v2 v2.0.5
[2020-06-09T13:35:57.261Z] go: finding google.golang.org/api v0.15.0
[2020-06-09T13:35:57.261Z] go: finding github.com/google/go-github/v28 v28.1.1
[2020-06-09T13:35:57.261Z] go: finding go.opencensus.io v0.22.2
[2020-06-09T13:35:57.261Z] go: finding github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7
[2020-06-09T13:36:44.031Z] /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-3857/src/github.com/elastic/apm-server/model/modeldecoder/context.go:223:5: var pageUrl should be pageURL
[2020-06-09T13:36:44.031Z] Makefile:203: recipe for target 'golint' failed
[2020-06-09T13:36:44.031Z] make: *** [golint] Error 1
[2020-06-09T13:36:44.032Z] + cleanup
[2020-06-09T13:36:44.032Z] + rm -rf /tmp/tmp.kwVh1cRIDU
[2020-06-09T13:36:45.067Z] Stage "Build and Test" skipped due to earlier failure(s)
[2020-06-09T13:36:45.190Z] Stage "linux build" skipped due to earlier failure(s)
[2020-06-09T13:36:45.192Z] Stage "windows build-test" skipped due to earlier failure(s)
[2020-06-09T13:36:45.192Z] Stage "OSX build-test" skipped due to earlier failure(s)
[2020-06-09T13:36:45.193Z] Stage "Unit Test" skipped due to earlier failure(s)
[2020-06-09T13:36:45.194Z] Stage "System and Environment Tests" skipped due to earlier failure(s)
[2020-06-09T13:36:45.195Z] Stage "Benchmarking" skipped due to earlier failure(s)
[2020-06-09T13:36:45.196Z] Stage "Check kibana Obj. Updated" skipped due to earlier failure(s)
[2020-06-09T13:36:45.197Z] Stage "Hey-Apm" skipped due to earlier failure(s)
[2020-06-09T13:36:45.198Z] Stage "Package" skipped due to earlier failure(s)
[2020-06-09T13:36:45.198Z] Stage "APM Integration Tests" skipped due to earlier failure(s)
[2020-06-09T13:36:45.242Z] Stage "Package" skipped due to earlier failure(s)
[2020-06-09T13:36:45.404Z] Failed in branch linux build
[2020-06-09T13:36:45.405Z] Failed in branch windows build-test
[2020-06-09T13:36:45.405Z] Failed in branch OSX build-test
[2020-06-09T13:36:45.406Z] Failed in branch Unit Test
[2020-06-09T13:36:45.407Z] Failed in branch System and Environment Tests
[2020-06-09T13:36:45.407Z] Failed in branch Benchmarking
[2020-06-09T13:36:45.408Z] Failed in branch Check kibana Obj. Updated
[2020-06-09T13:36:45.409Z] Failed in branch Hey-Apm
[2020-06-09T13:36:45.410Z] Failed in branch APM Integration Tests
[2020-06-09T13:36:45.466Z] Stage "Package" skipped due to earlier failure(s)
[2020-06-09T13:36:45.534Z] Failed in branch Package
[2020-06-09T13:36:46.011Z] Running on Jenkins in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-3857
[2020-06-09T13:36:46.323Z] [INFO] getVaultSecret: Getting secrets
[2020-06-09T13:36:46.413Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-09T13:36:47.071Z] + chmod 755 generate-build-data.sh
[2020-06-09T13:36:47.071Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-3857/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-3857/runs/3 FAILURE 554594
[2020-06-09T13:36:47.621Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-3857/runs/3/steps/?limit=10000 -o steps-info.json

Copy link
Member

@axw axw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just did a glance over, looks good so far.

I'm a little bit on the fence as to whether we should also add http.request.referrer based on page.referer, as suggested in #3827 (comment). Is that semantically correct? Might be worth checking with the RUM devs.

model/context.go Outdated Show resolved Hide resolved
model/context.go Outdated Show resolved Hide resolved
@jalvz jalvz marked this pull request as ready for review June 9, 2020 13:29
@jalvz jalvz requested a review from axw June 9, 2020 13:29
@graphaelli graphaelli mentioned this pull request Jun 10, 2020
16 tasks
if fragment := url.Fragment; fragment != "" {
out.Fragment = &fragment
}
host, port, err := net.SplitHostPort(url.Host)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aha, nice, we get this fixed as a bonus :)

Copy link
Member

@axw axw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Just one question about package_tests, otherwise LGTM.

@@ -102,6 +102,7 @@ func transactionRequiredKeys() *tests.Set {
"transaction.type",
"transaction.context.request.method",
"transaction.context.request.url",
//"transaction.context.page.url",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's this about?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...leftover 🙈

@jalvz jalvz merged commit 71c6b57 into elastic:master Jun 10, 2020
jalvz added a commit to jalvz/apm-server that referenced this pull request Jun 10, 2020
* Split Page URL into ECS parts and store it separately

* Store page referer separately as ECS field

* Update changelog
jalvz added a commit that referenced this pull request Jun 11, 2020
* Index page URL and Referer as ECS fields (#3857)

* Update approval files (#3867)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Duplicate page.url as url.*, to make it searchable
4 participants