Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Reporting] Sanitize 409 error log message #42495

Merged
merged 4 commits into from
Aug 3, 2019

Conversation

tsullivan
Copy link
Member

Summary

Closes #42417

Checklist

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

Release note: Added truncation to server log messages from Reporting to make 413 Error Status responses readable.

Example

server    log   [15:37:58.106] [debug][esqueue][reporting][worker] jyt907mh0eyw9d0062fkqaid - Completed job jyt9d2zl0eyw9d00627kl28x
server    log   [15:37:58.329] [esqueue][reporting][warn][worker] jyt907mh0eyw9d0062fkqaid - Failure saving job output jyt9d2zl0eyw9d00627kl28x: Request Entity Too Large :: {"path":"/.reporting-2019.07.28/_update/jyt9d2zl0eyw9d00627kl28x","query":{"if_seq_no":455,"if_primary_term":7},"body":"{\"doc\":{\"status\":\"completed\",\"completed_at\":\"2019-08-01T22:37:58.107Z\",\"output\":{\"content\":\"JVBERi0xLjMKJf////8KMTkgMCBvYmoKPDwKL1ByZWRpY3RvciAxNQovQ29sb3JzIDEKL0JpdHNQZXJDb21wb25lbnQgOAovQ29sdW1ucyA1NzcKPj4KZW5kb2JqCjIwIDAgb2JqCjw8Ci9MZW5ndGggMTQ5Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlCj4+CnN0cmVhbQp4nCWObQ6DIBBEuaz/TA9QiwYDCKjgB0k9bl9Zstk8ZmYXlFJ6HJ33275T1toQ46vvUbquE3HdNusckNaV8BLCnjMxDpYxZpwm3xgR18xzOQ7ALwtTMSVZwhTA5pASq2LrciUMkElUs3Ipkkf8aA0DiKKTEQug89x/fyki5pzfwyAuXyJ/nOdd63XfV63f5/kBWBmCdgplbmRzdHJlYW0KZW5kb2JqCjI0IDAgb2JqCjw8Ci9UeXBlIC9FeHRHU3RhdGUKL0NBIDEKPj4KZW5kb2JqCjI1IDAgb2JqCjw8Ci9UeXBlIC9FeHRHU3RhdGUKL2NhIDEKL0NBIDEKPj4KZW5kb2JqCjIzIDAgb2JqCjw8Ci9UeXBlIC9QYWdlCi9QYXJlbnQgMSAwIFIKL01lZGlhQm94IFswIDAgMTM2MiA5MDFdCi9Db250ZW50cyAyMSAwIFIKL1Jlc291cm
server    log   [15:37:58.345] [esqueue][reporting][warn][worker] The first 1000 characters of the error message were logged. The entire error message length: 18881581 characters.
server    log   [15:37:58.346] [esqueue][reporting][warn][worker] jyt907mh0eyw9d0062fkqaid - Failing job jyt9d2zl0eyw9d00627kl28x

@tsullivan tsullivan added release_note:enhancement review (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead v8.0.0 v7.4.0 labels Aug 1, 2019
@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticdog
Copy link
Contributor

retest

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@@ -29,7 +29,12 @@ function getLogger(opts, id, logLevel) {
const tags = ['worker', logLevel];

if (err) {
logger(`${message}: ${err.stack ? err.stack : err }`, tags);
const errString = `${message}: ${err.stack ? err.stack : err}`;
logger(errString.substring(0, 1000), tags);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

going to change this so we also show the last 1000 characters of a very long string

@tsullivan
Copy link
Member Author

tsullivan commented Aug 2, 2019

New example: message includes the tailing 1000 characters, which show a stack trace:

server    log   [16:15:22.861] [debug][esqueue][reporting][worker] jyuq26oq1rr889fb5f3d19n3 - Completed job jyuq50zt1rr889fb5fd45238
server    log   [16:15:23.070] [esqueue][reporting][warn][worker] jyuq26oq1rr889fb5f3d19n3 - Failure saving job output jyuq50zt1rr889fb5fd45238: Request Entity Too Large :: {"path":"/.reporting-2019.07.28/_update/jyuq50zt1rr889fb5fd45238","query":{"if_seq_no":525,"if_primary_term":8},"body":"{\"doc\":{\"status\":\"completed\",\"completed_at\":\"2019-08-02T23:15:22.861Z\",\"output\":{\"content\":\"JVBERi0xLjMKJf////8KMTkgMCBvYmoKPDwKL1ByZWRpY3RvciAxNQovQ29sb3JzIDEKL0JpdHNQZXJDb21wb25lbnQgOAovQ29sdW1ucyA1NzcKPj4KZW5kb2JqCjIwIDAgb2JqCjw8Ci9MZW5ndGggMTQ5Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlCj4+CnN0cmVhbQp4nCWObQ6DIBBEuaz/TA9QiwYDCKjgB0k9bl9Zstk8ZmYXlFJ6HJ33275T1toQ46vvUbquE3HdNusckNaV8BLCnjMxDpYxZpwm3xgR18xzOQ7ALwtTMSVZwhTA5pASq2LrciUMkElUs3Ipkkf8aA0DiKKTEQug89x/fyki5pzfwyAuXyJ/nOdd63XfV63f5/kBWBmCdgplbmRzdHJlYW0KZW5kb2JqCjI0IDAgb2JqCjw8Ci9UeXBlIC9FeHRHU3RhdGUKL0NBIDEKPj4KZW5kb2JqCjI1IDAgb2JqCjw8Ci9UeXBlIC9FeHRHU3RhdGUKL2NhIDEKL0NBIDEKPj4KZW5kb2JqCjIzIDAgb2JqCjw8Ci9UeXBlIC9QYWdlCi9QYXJlbnQgMSAwIFIKL01lZGlhQm94IFswIDAgMTM2MiA5MDFdCi9Db250ZW50cyAyMSAwIFIKL1Jlc291cm...hMjAyZTQxNWFkYTBjYTc0ZmQ4Pl0KPj4Kc3RhcnR4cmVmCjEzNzM5NzY1CiUlRU9GCg==\",\"content_type\":\"application/pdf\",\"size\":13741624}}}","statusCode":413,"response":"<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.17.2</center>\r\n</body>\r\n</html>\r\n"}
    at respond (/Users/tsullivan/elastic/kibana/node_modules/elasticsearch/src/lib/transport.js:315:15)
    at checkRespForFailure (/Users/tsullivan/elastic/kibana/node_modules/elasticsearch/src/lib/transport.js:274:7)
    at HttpConnector.<anonymous> (/Users/tsullivan/elastic/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)
    at IncomingMessage.wrapper (/Users/tsullivan/elastic/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
    at IncomingMessage.emit (events.js:194:15)
    at endReadableNT (_stream_readable.js:1103:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
server    log   [16:15:23.071] [esqueue][reporting][warn][worker] A partial version of the entire error message was logged. The entire error message length is: 18323433 characters.
server    log   [16:15:23.074] [esqueue][reporting][warn][worker] jyuq26oq1rr889fb5f3d19n3 - Failing job jyuq50zt1rr889fb5fd45238

Copy link
Contributor

@joelgriffith joelgriffith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Code checked only

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@tsullivan tsullivan merged commit c9e2a22 into elastic:master Aug 3, 2019
@tsullivan tsullivan deleted the reporting/error-sanitize-409 branch August 3, 2019 00:39
tsullivan added a commit to tsullivan/kibana that referenced this pull request Aug 3, 2019
* [Reporting] Sanitize 409 error log message

* [Reporting] Sanitize 409 error log message

* feedback changes
jloleysens added a commit to jloleysens/kibana that referenced this pull request Aug 5, 2019
…-or-edit-existing-rollup-job

* 'master' of github.com:elastic/kibana: (67 commits)
  [TSVB] Shim new platform (elastic#39169)
  [Metric Vis] Shim new platform (elastic#42240)
  [Tag Cloud] Shim new platform (elastic#42348)
  Disable flaky request lib tests. Add es_ui_shared plugin to CODEOWNERS.
  Add disk space percentage to node listing (elastic#42145)
  [SIEM] Add chart interactions - update date picker after brush selection on charts (elastic#42440)
  Document HTTP service (elastic#42331)
  [Reporting] Sanitize 409 error log message (elastic#42495)
  [docs][skip ci] Maps read only access (elastic#35561)
  [x-pack/ftr] refactor types to be more accurate/consistent wit… (elastic#42407)
  [DOCS] Updates images and content in Dashboard docs (elastic#42500)
  Allow sorting on multiple columns in Discover (elastic#41918)
  [Infra UI][Logs UI] Fix autocomplete to use proper derived index pattern (elastic#42287)
  [ftr/cheerio] improve cheerio types to include test subject me… (elastic#42534)
  Upgraded EUI 13.0.0 -> 13.1.1 (elastic#42298)
  Increase max-old-space-size for builds (elastic#42218)
  [Infra UI] Add cloud metrics and cloud/host info to metadata endpoint (elastic#41836)
  [Logs UI][a11y] Announce name of column on remove column button (elastic#41695)
  Inspector 👉 New Platform (elastic#42164)
  Make alerting properly space aware (elastic#42081)
  ...
jloleysens added a commit to jloleysens/kibana that referenced this pull request Aug 5, 2019
…s_autocomplete

* 'master' of github.com:elastic/kibana: (189 commits)
  [TSVB] Shim new platform (elastic#39169)
  [Metric Vis] Shim new platform (elastic#42240)
  [Tag Cloud] Shim new platform (elastic#42348)
  Disable flaky request lib tests. Add es_ui_shared plugin to CODEOWNERS.
  Add disk space percentage to node listing (elastic#42145)
  [SIEM] Add chart interactions - update date picker after brush selection on charts (elastic#42440)
  Document HTTP service (elastic#42331)
  [Reporting] Sanitize 409 error log message (elastic#42495)
  [docs][skip ci] Maps read only access (elastic#35561)
  [x-pack/ftr] refactor types to be more accurate/consistent wit… (elastic#42407)
  [DOCS] Updates images and content in Dashboard docs (elastic#42500)
  Allow sorting on multiple columns in Discover (elastic#41918)
  [Infra UI][Logs UI] Fix autocomplete to use proper derived index pattern (elastic#42287)
  [ftr/cheerio] improve cheerio types to include test subject me… (elastic#42534)
  Upgraded EUI 13.0.0 -> 13.1.1 (elastic#42298)
  Increase max-old-space-size for builds (elastic#42218)
  [Infra UI] Add cloud metrics and cloud/host info to metadata endpoint (elastic#41836)
  [Logs UI][a11y] Announce name of column on remove column button (elastic#41695)
  Inspector 👉 New Platform (elastic#42164)
  Make alerting properly space aware (elastic#42081)
  ...
tsullivan added a commit that referenced this pull request Aug 5, 2019
* [Reporting] Sanitize 409 error log message

* [Reporting] Sanitize 409 error log message

* feedback changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
(Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead release_note:enhancement review v7.4.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Reporting] "Request Entity Too Large" message should not log the entity
4 participants