bestpractices: add crash pre-requisites

[gireeshpunathil]

This covers the preparation of crash diagnostics
Few caveats:

• I am not good at markup, so please advise on the structure
• As this will form a baseline for the rest of the docs, a thorough review is requested: both in the format as well as the content.
  thanks!

Refs: #254

[mhdawson]

It's not clear to me if this is to address the problem of gcore not working?

[gireeshpunathil]

@mhdawson - addressed, added a line that explains what we are doing.

[mhdawson]

Pushed a few commits to do minor fixup and some suggested changes as well.

A more structural suggestion is that we should organize it into the following format:

Introduction

What you have for the introduction with additional paragraphs for

• mentioning the key artifacts for exploring crashes (core dumps, diagnostic report?)

Common issues

Cover common issues in generating the key artifacts.

• Disk space
• ulimits

Recommended best practice

This section provides specific recommendations for how to configure your systems in advance to be ready in order to be ready to investigate crashes.

Ensuring enough disk space

your existing content on how to size. Maybe talk about cleanup/management of cores to avoid space problems here instead of later on?

Configuring to ensure core generation

• ulmits (what you have + would be good to have a recommendation on how to check/configure the har limits)
• specific recommendation for core file naming/patterns and how to set them
• recommendations for cleanup/management of cores that are generated (otherwise can result in out of space for application)

Additional information

• Manual Dump generation

Let me know what you think about this structure/organization.

[gireeshpunathil]

@mhdawson - I have addressed all of your review comments. PTAL, thanks!

[mhdawson]

LGTM

[gireeshpunathil]

thanks @mhdawson . Can I get one more review please!

[mhdawson]

LGTM

[hekike]

Additionally, you could call out that the user will need the exact same node binary to inspect your core dump. Or they need to bundle it into the core dump.

[gireeshpunathil]

Additionally, you could call out that the user will need the exact same node binary to inspect your core dump. Or they need to bundle it into the core dump.

@hekike - can I add that statement in my second part? as here we are talking about the production system, and this one is for the dev host where we will be performing crash analysis, it makes sense to talk about that when we mention the debugging basics? PLMK.

[hekike]

@gireeshpunathil makes sense!
Thanks for the PR!

[gireeshpunathil]

landed as 38a577a