build: add Travis job to check for dead URL links #27267

richardlau · 2019-04-16T20:57:49Z

In the long run we should add tools to prevent this ?
I wonder can https:/JustinBeckwith/linkinator integrate to travis-ci ? (Suggested by @bnb )

Seems easy enough to put together.

This is a proof of concept of running https:/JustinBeckwith/linkinator
on our built HTML docs to find broken URL links. I don't think we want to be
running this for every PR as when I run it locally it checks 900+ links which could
be seen as a denial of service attack if we run it too often.

Also it seems to be flagging some URL's as broken which I appear to be able to
navigate to in web browser (although there does appear to be redirecting going
on). If anyone wants to look into that or adopt this PR then great as I probably
won't be spending much time on it.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

nodejs-github-bot · 2019-04-16T20:57:51Z

Lite-CI: https://ci.nodejs.org/job/node-test-pull-request-lite-pipeline/3276

addaleax · 2019-04-16T21:19:56Z

.travis.yml

+ - make doc-only
+ - npm install -g linkinator
+ script:
+ - linkinator out/doc/api -r


Maybe just scanning all.html is a good idea? Or does that just not make a difference?

Can we make it only run when the docs are touched? (Like a make target that depends on docs?)

Can we use the same mechanism that add doc label to PR ?

Added a change that greps the PR's patch diff for *.md files in doc/api and only runs the scan if grep returns something.

For local files linkinator works on a directory level so to scan, say, just all.html would involve moving files into a temporary directory. If someone has time maybe they can run linkinator on just all.html and see how many links it scans.

gengjiawen · 2019-04-17T02:38:05Z

Looks like some are false positive: https://travis-ci.com/nodejs/node/jobs/193473667#L688.

richardlau · 2019-04-17T04:59:53Z

Looks like some are false positive: https://travis-ci.com/nodejs/node/jobs/193473667#L688.

Like I said in the OP:

Also it seems to be flagging some URL's as broken which I appear to be able to
navigate to in web browser (although there does appear to be redirecting going
on).

e.g. https://travis-ci.com/nodejs/node/jobs/193473667#L688

   [404] https://chromedevtools.github.io/devtools-protocol/v8/Debugger

-bash-4.2$ curl --head https://chromedevtools.github.io/devtools-protocol/v8/Debugger
HTTP/1.1 404 Not Found
Server: GitHub.com
Content-Type: text/html; charset=utf-8
ETag: "5cb52d6a-1f0"
Access-Control-Allow-Origin: *
X-GitHub-Request-Id: 2612:7F6A:221DC7:2D2A1F:5CB6B0D2
Content-Length: 496
Accept-Ranges: bytes
Date: Wed, 17 Apr 2019 04:51:31 GMT
Via: 1.1 varnish
Age: 0
Connection: keep-alive
X-Served-By: cache-mdw17362-MDW
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1555476691.380921,VS0,VE29
Vary: Accept-Encoding
X-Fastly-Request-ID: 8ced15e319228e1c9c4666e404a8a66c20f7d9d8

-bash-4.2$

-bash-4.2$ curl https://chromedevtools.github.io/devtools-protocol/v8/Debugger
<!DOCTYPE html><meta charset="utf-8"><title>Chrome DevTools Protocol Viewer (redirecting)</title><script>// Copyright (c) 2016 Rafael Pedicini, licensed under the MIT License
var segmentCount=1,l=window.location;l.replace(l.protocol+'//'+l.hostname+(l.port?':'+l.port:'')+l.pathname.split('/').slice(0,1+segmentCount).join('/')+'/?p=/'+l.pathname.slice(1).split('/').slice(segmentCount).join('/').replace(/&/g,'~and~')+(l.search?'&q='+l.search.slice(1).replace(/&/g,'~and~'):'')+l.hash);</script>-bash-4.2$

It seems very weird for a 404 Not Found response page to be redirecting -- There are specific HTTP response codes for redirection.

richardlau · 2019-04-17T05:03:28Z

Looks like some are false positive: https://travis-ci.com/nodejs/node/jobs/193473667#L688.

Like I said in the OP:

Also it seems to be flagging some URL's as broken which I appear to be able to
navigate to in web browser (although there does appear to be redirecting going
on).

e.g. https://travis-ci.com/nodejs/node/jobs/193473667#L688

   [404] https://chromedevtools.github.io/devtools-protocol/v8/Debugger

-bash-4.2$ curl --head https://chromedevtools.github.io/devtools-protocol/v8/Debugger
HTTP/1.1 404 Not Found
Server: GitHub.com
Content-Type: text/html; charset=utf-8
ETag: "5cb52d6a-1f0"
Access-Control-Allow-Origin: *
X-GitHub-Request-Id: 2612:7F6A:221DC7:2D2A1F:5CB6B0D2
Content-Length: 496
Accept-Ranges: bytes
Date: Wed, 17 Apr 2019 04:51:31 GMT
Via: 1.1 varnish
Age: 0
Connection: keep-alive
X-Served-By: cache-mdw17362-MDW
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1555476691.380921,VS0,VE29
Vary: Accept-Encoding
X-Fastly-Request-ID: 8ced15e319228e1c9c4666e404a8a66c20f7d9d8

-bash-4.2$

-bash-4.2$ curl https://chromedevtools.github.io/devtools-protocol/v8/Debugger
<!DOCTYPE html><meta charset="utf-8"><title>Chrome DevTools Protocol Viewer (redirecting)</title><script>// Copyright (c) 2016 Rafael Pedicini, licensed under the MIT License
var segmentCount=1,l=window.location;l.replace(l.protocol+'//'+l.hostname+(l.port?':'+l.port:'')+l.pathname.split('/').slice(0,1+segmentCount).join('/')+'/?p=/'+l.pathname.slice(1).split('/').slice(segmentCount).join('/').replace(/&/g,'~and~')+(l.search?'&q='+l.search.slice(1).replace(/&/g,'~and~'):'')+l.hash);</script>-bash-4.2$

It seems very weird for a 404 Not Found response page to be redirecting -- There are specific HTTP response codes for redirection.

JustinBeckwith/linkinator#24

refack · 2019-04-18T18:33:31Z

.travis.yml

+ - npm install -g linkinator
+ script:
+ - if [ "${TRAVIS_PULL_REQUEST}" != "false" ]; then
+ DOC_FILES=`curl -sL https:/nodejs/node/pull/${TRAVIS_PULL_REQUEST}.patch | grep -o '\bdoc/api/.*\.md\b'`;


Suggested change

DOC_FILES=`curl -sL https:/nodejs/node/pull/${TRAVIS_PULL_REQUEST}.patch | grep -o '\bdoc/api/.*\.md\b'`;

DOC_FILES=`git diff --name-only HEAD...$TRAVIS_BRANCH | grep -o '\bdoc/api/.*\.md\b'`;

Refs: https://stackoverflow.com/questions/41145041/list-files-modified-in-a-pull-request-within-travis

I think we had trouble with commit message linting before with $TRAVIS_BRANCH which is why we ended up downloading the patch files but I forget the details.

refack · 2019-04-18T18:34:15Z

.travis.yml

+ - make doc-only
+ - npm install -g linkinator
+ script:
+ - if [ "${TRAVIS_PULL_REQUEST}" != "false" ]; then


Redundant because of

node/.travis.yml

Line 81 in 3ed6684

if: type = pull_request

refack · 2019-04-18T18:39:57Z

Nice idea, but why not run it in our CI? (I'm thinking it might be a good idea anyway to move out the make run-ci target into a script in ./tools/)?
Also as for the trigger; changes are a good trigger, but even existing links can go stale, so an additional croned trigger might be prudent.

richardlau · 2019-04-18T18:49:49Z

Nice idea, but why not run it in our CI? (I'm thinking it might be a good idea anyway to move out the make run-ci target into a script in ./tools/)?
Also as for the trigger; changes are a good trigger, but even existing links can go stale, so an additional croned trigger might be prudent.

Yeah, it probably makes more sense to run it on our CI. It could be run on the nightlies rather than PR's so that way we won't be hammering remote webservers.

nodejs-github-bot added the build Issues and PRs related to build files or the CI. label Apr 16, 2019

richardlau mentioned this pull request Apr 16, 2019

tools,doc: fix 404 broken links in docs #27168

Closed

2 tasks

addaleax reviewed Apr 16, 2019

View reviewed changes

Trott added the wip Issues and PRs that are still a work in progress. label Apr 16, 2019

richardlau added 2 commits April 18, 2019 12:33

build: add Travis job to check for dead URL links

2bdc627

fixup! build: add Travis job to check for dead URL links

2c15115

richardlau force-pushed the dead-urls-tell-no-tales branch from 3022b98 to 2c15115 Compare April 18, 2019 16:49

richardlau added 2 commits April 18, 2019 13:06

fixup! build: add Travis job to check for dead URL links

f6dbf75

fixup! build: add Travis job to check for dead URL links

3ed6684

refack reviewed Apr 18, 2019

View reviewed changes

richardlau mentioned this pull request May 17, 2019

update N-API docs #27745

Closed

3 tasks

richardlau closed this Aug 23, 2019

richardlau mentioned this pull request Sep 9, 2019

Add linkinator. nodejs/nodejs.org#2565

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: add Travis job to check for dead URL links #27267

build: add Travis job to check for dead URL links #27267

richardlau commented Apr 16, 2019 •

edited

Loading

nodejs-github-bot commented Apr 16, 2019

addaleax Apr 16, 2019

joyeecheung Apr 18, 2019

gengjiawen Apr 18, 2019

richardlau Apr 18, 2019 •

edited

Loading

gengjiawen commented Apr 17, 2019

richardlau commented Apr 17, 2019

richardlau commented Apr 17, 2019

refack Apr 18, 2019

richardlau Apr 18, 2019

refack Apr 18, 2019

refack commented Apr 18, 2019

richardlau commented Apr 18, 2019

	DOC_FILES=`curl -sL https:/nodejs/node/pull/${TRAVIS_PULL_REQUEST}.patch \| grep -o '\bdoc/api/.*\.md\b'`;
	DOC_FILES=`git diff --name-only HEAD...$TRAVIS_BRANCH \| grep -o '\bdoc/api/.*\.md\b'`;

build: add Travis job to check for dead URL links #27267

build: add Travis job to check for dead URL links #27267

Conversation

richardlau commented Apr 16, 2019 • edited Loading

Checklist

nodejs-github-bot commented Apr 16, 2019

addaleax Apr 16, 2019

Choose a reason for hiding this comment

joyeecheung Apr 18, 2019

Choose a reason for hiding this comment

gengjiawen Apr 18, 2019

Choose a reason for hiding this comment

richardlau Apr 18, 2019 • edited Loading

Choose a reason for hiding this comment

gengjiawen commented Apr 17, 2019

richardlau commented Apr 17, 2019

richardlau commented Apr 17, 2019

refack Apr 18, 2019

Choose a reason for hiding this comment

richardlau Apr 18, 2019

Choose a reason for hiding this comment

refack Apr 18, 2019

Choose a reason for hiding this comment

refack commented Apr 18, 2019

richardlau commented Apr 18, 2019

richardlau commented Apr 16, 2019 •

edited

Loading

richardlau Apr 18, 2019 •

edited

Loading