MCI Flooding

[peardox]

@lsmoura

MCI appear to have put a flood stop mechanism in place

I was able to partially circumvent this but adding a setInterval to compareToMagicCardsInfo()

The obvious strategy is to ask MCI if we can mirror purely for the purposes of mtgjson and only available for mtgjson users or pass them some header to confirm its from the user of mtgjson

It appears that they've implemented such limitations within the last 24 hours

You can re-run but only get 10 cards at a time

The best I got was about 70 before it went wonky

I then did a new build on an AWS Ubuntu 14,04 and tried again

10 into MCI - we die (I was running on a much more connected server)

Another solution that occurs is to rate-limit MCI requests - typically this would be one per second (but tests make me doubt this)

You'll see this when MCI gets annoyed with you...

Requesting from web: http://magiccards.info/ori/en/140.html
Error downloading: http://magiccards.info/ori/en/51.html
Error: connect ECONNREFUSED 23.92.17.120:80
at Object.exports._errnoException (util.js:890:11)
at exports._exceptionWithHostPort (util.js:913:20)
at TCPConnectWrap.afterConnect as oncomplete
/root/mtgjson/shared/shared.js:653
throw(err);
^

Error: connect ECONNREFUSED 23.92.17.120:80
at Object.exports._errnoException (util.js:890:11)
at exports._exceptionWithHostPort (util.js:913:20)
at TCPConnectWrap.afterConnect as oncomplete

[peardox]

Addendum

I may be wrong - see below

This repeats - I'll try BFZ

Ripping Set: Magic Origins
Getting card lists...
Magic Origins
Processing 278 multiverseids
Processing variations...
Processing 15 multiverseids
Adding additional fields...
Fixing color identity for double-faced cards...
Adding foreign names to cards...
Adding legalities to cards...
Adding printings to cards...
Doing set corrections...
Comparing cards to MagicCards.info...
Requesting from web: http://magiccards.info/ori/en/142.html
Requesting from web: http://magiccards.info/ori/en/176.html
Requesting from web: http://magiccards.info/ori/en/246.html
Requesting from web: http://magiccards.info/ori/en/143.html
Requesting from web: http://magiccards.info/ori/en/96.html
Requesting from web: http://magiccards.info/ori/en/57.html
Requesting from web: http://magiccards.info/ori/en/144.html
Requesting from web: http://magiccards.info/ori/en/284.html
Requesting from web: http://magiccards.info/ori/en/145.html
Requesting from web: http://magiccards.info/ori/en/146.html
Error downloading: http://magiccards.info/ori/en/142.html
Error: connect ECONNREFUSED 23.92.17.120:80
at Object.exports._errnoException (util.js:890:11)
at exports._exceptionWithHostPort (util.js:913:20)
at TCPConnectWrap.afterConnect as oncomplete
/home/simon/api/mtgjson/shared/shared.js:656
throw(err);
^

Error: connect ECONNREFUSED 23.92.17.120:80
at Object.exports._errnoException (util.js:890:11)
at exports._exceptionWithHostPort (util.js:913:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:

[peardox]

Another addendum but a diffo issue

They seem to be letting us in again but I'm getting this now...

I'll try another few sets

Requesting from web: http://magiccards.info/ori/en/82.html
no MCIArtist! for url /ori/en/203.html (cache: /home/simon/api/mtgjson/cache/6/615e0b6daab2b8f43583e3fd2e50ff5fe2114c31)
ARTIST: Valeron Wardens (398447) has artist but MagicCardsInfo (/ori/en/203.html) does not.
Requesting from web: http://magiccards.info/ori/en/279.html
Requesting from web: http://magiccards.info/ori/en/126.html
Error downloading: http://magiccards.info/ori/en/205.html
undefined
Error downloading: http://magiccards.info/ori/en/167.html
undefined
Requesting from web: http://magiccards.info/ori/en/83.html
Error downloading: http://magiccards.info/ori/en/41.html
undefined
FLAVOR: Vine Snare (398658) has flavor but MagicCardsInfo (/ori/en/205.html) does not.
no MCIArtist! for url /ori/en/205.html (cache: /home/simon/api/mtgjson/cache/3/38bc735d39a281dde17cf42d89fd318ce9ee95d4)
ARTIST: Vine Snare (398658) has artist but MagicCardsInfo (/ori/en/205.html) does not.
FLAVOR: Volcanic Rambler (398636) has flavor but MagicCardsInfo (/ori/en/167.html) does not.
no MCIArtist! for url /ori/en/167.html (cache: /home/simon/api/mtgjson/cache/9/95a43f4a243b485ebe4de129e27ce0b668f65a9c)
ARTIST: Volcanic Rambler (398636) has artist but MagicCardsInfo (/ori/en/167.html) does not.
Error downloading: http://magiccards.info/ori/en/243.html
undefined
Error downloading: http://magiccards.info/ori/en/279.html
undefined
Requesting from web: http://magiccards.info/ori/en/206.html
Requesting from web: http://magiccards.info/ori/en/84.html
FLAVOR: War Oracle (398621) has flavor but MagicCardsInfo (/ori/en/41.html) does not.
no MCIArtist! for url /ori/en/41.html (cache: /home/simon/api/mtgjson/cache/2/27d184d863809d3cf8dc8c61d365865782ce7042)
ARTIST: War Oracle (398621) has artist but MagicCardsInfo (/ori/en/41.html) does not.
Requesting from web: http://magiccards.info/ori/en/207.html
Error downloading: http://magiccards.info/ori/en/82.html
undefined
Requesting from web: http://magiccards.info/ori/en/252.html
FLAVOR: War Horn (398673) has flavor but MagicCardsInfo (/ori/en/243.html) does not.
no MCIArtist! for url /ori/en/243.html (cache: /home/simon/api/mtgjson/cache/a/aa8f82b9e3c28104af7daf2c5b649664df0b5bb3)
ARTIST: War Horn (398673) has artist but MagicCardsInfo (/ori/en/243.html) does not.
FLAVOR: Weave Fate (401458) has flavor but MagicCardsInfo (/ori/en/279.html) does not.
no MCIArtist! for url /ori/en/279.html (cache: /home/simon/api/mtgjson/cache/e/e45efa7db07b149fe5ff7b8034c4057122b0aca7)
ARTIST: Weave Fate (401458) has artist but MagicCardsInfo (/ori/en/279.html) does not.
FLAVOR: Watercourser (398540) has flavor but MagicCardsInfo (/ori/en/82.html) does not.
no MCIArtist! for url /ori/en/82.html (cache: /home/simon/api/mtgjson/cache/b/bf96f1c0a3c231a8885cdb4c18806db68f28f71c)
ARTIST: Watercourser (398540) has artist but MagicCardsInfo (/ori/en/82.html) does not.
Requesting from web: http://magiccards.info/ori/en/208.html
Requesting from web: http://magiccards.info/ori/en/42.html
Requesting from web: http://magiccards.info/ori/en/219.html
Requesting from web: http://magiccards.info/ori/en/209.html
Error downloading: http://magiccards.info/ori/en/206.html
undefined
Error downloading: http://magiccards.info/ori/en/84.html
undefined
Error downloading: http://magiccards.info/ori/en/252.html
undefined
FLAVOR: Wild Instincts (398508) has flavor but MagicCardsInfo (/ori/en/206.html) does not.
no MCIArtist! for url /ori/en/206.html (cache: /home/simon/api/mtgjson/cache/4/459da6495813134113b87f5a1f1e8c9c92178bdc)
ARTIST: Wild Instincts (398508) has artist but MagicCardsInfo (/ori/en/206.html) does not.
FLAVOR: Willbreaker (398502) has flavor but MagicCardsInfo (/ori/en/84.html) does not.
no MCIArtist! for url /ori/en/84.html (cache: /home/simon/api/mtgjson/cache/3/3f73321504ae907092dbdd0a172fc88ec3788103)
ARTIST: Willbreaker (398502) has artist but MagicCardsInfo (/ori/en/84.html) does not.
no MCIArtist! for url /ori/en/252.html (cache: /home/simon/api/mtgjson/cache/2/29a292a223090c980629109717fad865c456e6c4)
ARTIST: Yavimaya Coast (398566) has artist but MagicCardsInfo (/ori/en/252.html) does not.
Error downloading: http://magiccards.info/ori/en/208.html
undefined
FLAVOR: Yeva's Forcemage (398424) has flavor but MagicCardsInfo (/ori/en/208.html) does not.
no MCIArtist! for url /ori/en/208.html (cache: /home/simon/api/mtgjson/cache/b/b3b2f1af9bb60a2c6334d503e03520c96983528c)
ARTIST: Yeva's Forcemage (398424) has artist but MagicCardsInfo (/ori/en/208.html) does not.
Error downloading: http://magiccards.info/ori/en/219.html
undefined
Error downloading: http://magiccards.info/ori/en/209.html
undefined
FLAVOR: Zendikar Incarnate (398661) has flavor but MagicCardsInfo (/ori/en/219.html) does not.
no MCIArtist! for url /ori/en/219.html (cache: /home/simon/api/mtgjson/cache/c/c53c9d3bb5a1154d8ec3dcd318c2d1b05893500f)
ARTIST: Zendikar Incarnate (398661) has artist but MagicCardsInfo (/ori/en/219.html) does not.
FLAVOR: Zendikar's Roil (398518) has flavor but MagicCardsInfo (/ori/en/209.html) does not.
no MCIArtist! for url /ori/en/209.html (cache: /home/simon/api/mtgjson/cache/3/33386ddc4b12337c6aab432f68a8dc97227316d3)
ARTIST: Zendikar's Roil (398518) has artist but MagicCardsInfo (/ori/en/209.html) does not.
Comparing cards to essentialmagic.com...
Requesting from web: http://www.essentialmagic.com/cardsets/Spoiler.asp?ID=107
simon@ubu:~/api/mtgjson/build$ node buildSet.js ORI
Doing sets: ORI

[peardox]

Rats

Blocked again

I did a clone yesterday - I'll try it on that

[lsmoura]

"MCI flooding" is fixed on the latest release. No more downloading stuff in parallel.