Modify coin selection algorithm to support minting and burning #2725

sevanspowell · 2021-06-18T07:36:34Z

Issue Number

ADP-997
ADP-346

Pre-merge checks

Perform soak test of test suite to ensure no flakiness has been introduced.
(✅ 200 test runs with --match "RoundRobin" passed with no failures.)

Overview

This PR modifies the coin selection algorithm to account for minting and burning.

Modifies "performSelection" to account for minted and burnt tokens. On a superficial level, mints provide extra tokens to use, where burns increase the amount of tokens we need to select.
Modifies the "makeChange" algorithm to account for minted and burnt tokens. When tokens are minted they are added to the largest change bundle. When tokens are burnt they are removed from the smallest change bundles until the burn has been satisfied. See the code for more detailed commentary.
Adds property tests that ensure the "addMintChange" and "removeBurnChange" family of functions:
- maintain number of change bundles,
- maintain partial ascending ordering of change bundles
- modify the total value of the change bundle by their respective mint/burn amounts.
Adds a few unit tests to ensure the change generation occurs as we expect it to.
Adds some debugging information to the coin selection tests to make debugging in the presence of minting and burning easier.

lib/core/src/Cardano/Wallet/Api/Types.hs

lib/core/test/unit/Cardano/Wallet/Primitive/CoinSelection/MA/RoundRobinSpec.hs

piotr-iohk · 2021-06-18T08:24:07Z

🤔 Interesting, sending POST on this branch to the ep gives me 405:

$ curl -X POST http://localhost:8090/v2/wallets/1b0aa24994b4181e79116c131510f2abf6cdaa4f/assets \
-d '{"mint_burn":{"monetary_policy_index":"11","asset_name":"484148414841","operations":[{"mint":{"receiving_address":"addr_test1qq223dyrt96yx79rz3yda2vs6xjx28ts2ax39hp8cxq2p0shpgwhzykzha0z6w27gxsps40gurvxjt2vxh26009q5wqqxpj2ey","amount":{"quantity":14,"unit":"assets"}}},{"burn":{"quantity":14,"unit":"assets"}}]},"passphrase":"Secure Passphrase","metadata":{"0":{"string":"cardano"},"1":{"int":14},"2":{"bytes":"2512a00e9653fe49a44a5886202e24d77eeb998f"},"3":{"list":[{"int":14},{"int":42},{"string":"1337"}]},"4":{"map":[{"k":{"string":"key"},"v":{"string":"value"}},{"k":{"int":14},"v":{"int":42}}]}},"time_to_live":{"quantity":10,"unit":"second"}}' -H "Content-Type: application/json" | jq


{
  "code": "method_not_allowed",
  "message": "You've reached a known endpoint but I don't know how to handle the HTTP method specified. Please double-check both the endpoint and the method: one of them is likely to be incorrect (for example: POST instead of PUT, or GET instead of POST...)."
}

sevanspowell · 2021-06-18T09:05:36Z

Interesting, sending POST on this branch to the ep gives me 405:

$ curl -X POST http://localhost:8090/v2/wallets/1b0aa24994b4181e79116c131510f2abf6cdaa4f/assets \
-d '{"mint_burn":{"monetary_policy_index":"11","asset_name":"484148414841","operations":[{"mint":{"receiving_address":"addr_test1qq223dyrt96yx79rz3yda2vs6xjx28ts2ax39hp8cxq2p0shpgwhzykzha0z6w27gxsps40gurvxjt2vxh26009q5wqqxpj2ey","amount":{"quantity":14,"unit":"assets"}}},{"burn":{"quantity":14,"unit":"assets"}}]},"passphrase":"Secure Passphrase","metadata":{"0":{"string":"cardano"},"1":{"int":14},"2":{"bytes":"2512a00e9653fe49a44a5886202e24d77eeb998f"},"3":{"list":[{"int":14},{"int":42},{"string":"1337"}]},"4":{"map":[{"k":{"string":"key"},"v":{"string":"value"}},{"k":{"int":14},"v":{"int":42}}]}},"time_to_live":{"quantity":10,"unit":"second"}}' -H "Content-Type: application/json" | jq


{
  "code": "method_not_allowed",
  "message": "You've reached a known endpoint but I don't know how to handle the HTTP method specified. Please double-check both the endpoint and the method: one of them is likely to be incorrect (for example: POST instead of PUT, or GET instead of POST...)."
}

Ah, interesting @piotr-iohk. My stub definition was malformed. Should be fixed now. Of course by "fixed" I mean the stub will still return:

< HTTP/1.1 500 Internal Server Error
< Transfer-Encoding: chunked
< Date: Fri, 18 Jun 2021 09:04:34 GMT
< Server: Warp/3.3.14
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host localhost left intact
Something went wrong

Because the API endpoint is still stubbed, I've only changed coin selection.

It should correctly verify the form of the request data though!

jonathanknowles

Hi @sevanspowell

Many thanks for creating this PR.

I've not yet finished looking through it, but what I've seen so far looks good.

I've added a few comments and suggestions. Will continue looking tomorrow!

lib/core/src/Cardano/Wallet/Primitive/CoinSelection/MA/RoundRobin.hs

lib/core/src/Cardano/Wallet/Api/Types.hs

lib/core/src/Cardano/Wallet/Primitive/CoinSelection/MA/RoundRobin.hs

lib/core/src/Cardano/Wallet/Api/Server.hs

jonathanknowles

Hi Sam.

Some more comments. I'm about to look at the property tests. Will leave more comments later!

lib/core/src/Cardano/Wallet/Api/Server.hs

lib/core/src/Cardano/Wallet/Primitive/CoinSelection/MA/RoundRobin.hs

jonathanknowles

Hi @sevanspowell . Here's a first review of the property tests.

Will add more comments later!

jonathanknowles · 2021-06-24T07:28:52Z

lib/core/test/unit/Cardano/Wallet/Primitive/CoinSelection/MA/RoundRobinSpec.hs

+ -- "sort it". One quick way to get ascending partial order is to make the
+ -- change maps ourself. Presuming of course
+ -- "makeChangeForNonUserSpecifiedAssets" is correctly implemented, which
+ -- other tests should confirm.


Suggestion: consider using NE.scanl to generate values in ascending partial order.

You're right that an arbitrary list of TokenMap values [m1, m2, m3, ..., mi] is not ordered.

However, if we treat the generated list as a list of differences, then we can transform those differences into a list that is guaranteed to be in ascending partial order:

[ m1 , m1 <> m2 , m1 <> m2 <> m3 , ... , mi <> m2 <> m3 <> ... <> mi ]

Full implementation:

prop_addMintValueToChangeMaps_order :: (AssetId, TokenQuantity) -> NonEmpty TokenMap -> Property prop_addMintValueToChangeMaps_order mint changeMapDiffs = property $ inAscendingPartialOrder $ addMintValueToChangeMaps mint changeMaps where -- A list of change maps already in ascending partial order. changeMaps = NE.scanl (<>) TokenMap.empty changeMapDiffs

Where can I download your smarts? :) Addressed in e77bcc4.

lib/core/test/unit/Cardano/Wallet/Primitive/CoinSelection/MA/RoundRobinSpec.hs

lib/core/src/Cardano/Wallet/Primitive/Types/TokenQuantity.hs

lib/core/test/unit/Cardano/Wallet/Primitive/CoinSelection/MA/RoundRobinSpec.hs

lib/core/src/Cardano/Wallet/Primitive/Types/TokenQuantity.hs

We remove the nested function composition in favour of simple function composition.

This error should currently never be thrown, as we do not currently provide a method for the user to mint or burn tokens through the API. Nevertheless, our test suite requires that we document the error in our API specification.

This function will be used to verify coverage in coin selection properties.

…ction_small`.

…riteria`. In particular, we add coverage for the case where some minted values are not spent or burned, so we can trigger the `OutputsInsufficient` error.

This assertion should have been negated. The error was only revealed once we had deliberately tweaked the generator to generate selection criteria with some minted values that are not spent or burned.

…ta`.

In the `genAssetsToMintAndBurn` subgenerator for `genMakeChangeData`, we deliberately create the possibliity of a non-empty intersection between the sets of assets to mint and assets to burn.

jonathanknowles · 2021-07-08T09:36:33Z

bors try

jonathanknowles

I've given this a fairly exhaustive review, and added some missing coverage to the test suite. I think it's good to go. 👍🏻

iohk-bors · 2021-07-08T10:53:31Z

try

Build succeeded:

jonathanknowles · 2021-07-08T11:39:13Z

bors r+

iohk-bors · 2021-07-08T13:57:14Z

Build succeeded:

2756: Add coverage checks to property tests for `TokenMap.intersection` r=jonathanknowles a=jonathanknowles # Issue Number ADP-997 # Pre-merge checks - [x] Perform soak test of test suite to ensure no flakiness has been introduced. (✅ 1000 test runs with `--match "/Token map properties/Arithmetic/prop_intersection"` passed with no failures.) # Overview PR #2725 added a `TokenMap.intersection` function and property tests. This PR: - [x] adds coverage checks to the `prop_intersection_*` series of properties. - [x] where necessary, adjusts property inputs in order to attain the required level of coverage. Co-authored-by: Jonathan Knowles <[email protected]>

sevanspowell requested review from jonathanknowles, rvl and piotr-iohk June 18, 2021 07:36

sevanspowell self-assigned this Jun 18, 2021

sevanspowell commented Jun 18, 2021

View reviewed changes

lib/core/src/Cardano/Wallet/Api/Types.hs Outdated Show resolved Hide resolved

sevanspowell commented Jun 18, 2021

View reviewed changes

lib/core/test/unit/Cardano/Wallet/Primitive/CoinSelection/MA/RoundRobinSpec.hs Show resolved Hide resolved

sevanspowell force-pushed the sevanspowell/ADP-346/mint-burn-coin-select branch from fd4960f to bfd7afa Compare June 18, 2021 09:06

jonathanknowles reviewed Jun 22, 2021

View reviewed changes

KtorZ reviewed Jun 22, 2021

View reviewed changes

lib/core/src/Cardano/Wallet/Api/Server.hs Outdated Show resolved Hide resolved

jonathanknowles reviewed Jun 24, 2021

View reviewed changes

sevanspowell commented Jun 25, 2021

View reviewed changes

lib/core/test/unit/Cardano/Wallet/Primitive/CoinSelection/MA/RoundRobinSpec.hs Outdated Show resolved Hide resolved

sevanspowell commented Jun 25, 2021

View reviewed changes

lib/core/test/unit/Cardano/Wallet/Primitive/CoinSelection/MA/RoundRobinSpec.hs Outdated Show resolved Hide resolved

jonathanknowles self-requested a review June 25, 2021 09:52

jonathanknowles force-pushed the sevanspowell/ADP-346/mint-burn-coin-select branch from 142c2ee to 0929921 Compare June 29, 2021 07:25

jonathanknowles force-pushed the sevanspowell/ADP-346/mint-burn-api branch from fb3abfc to 7c0636d Compare June 29, 2021 07:27

jonathanknowles force-pushed the sevanspowell/ADP-346/mint-burn-coin-select branch 5 times, most recently from acf5628 to 2295304 Compare June 29, 2021 08:35

jonathanknowles reviewed Jun 29, 2021

View reviewed changes

lib/core/src/Cardano/Wallet/Primitive/Types/TokenQuantity.hs Outdated Show resolved Hide resolved

jonathanknowles force-pushed the sevanspowell/ADP-346/mint-burn-coin-select branch 6 times, most recently from c5a1985 to efdf9a6 Compare July 6, 2021 04:25

jonathanknowles added 10 commits July 8, 2021 03:36

Simplify implementation of changeForNonUserSpecifiedAssets.

4657ea6

Make definition of totalOutputValue consistent with totalInputValue.

35ec8b9

Simplify folds in {addMint,removeBurn}values{To,From}ChangeMaps.

a44880e

We remove the nested function composition in favour of simple function composition.

Simplify definition of function fullBalance.

36e4cce

Fix typo in documentation for TokenQuantity.difference.

902b75c

Add property TokenQuantitySpec.prop_add_difference.

93c8948

Clarify auxilliary function definitions in RoundRobinSpec.

28ba0a7

Use consistent terminology for minted and burned tokens.

18be06b

Add the TokenMap.intersection function.

4b38d41

This function will be used to verify coverage in coin selection properties.

jonathanknowles force-pushed the sevanspowell/ADP-346/mint-burn-coin-select branch from 7e208a0 to 36b2be9 Compare July 8, 2021 05:42

jonathanknowles added 6 commits July 8, 2021 08:36

Add coverage checks for minted and burned assets in `prop_performSele…

b02cb1a

…ction_small`.

Enhance the generators for minted and burned assets in `genSelectionC…

5b58f1b

…riteria`. In particular, we add coverage for the case where some minted values are not spent or burned, so we can trigger the `OutputsInsufficient` error.

Fix erroneous assertion in onOutputsInsufficient.

4d22d58

This assertion should have been negated. The error was only revealed once we had deliberately tweaked the generator to generate selection criteria with some minted values that are not spent or burned.

Extract out function genAssetsToMintAndBurn within `genMakeChangeDa…

8eb87a8

…ta`.

Add coverage checks for minting and burning to prop_makeChange.

96b6a75

Add missing coverage for prop_makeChange.

378d35f

In the `genAssetsToMintAndBurn` subgenerator for `genMakeChangeData`, we deliberately create the possibliity of a non-empty intersection between the sets of assets to mint and assets to burn.

jonathanknowles force-pushed the sevanspowell/ADP-346/mint-burn-coin-select branch from 36b2be9 to 378d35f Compare July 8, 2021 08:38

iohk-bors bot added a commit that referenced this pull request Jul 8, 2021

Try #2725:

9d11db6

jonathanknowles self-requested a review July 8, 2021 09:37

jonathanknowles approved these changes Jul 8, 2021

View reviewed changes

iohk-bors bot merged commit 9ef7a6a into master Jul 8, 2021

iohk-bors bot deleted the sevanspowell/ADP-346/mint-burn-coin-select branch July 8, 2021 13:57

WilliamKingNoel-Bot pushed a commit that referenced this pull request Jul 8, 2021

docs: Merge #2725

c00507b

jonathanknowles mentioned this pull request Jul 9, 2021

Add coverage checks to property tests for TokenMap.intersection #2756

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modify coin selection algorithm to support minting and burning #2725

Modify coin selection algorithm to support minting and burning #2725

sevanspowell commented Jun 18, 2021 •

edited by jonathanknowles

Loading

piotr-iohk commented Jun 18, 2021

sevanspowell commented Jun 18, 2021 •

edited

Loading

jonathanknowles left a comment

jonathanknowles left a comment

jonathanknowles left a comment

jonathanknowles Jun 24, 2021 •

edited

Loading

sevanspowell Jun 25, 2021

jonathanknowles commented Jul 8, 2021

jonathanknowles left a comment

iohk-bors bot commented Jul 8, 2021

jonathanknowles commented Jul 8, 2021

iohk-bors bot commented Jul 8, 2021

Modify coin selection algorithm to support minting and burning #2725

Modify coin selection algorithm to support minting and burning #2725

Conversation

sevanspowell commented Jun 18, 2021 • edited by jonathanknowles Loading

Issue Number

Pre-merge checks

Overview

piotr-iohk commented Jun 18, 2021

sevanspowell commented Jun 18, 2021 • edited Loading

jonathanknowles left a comment

Choose a reason for hiding this comment

jonathanknowles left a comment

Choose a reason for hiding this comment

jonathanknowles left a comment

Choose a reason for hiding this comment

jonathanknowles Jun 24, 2021 • edited Loading

Choose a reason for hiding this comment

sevanspowell Jun 25, 2021

Choose a reason for hiding this comment

jonathanknowles commented Jul 8, 2021

jonathanknowles left a comment

Choose a reason for hiding this comment

iohk-bors bot commented Jul 8, 2021

try

jonathanknowles commented Jul 8, 2021

iohk-bors bot commented Jul 8, 2021

sevanspowell commented Jun 18, 2021 •

edited by jonathanknowles

Loading

sevanspowell commented Jun 18, 2021 •

edited

Loading

jonathanknowles Jun 24, 2021 •

edited

Loading