Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RAPIDE & IPSL: first proof of concept #1

Draft
wants to merge 41 commits into
base: main
Choose a base branch
from
Draft

RAPIDE & IPSL: first proof of concept #1

wants to merge 41 commits into from

Conversation

Jorropo
Copy link
Collaborator

@Jorropo Jorropo commented Feb 18, 2023

Closes #2 #3 #4

Moves of ipfs#148

@Jorropo Jorropo self-assigned this Feb 18, 2023
@ipfs ipfs deleted a comment from welcome bot Feb 18, 2023
This still requires lots of work.

This include a mostly complete compiler, (minus a few types and literals).
More nodes and scopes will be required.
Will be fine performance wise once #57 is fixed.
This include a working single threaded structure for RAPIDE.

Todo:
- Multithreading
- dontGoThere map for serverDrivenWorker
- Racing
This help avoiding situations where you go race someone else, in a shallow dag, and you get a block next to them in the traversal.
We do not have consensus nor usecase for the language part of IPSL,
to avoid long debates for some not yet usefull pieces of code,
I've removed it and we will handle merging this part seperately (or never merge it).
In 719b5e6 I changed `download.expand` to release the node lock before returning.
This allows other workers to access the just expanded node while we walk up the chain ancestry.
This mean that when we relock the node later to add it to our list of tasks, someone else may have downloaded some of the blocks already and thus they are not in the childrens slice and thus we wont add them to our list.

This also solve the case where you give a Traversal that is not supported by the underlying protocol,
in that case we want to kill the current download and retry deeper, while previously we would hard error on this and kill the worker.
This happen when duplicated blocks are part of the dag.
@Jorropo Jorropo force-pushed the rapide branch 2 times, most recently from b38c1c4 to e93fb4b Compare March 23, 2023 10:47
This fallsback to fastrand in go1.21 and is well fast enough.
@lidel lidel marked this pull request as draft August 7, 2023 13:31
@Jorropo Jorropo assigned Jorropo and unassigned Jorropo Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Rapide CAR over HTTP (aka, download from gateway) protocol implementation
1 participant