Routing: conform DHT to common Router interface #9079

ajnavarro · 2022-07-04T09:09:09Z

Description

Delegated routing implementation (#8997) currently only supports reframe routers. Implement DHT routers so they can be added as a new type on delegated routing config list. Actual DHT config initialization and behavior must work as before.

New design document here: https://hackmd.io/@ajnavarro/HJr059wC9

Config documentation: Add a new routing type and explain different possible parameters
Tests: add new tests on delegatred_test.go
~~Deprecate any old configuration param used to create the DHT~~ (previous, simpler configuration will not be deprecated)
Support any DHT kind that is already supported today (including experimental DHT clients)
Use special routers to support the different router behaviors (New Routing helpers: parallel and sequential. libp2p/go-libp2p-routing-helpers#56)

The text was updated successfully, but these errors were encountered:

BigLep · 2022-07-25T22:48:03Z

@ajnavarro : Is this really about "delegated routing", or about having all our routing options (ipfs/go-delegated-routing and libp2p/go-libp2p-kad-dht) conforming to common pattern/interface?

ajnavarro · 2022-07-26T09:19:26Z

It's the latter, having all our routing options conforming to a common pattern.

ajnavarro · 2022-08-01T16:31:04Z

Example of delegated routing configuration with DHT router:

{
  "Type": "reframe",
  "Enabled": true,
  "Parameters": {
    "Endpoint": "https://cid.contact/reframe",
    "Priority": 100
  }
},
{
  "Type": "dht",
  "Enabled": true,
  "Parameters": {
    "Priority": 200,
    "DHTMode": "dhtclient", # It can also be "dht", "dhtserver" and "none" too. "dht" by default.
    "TrackFullNetwork": false, # experimental dht
    "Concurrency": 10 # we can expose internal configuration params to tweak DHTs.
    "Prefix": "/ipfs" # shall we expose used prefix?
  }
}

ajnavarro · 2022-08-02T15:06:03Z

I'm having some problems refactoring how DHT is created on Kubo. The main DHT Router is created at the same time than Host here:

kubo/core/node/libp2p/host.go

Lines 56 to 65 in 7ec0c24

 opts = append(opts, libp2p.Routing(func(h host.Host) (routing.PeerRouting, error) { 

 r, err := params.RoutingOption( 

 ctx, h, 

 params.Repo.Datastore(), 

 params.Validator, 

 bootstrappers..., 

 ) 

 out.Routing = r 

 return r, err 

 }))

It uses RoutingOption and a list of libp2p.Option.

libp2p.Option are obtained from here: https:/ipfs/kubo/blob/master/core/node/libp2p/libp2p.go
RoutingOption is obtained from builder.go:

kubo/core/node/builder.go

Lines 108 to 110 in 7ec0c24

routingOption := fx.Provide(func() libp2p.RoutingOption {

return cfg.Routing

})

I feel that everything is more complicated than it should be, so I'm tempted to refactor all DHT-related code to make it easier to update in the future. Maybe we don't want to spend time on this, so @lidel @guseggert @aschmahmann @Jorropo I would love to have some feedback about the following possible solutions:

Refactor all code related to RoutingOption to make it simpler. The previous DHT configuration will still work as before. New DHT routers created by the new configuration will use global libp2p.Options like ForceReachability and ConnectionManager
Leave the main DHT as it is. Make it possible to add more DHT Routers apart from the main DHT. AFAIK the only part that is making direct use of the DHT is AutoRelayFeeder. We can add a configuration flag that will make that DHT to be used on AutoRelayFeeder if needed.
Other proposed solutions.

WDYT?

ajnavarro · 2022-08-04T09:27:47Z

As we discussed, I will make a comparison between config files before, and after the change (I'll remove the irrelevant parts to have less noise):

Before:

Config file

{
  "Identity": {
    [...]
  },
  "Datastore": {
    [...]
  },
  "Addresses": {
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip6/::/udp/4001/quic"
    ],
    "Announce": [],
    "AppendAnnounce": [],
    "NoAnnounce": [],
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Gateway": "/ip4/127.0.0.1/tcp/8080"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true
    }
  },
  "Routing": {
    "Type": "dht",
    "Routers": {}
  },
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"
  ],
  "Gateway": {
    [...]
  },
  "API": {
    "HTTPHeaders": {}
  },
  "Swarm": {
    "AddrFilters": null,
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {},
    "RelayService": {},
    "Transports": {
      "Network": {},
      "Security": {},
      "Multiplexers": {}
    },
    "ConnMgr": {
      "Type": "basic",
      "LowWater": 600,
      "HighWater": 900,
      "GracePeriod": "20s"
    },
    "ResourceMgr": {}
  },
  "AutoNAT": {},
  "Pubsub": {
    "Router": "",
    "DisableSigning": false
  },
  "Peering": {
    "Peers": null
  },
  "DNS": {
    "Resolvers": {}
  },
  "Migration": {
    [...]
  },
  "Provider": {
    "Strategy": ""
  },
  "Reprovider": {
    [...]
  },
  "Experimental": {
    [...]
    "AcceleratedDHTClient": false
  },
  "Plugins": {
    [...]
  },
  "Pinning": {
    [...]
  },
  "Internal": {}
}

Bootstrap is generic for all DHTs
Routing.Type is where dht is specified.
AcceleratedDHTClient is on Experimental flags.

After:

Config file

{
  "Identity": {
    [...]
  },
  "Datastore": {
    [...]
  },
  "Addresses": {
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip6/::/udp/4001/quic"
    ],
    "Announce": [],
    "AppendAnnounce": [],
    "NoAnnounce": [],
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Gateway": "/ip4/127.0.0.1/tcp/8080"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true
    }
  },
  "Routing": {
    "Routers": {
	 "storetheindex": {
	    "Type": "reframe",
	    "Enabled": true,
	    "Parameters": {
	      "Endpoint": "https://cid.contact/reframe",
	      "Priority": 100
	    }
	  },
	 "ipfs-public-dht" : {
	 	"Type" : "dht"
		"Enabled" : "true"
		"Methods" : ["put-ipns", "get-p2p-providers", ...]
		"Parameters" : {
			"Mode" : "auto"
			"TrackFullNetworkDHT": true,
			"ProtocolID" : "/ipfs/kad/1.0.0"
			"Bootstrappers" : [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"
  ]
				"Public-IP-Network" : "true"
				}
		}
		"ipfs-lan-dht" : {
				"Type" : "dht"
				"Enabled" : "false"
				"Methods" : ["put-ipns", "get-p2p-providers", ...]
				"Parameters" : {
					"Mode" : "server"
					"ProtocolID" : "/ipfs/lan/kad/1.0.0"
					"Bootstrappers" : []
					"Public-IP-Network" : "false"
				}
		}
	}
  },
  "Gateway": {
    [...]
  },
  "API": {
    "HTTPHeaders": {}
  },
  "Swarm": {
    "AddrFilters": null,
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {},
    "RelayService": {},
    "Transports": {
      "Network": {},
      "Security": {},
      "Multiplexers": {}
    },
    "ConnMgr": {
      "Type": "basic",
      "LowWater": 600,
      "HighWater": 900,
      "GracePeriod": "20s"
    },
    "ResourceMgr": {}
  },
  "AutoNAT": {},
  "Pubsub": {
    "Router": "",
    "DisableSigning": false
  },
  "Peering": {
    "Peers": null
  },
  "DNS": {
    "Resolvers": {}
  },
  "Migration": {
    [...]
  },
  "Provider": {
    "Strategy": ""
  },
  "Reprovider": {
    [...]
  },
  "Experimental": {
    [...]
  },
  "Plugins": {
    [...]
  },
  "Pinning": {
    [...]
  },
  "Internal": {}
}

Global Bootstrap config param disappear.
Routing.Type is removed.
AcceleratedDHTClient is replaced by TrackFullNetworkDHT

Steps:

Refactor how DHT is created and how it interacts with other modules to make it possible to create several ones.
Add a new default config that works as before, adding the WAN and LAN DHTs on Routers config.
Add tests for everything (wrong config params validation, new default config === old default config...)
(2 days - getting tests to pass)
Create a migration. I need to familiarize with migrations repo
Test migration
(challenge here is that need to make sure don't screw anything u. Really need to scrutinize for corner cases. Not so bad here because we're doing config file migration. 1 week)
(we have other stuff we want to include in migration code)

BigLep · 2022-08-10T21:06:16Z

@ajnavarro : I'm not going to be able to engage thoroughly here, but wanted to drop a couple of thoughts:

Either here or somewhere else, I want us to be clear about the configuration we expect bifrost/ipfs.io gateway to use since that is the main usecase we're focused on.
Why doesn't "storetheindex" have "Methods"?
What's the full definition of supported "Methods"

BigLep · 2022-08-10T21:20:43Z

@ajnavarro : As I was looking at this and thinking about #9083, I was wondering if we should generalize a bit more here.

This should have more design but throwing this out as a starting point. Feel free to disregard if it's not useful, not moving in the right direction, is overcomplicating, etc.

Basically I'm wondering if our routing configuration should look more like:

"Routing": {
  "Routers": {
    "storetheindex": {...}, # what you already had
    "ipfs-public-dht" : {...}, # what you already had
    "ipfs-lan-dht" : {...}, # what you already had
    "peer-routing" : {
      "Type" : "bitswap-peer", 
      "Enabled" : true,
      "Parameters" : {...}
    },
    "parallel-get-p2p-providers" : {
      "Type" : "parallel", # A special router we create that can encapsulate other Routers
      "Parameters" : {
        "Routers" : [
          {
            "Name" : "storetheindex",
            "TimeoutMs" : 200, # Optional - can rely on the outer timeout as well.  Not sure how useful this is.  
          }, {
            "Name" : "ipfs-public-dht", 
          }
        ],
        "TimeoutMs" : "500",
        "ReturnStrategy" : "BestEffort" 
                           # This could be one of
                           # "First": return the response from the router whose response we get first
                           # "BestEffort": Combine all the results we get by the outer timeout value.
                           # "AllRequired": Only return a result within the timeout if we get a response from all the routers.  Responses are merged.  
      }
    },
    "get-p2p-providers-chain" : {
      "Type" : "parallel",
      "Parameters" : {
        "Routers" : [
          {
            "Name" : "peer-routing",
            "DelayMs" : 0
          }, {
            "Name" : "parallel-get-p2p-providers",
            "DelayMs" : 500 # Wait 500ms before starting this.  If we get a response from peer-routing sooner, we'll have already returned because of the "First" return strategy.
          },
        ],
        "TimeoutMs" : 2000,
        "ReturnStrategy" : "First",
      }
    }

  },
  "MethodToRouterMap" : {
    "put-ipns" : "ipfs-public-dht",
    "get-p2p-providers" : "get-p2p-providers-chain",
    ... # There should be a mapping from every method to a router
  }
}

ajnavarro · 2022-08-11T10:53:57Z

@BigLep so the main idea is to create a special router that can encapsulate other routers defining how and when we call them.
Main problems I can see here:

How do we call root Routers? All in parallel? Shall we create a Routers list and also a RoutingStrategies list? this will allow us to compose Callers using Routers.
Do we apply the same strategy to all Router methods? maybe that won't make sense.
Where should we limit methods that will be called per router? I think we should move that to the Strategy definition, because depending on the strategy, maybe we want to use some methods or others.

BigLep · 2022-08-11T13:18:07Z

@ajnavarro : I'll quickly engage to try and get the idea across. Please don't take this as direction for what we have to go.

My idea here is that we should have a map of method to router to use for that method (MethodToRouterMap). This map should define the router for every method. The router for a method can be a "base" router that actually makes network calls (e.g., "store-the-index", "ipfs-public-dht" in my example config) or it can be a composite router like "get-p2p-providers-chain" (which itself is composed of a "base" router "peer-routing" and a composite router "parallel-get-p2p-providers"). I think a picture would help here a lot.

Do we apply the same strategy to all Router methods? maybe that won't make sense.

The "strategy" used for a method depends on the router that is specified for that method. For a specific method, one can create a new router composing a different "base" router and strategy if needed.

Where should we limit methods that will be called per router?

I think that lives in MethodToRouterMap. If a router isn't listed for a given method, then it won't be invoked for that method. This would get rid of the "Methods" map key you have in #9079 (comment).

BigLep · 2022-08-12T07:07:12Z

@ajnavarro : concerning the done criteria for this issue, it doesn't specify how multiple routers will be combined in terms of execution ordering and result merging. Will this use the "Routers Wrapper" mentioned in #9083 (comment) ?

BigLep · 2022-08-12T17:53:24Z

Will this use the "Routers Wrapper" mentioned in #9083 (comment) ?

Per 2022-08-12 conversation, yes, this work is using https:/libp2p/go-libp2p-routing-helpers

BigLep · 2022-08-12T17:54:51Z

@ajnavarro : per 2022-08-12 conversation, here's the issue about designing comprehensive routing config: #9188 . We're going to pause this current work to do some design to make sure we're aligned on our end state.

New multi-router configuration system based on https://hackmd.io/G1KRDEX5T3qyfoBMkIrBew#Methods - Added a new routing type: "custom" - Added specific struct types for different Routers (instead of map[string]interface{}) - Added `Duration` config type, to make easier time string parsing - Added config documentation. - Use the latest go-delegated-routing library version with GET support. - Added changelog notes for this feature. It: - closes #9157 - closes #9079 - closes #9186

ajnavarro added the kind/feature A new feature label Jul 4, 2022

ajnavarro mentioned this issue Jul 4, 2022

feat(routing): Delegated Routing #8997

Merged

ajnavarro mentioned this issue Jul 25, 2022

delegated-routing support v1: use configurable delegated-routing endpoints for content-routing #8775

Closed

9 tasks

BigLep mentioned this issue Jul 25, 2022

[Tracking issue] http delegated routing used in production #9150

Closed

9 tasks

BigLep changed the title ~~Delegated Routing: Implement DHT~~ Routing: conform DHT to common Router interface Jul 26, 2022

ajnavarro self-assigned this Aug 2, 2022

ajnavarro mentioned this issue Aug 9, 2022

Routing: conform DHT to common Router interface #9176

Closed

This was referenced Aug 12, 2022

[Tracker] Kubo 0.16 migration list ipfs/fs-repo-migrations#159

Closed

Design desired comprehensive routing config #9188

Closed

ajnavarro mentioned this issue Sep 19, 2022

feat: Delegated routing with custom configuration. #9274

Merged

ajnavarro closed this as completed in #9274 Sep 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Routing: conform DHT to common Router interface #9079

Routing: conform DHT to common Router interface #9079

ajnavarro commented Jul 4, 2022 •

edited

Loading

BigLep commented Jul 25, 2022

ajnavarro commented Jul 26, 2022

ajnavarro commented Aug 1, 2022

ajnavarro commented Aug 2, 2022 •

edited

Loading

ajnavarro commented Aug 4, 2022 •

edited by BigLep

Loading

BigLep commented Aug 10, 2022

BigLep commented Aug 10, 2022 •

edited

Loading

ajnavarro commented Aug 11, 2022 •

edited

Loading

BigLep commented Aug 11, 2022 •

edited

Loading

BigLep commented Aug 12, 2022 •

edited

Loading

BigLep commented Aug 12, 2022

BigLep commented Aug 12, 2022

Routing: conform DHT to common Router interface #9079

Routing: conform DHT to common Router interface #9079

Comments

ajnavarro commented Jul 4, 2022 • edited Loading

Description

BigLep commented Jul 25, 2022

ajnavarro commented Jul 26, 2022

ajnavarro commented Aug 1, 2022

ajnavarro commented Aug 2, 2022 • edited Loading

ajnavarro commented Aug 4, 2022 • edited by BigLep Loading

Before:

After:

Steps:

BigLep commented Aug 10, 2022

BigLep commented Aug 10, 2022 • edited Loading

ajnavarro commented Aug 11, 2022 • edited Loading

BigLep commented Aug 11, 2022 • edited Loading

BigLep commented Aug 12, 2022 • edited Loading

BigLep commented Aug 12, 2022

BigLep commented Aug 12, 2022

ajnavarro commented Jul 4, 2022 •

edited

Loading

ajnavarro commented Aug 2, 2022 •

edited

Loading

ajnavarro commented Aug 4, 2022 •

edited by BigLep

Loading

BigLep commented Aug 10, 2022 •

edited

Loading

ajnavarro commented Aug 11, 2022 •

edited

Loading

BigLep commented Aug 11, 2022 •

edited

Loading

BigLep commented Aug 12, 2022 •

edited

Loading