[🚀 Feature]: Make external URI configurable

[utamas]

Feature and motivation

Feature

Add --external-uri configuration parameter to server section. This will allow nodes to provide a URI to hub (or distributor) which is route-able when SE node is running behind some sort proxy.

Motivation

There are cases when nodes cannot report self URI that is route-able from hub during node registration process in which case the node get's stuck in Sending registration event... loop, while the hub logs will show connection timeouts or unknown host exceptions.

I believe this can be easily fixed in org.openqa.selenium.grid.server.BaseServerOptions#getExternalUri by first reading this newly added external-uri parameter. When empty, implementation can fall back to the current one otherwise it could just use the provided value.

Our concrete use case:

We run ~5000 selenium tests. For performance reasons we deploy ~60 instances of our app each bundled with a SE node across ~20 single node k8s clusters (so each cluster runs 3 apps each with a chrome SE node). One extra k8s cluster runs on the CI agent itself where SE hub is running. For various reasons - which for brevity I don't want to detail - these clusters communicate with each other via node ports (on private IPs).

So I end up in a situation where I can configure SE node so it can reach SE hub via (let's say 192.168.1.2:30027-30030) and I can configure it to report its own IP correctly (let's say 192.168.1.3, via --host), but I get into a problem with --port.
If I leave it on it's default (5555), then hub will try to connect to it on 192.168.1.3:5555 (but that is not route-able). One solution that works is picking a port from node port range and use that port both for node port and start SE node on that port inside the container but this has an unwanted administration overhead as well as adding non deterministic behavior as static node port range is only available on k8s version 1.27 in a feature that is not generally available yet.

It is a lot simpler if this feature is added.

Usage example

Until docker part is updated we could use SE_OPTS to supply --external-uri=http://<node-ip>:<node-port>.

[github-actions]

@utamas, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[diemol]

Have you tried the --bind-host flag?

[utamas]

That flag is set by docker packaging to false by default. I didn't try to change that for two reasons:

• It would bind to the container IP (which is not resolvable for hub, cause they are on completely different networks
• it would only solve half of my problem, cause I cannot expose the service on 5555 to hub

Or am I missing something?

[diemol]

If you set it to false you can set any host/port you need to be reported. As long as it maps with the host where the container is running. Have you tried it?

[utamas]

Sorry, I didn't answer with yes or no.

So, yes. I tried. While it works this approach has a couple of downsides:

• it forces me to use kubernetes version 1.27 (at least) in order to have a more or less stable deployment,
• even then I would rely on a feature that is not generally available yet and finally,
• it would add complexity to my deployment code as I would have to associate ports to certain SE nodes manually and make sure it does not collide with other services.

What I'm doing ATM is I create a node port type k8s service that points to 5555 before actually applying deployment that runs SE nodes. This way when I apply the deployment I already know what port value I need to set to --external-uri. The reason why I can't set the port with a similar trick is that when I create the k8s service I would already need to know what node port I would get (dynamically) which introduces a chicken and egg problem.

I hope I'm not very cryptic.

[utamas]

Here is a snippet from my code:

resource "kubernetes_service_v1" "to_hub" {
  metadata {
    // redacted for brevity 
  }
  spec {
    type     = "NodePort"
    // redacted for brevity 

    port {
      port        = 5555
      target_port = 5555
      // by not picking a value for node_port, I always get an available one by random
    }
  }
}

resource "kubernetes_deployment_v1" "main" {
  // this is important, cause it makes sure that a node port is allocated and known by the time terraform starts to create this deployment
  depends_on = [kubernetes_service_v1.to_hub]

  metadata {
    // redacted for brevity 
  }
  spec {
    // redacted for brevity 
    template {
      // redacted for brevity 
      spec {
        // redacted for brevity 
        container {
          image = "${var.image_registry}/node-chrome:114.0-chromedriver-114.0-grid-4.10.0-20230614-patched"

          dynamic "env" {
            for_each = {
              SE_NODE_GRID_URL            = "http://${var.hub_host}:${var.hub_port}"
              SE_EVENT_BUS_HOST           = var.hub_host
              SE_EVENT_BUS_PUBLISH_PORT   = var.event_bus_publish_port
              SE_EVENT_BUS_SUBSCRIBE_PORT = var.event_bus_subscribe_port
              SE_NODE_SESSION_TIMEOUT     = 7200

              SE_NODE_HOST = var.cluster_address
              SE_OPTS = "--external-uri http://${var.cluster_address}:${kubernetes_service_v1.to_hub.spec.0.port.0.node_port}"
            }
            content {
              name  = env.key
              value = env.value
            }
          }

          port {
            container_port = 5555
          }

[utamas]

@diemol could you please remote the awaiting answer tag?

[titusfortner]

@diemol could you please remote the awaiting answer tag?

We use that tag to know which items to see if we need to follow up on. Hard to juggle 200 open issues. 😄
Diego has been traveling this week.

Either he or maybe @pujagani will hopefully be able to verify #12508 soon.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.