Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Not able to use clickhouse profile #1249

Closed
1 task done
petershenri opened this issue Oct 10, 2024 · 2 comments · Fixed by #1255
Closed
1 task done

[Bug] Not able to use clickhouse profile #1249

petershenri opened this issue Oct 10, 2024 · 2 comments · Fixed by #1255
Labels
bug Something isn't working execution:docker Related to Docker execution environment profile:clickhouse Related to Clickhouse ProfileConfig triage-needed Items need to be reviewed / assigned to milestone

Comments

@petershenri
Copy link
Contributor

petershenri commented Oct 10, 2024

Astronomer Cosmos Version

main (development)

If "Other Astronomer Cosmos version" selected, which one?

1.7.0

dbt-core version

1.8.7

Versions of dbt adapters

dbt-adapters==1.7.0
dbt-clickhouse==1.8.2
dbt-common==1.10.0
dbt-core==1.8.7
dbt-extractor==0.5.1
dbt-semantic-interfaces==0.5.1

LoadMode

AUTOMATIC

ExecutionMode

DOCKER

InvocationMode

None

airflow version

2.9.2

Operating System

MACOS

If a you think it's an UI issue, what browsers are you seeing the problem on?

No response

Deployment

Docker-Compose

Deployment details

No response

What happened?

Hello,

When I try to connect to clickhouse with astronomer, I've got the error : DB::Exception: default: Authentication failed: password is incorrect, or there is no user with such name.

If I go to :
cosmos/profiles/clickhouse/user_pass.py
I can see :

    required_fields = [
        "host",
        "login",
        "schema",
        "clickhouse",
    ]
    secret_fields = [
        "password",
    ]
    airflow_param_mapping = {
        "host": "host",
        "login": "login",
        "password": "password",
        "port": "port",
        "schema": "schema",
        "clickhouse": "extra.clickhouse",
    }

Shouldn't it be with user instead ?

    required_fields = [
        "host",
        "user",
        "password",
        "dbname",
    ]
    secret_fields = [
        "password",
    ]
    airflow_param_mapping = {
        "host": "host",
        "user": "login",
        "password": "password",
        "port": "port",
        "dbname": "schema",
        "keepalives_idle": "extra.keepalives_idle",
        "sslmode": "extra.sslmode",
    }

My working configuration on DBT is :

project:
  outputs:
    stage:
      type: clickhouse
      user: "{{ env_var('DBT_USER') }}"
      password: "{{ env_var('DBT_PASSWORD') }}"
      #optional fields
      driver: native
      port: 9440
      host: myhost
      secure: true
      cluster: "all-replicated"
  target: stage

My unworking configuration on airflow :
image

profile_config = ProfileConfig(
    profile_name="project",
    target_name="stage",
    profile_mapping=ClickhouseUserPasswordProfileMapping(
        conn_id=f"clickhouse_{CONN_ENV}",
        profile_args={
            "cluster": "all-replicated",
            "secure": True,
        },
    ),
)

Thanks a lot,

Relevant log output

No response

How to reproduce

Launching airflow on docker and try to connect to clickhouse

Anything else :)?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Contact Details

@petershenri petershenri added bug Something isn't working triage-needed Items need to be reviewed / assigned to milestone labels Oct 10, 2024
Copy link

dosubot bot commented Oct 10, 2024

Hey @petershenri! I'm here to help you with your issue. I'm a bot that can assist you with bug fixes, answer questions, and guide you on becoming a contributor. Let's tackle this together!

To resolve the authentication error, ensure your Airflow configuration matches the required fields for the Clickhouse profile. The ClickhouseUserPasswordProfileMapping class requires the following fields:

  • host
  • login (this should map to user in your DBT configuration)
  • schema (this should map to dbname)
  • clickhouse

Make sure the password is set as an environment variable. Your Airflow connection should map these parameters correctly:

airflow_param_mapping = {
    "host": "host",
    "login": "login",  # Ensure this is set to your DBT 'user'
    "password": "password",
    "port": "port",
    "schema": "schema",  # Ensure this is set to your DBT 'dbname'
    "clickhouse": "extra.clickhouse",
}

Adjust your Airflow configuration to ensure these mappings are correct, especially the login and schema fields [1].

To continue talking to Dosu, mention @dosu.

@dosubot dosubot bot added execution:docker Related to Docker execution environment profile:clickhouse Related to Clickhouse ProfileConfig labels Oct 10, 2024
@petershenri
Copy link
Contributor Author

Hi,

Update on my issue.
It's working with the user instead of login :

"""Maps Airflow Postgres connections using user + password authentication to dbt profiles."""

from __future__ import annotations

from typing import Any

from ..base import BaseProfileMapping


class ClickhouseUserPasswordProfileMapping(BaseProfileMapping):
    """
    Maps Airflow generic connections using user + password authentication to dbt Clickhouse profiles.
    https://docs.getdbt.com/docs/core/connect-data-platform/clickhouse-setup
    """

    airflow_connection_type: str = "generic"
    dbt_profile_type: str = "clickhouse"
    default_port = 9000
    is_community = True

    required_fields = [
        "host",
        "user",
        "schema",
        "clickhouse",
    ]
    secret_fields = [
        "password",
    ]
    airflow_param_mapping = {
        "host": "host",
        "user": "login",
        "password": "password",
        "port": "port",
        "schema": "schema",
        "clickhouse": "extra.clickhouse",
    }

    def _set_default_param(self, profile_dict: dict[str, Any]) -> dict[str, Any]:
        if not profile_dict.get("driver"):
            profile_dict["driver"] = "native"

        if not profile_dict.get("port"):
            profile_dict["port"] = self.default_port

        if not profile_dict.get("secure"):
            profile_dict["secure"] = False
        return profile_dict

    @property
    def profile(self) -> dict[str, Any | None]:
        """Gets profile. The password is stored in an environment variable."""
        profile_dict = {
            **self.mapped_params,
            **self.profile_args,
            # password should always get set as env var
            "password": self.get_env_var_format("password"),
        }

        return self.filter_null(self._set_default_param(profile_dict))

    @property
    def mock_profile(self) -> dict[str, Any | None]:
        """Gets mock profile."""

        profile_dict = {
            **super().mock_profile,
        }

        return self._set_default_param(profile_dict)

I'm open to create a PR on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working execution:docker Related to Docker execution environment profile:clickhouse Related to Clickhouse ProfileConfig triage-needed Items need to be reviewed / assigned to milestone
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant