Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance new JDBC driver #769

Merged
merged 9 commits into from
Dec 8, 2021
Merged

Enhance new JDBC driver #769

merged 9 commits into from
Dec 8, 2021

Conversation

zhicwu
Copy link
Contributor

@zhicwu zhicwu commented Nov 30, 2021

This is derived from PR #747 focusing on below items:

  • add more connection properties: autoCommit, fetchSize, namedParameter, and wrapperObject
  • basic JDBC escape syntax support
    • {d '2021-11-01'} -> date'2021-11-01'
    • {t '12:34:56'} -> timestamp'1970-01-01 12:34:56'
    • {ts '2021-11-01 12:34:56'} -> timestamp'2021-11-01 12:34:56'
    • {ts '2021-11-01 12:34:56.78'} -> toDateTime64('2021-11-01 12:34:56.78', 2)
    • {tt 'my_temp\'_''table'} -> `my_temp'_'table`
      Note: tt is short for temporary table. It's a hint for using external table or creating a session-level temporary table.
  • enhance prepared statement
    • multiple values for batch insertion
    • leverage JDBC escape syntax and external table for non-insert query(in a similar way as input function based approach)
  • share test cases for http and grpc clients
  • better support of AggregateFunction(and further optimization to bitmap)
  • implement unwrap methods to expose Java client API for JDBC users
  • use HttpClient instead of HttpURLConnection in JDK11 or above
    Note: noticed error HTTP/1.1 header parser received no bytes occasianly during stress test.
  • treat ASCII and Unicode string differently
  • enhance multi-format support(grpc is inconsistent with http)
  • fix reported issues in new JDBC driver

@zhicwu zhicwu mentioned this pull request Nov 30, 2021
7 tasks
@github-actions
Copy link

Benchmark                                (client)  (connection)  (statement)   (type)   Mode  Cnt     Score    Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20   255.730 ± 33.443  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20   244.872 ± 27.911  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20   241.383 ± 29.185  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20   244.024 ± 26.123  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   259.489 ± 31.722  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   253.774 ± 26.347  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   254.491 ± 28.010  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   254.423 ± 30.587  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20  1138.977 ± 72.510  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20  1109.145 ± 86.095  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20  1118.036 ± 95.677  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20  1132.939 ± 91.070  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   810.937 ± 67.500  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   781.523 ± 84.210  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   805.998 ± 46.024  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   806.811 ± 50.495  ops/s

@zhicwu zhicwu linked an issue Nov 30, 2021 that may be closed by this pull request
@github-actions
Copy link

Benchmark                                (client)  (connection)  (statement)   (type)   Mode  Cnt     Score     Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20   283.379 ±  35.336  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20   269.148 ±  30.990  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20   269.472 ±  26.237  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20   268.098 ±  27.125  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   283.133 ±  30.592  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   282.643 ±  31.169  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   281.800 ±  30.597  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   281.942 ±  29.726  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20  1608.059 ±  92.305  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20  1558.606 ± 112.378  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20  1613.891 ±  91.719  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20  1592.420 ±  68.266  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20  1096.276 ±  48.510  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20  1061.168 ±  77.316  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20  1101.689 ±  89.135  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20  1072.805 ±  79.278  ops/s

@zhicwu zhicwu linked an issue Dec 4, 2021 that may be closed by this pull request
@zhicwu zhicwu added this to the 0.3.2 Release milestone Dec 4, 2021
@github-actions
Copy link

github-actions bot commented Dec 8, 2021

Benchmark                                (client)  (connection)  (statement)   (type)   Mode  Cnt     Score    Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20   279.425 ± 33.855  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20   266.141 ± 29.106  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20   269.157 ± 27.881  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20   267.814 ± 28.750  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   278.323 ± 29.491  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   278.794 ± 31.054  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   280.934 ± 34.863  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   279.337 ± 32.828  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20  1618.425 ± 73.625  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20  1612.605 ± 58.300  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20  1616.692 ± 77.615  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20  1583.434 ± 92.444  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20  1092.021 ± 85.166  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20  1092.787 ± 71.698  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20  1109.759 ± 77.649  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20  1067.039 ± 68.852  ops/s

@github-actions
Copy link

github-actions bot commented Dec 8, 2021

Benchmark                                (client)  (connection)  (statement)   (type)   Mode  Cnt     Score     Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20   266.121 ±  37.927  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20   257.983 ±  31.577  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20   252.154 ±  30.784  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20   252.245 ±  29.386  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   270.608 ±  29.727  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   269.109 ±  32.579  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   268.763 ±  29.211  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   266.735 ±  28.828  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20  1368.578 ± 145.139  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20  1427.551 ±  88.778  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20  1465.133 ± 101.509  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20  1441.248 ±  88.458  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   994.004 ±  62.392  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   976.685 ±  98.294  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   972.701 ±  89.502  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   979.755 ±  86.927  ops/s

@github-actions
Copy link

github-actions bot commented Dec 8, 2021

Benchmark                                (client)  (connection)  (statement)   (type)   Mode  Cnt     Score     Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20   260.208 ±  29.242  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20   252.733 ±  28.197  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20   248.445 ±  29.474  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20   252.653 ±  26.569  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   263.851 ±  30.288  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   260.102 ±  28.452  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   265.185 ±  30.907  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   263.373 ±  29.602  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20  1436.243 ±  86.496  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20  1392.293 ±  89.150  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20  1412.571 ± 139.486  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20  1464.941 ±  66.823  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   981.946 ±  79.988  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   949.271 ±  66.203  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   993.618 ±  69.689  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   953.762 ±  80.688  ops/s

@github-actions
Copy link

github-actions bot commented Dec 8, 2021

Benchmark                                (client)  (connection)  (statement)   (type)   Mode  Cnt     Score    Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20   260.092 ± 33.518  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20   245.143 ± 28.972  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20   248.228 ± 30.320  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20   243.589 ± 28.839  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   260.548 ± 30.711  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   262.268 ± 25.758  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   258.676 ± 29.638  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   256.359 ± 29.216  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20  1418.451 ± 91.482  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20  1383.994 ± 94.453  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20  1417.434 ± 81.900  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20  1403.564 ± 80.492  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   917.643 ± 84.609  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   898.396 ± 73.978  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   925.803 ± 86.132  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   886.257 ± 76.310  ops/s

@zhicwu
Copy link
Contributor Author

zhicwu commented Dec 8, 2021

Merge to create test build - will have another PR to follow up.

@github-actions
Copy link

github-actions bot commented Dec 8, 2021

Benchmark                                (client)  (connection)  (statement)   (type)   Mode  Cnt     Score     Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20   300.201 ±  34.783  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20   279.710 ±  32.110  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20   268.165 ±  28.461  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20   276.630 ±  31.574  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   276.698 ±  35.829  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   283.046 ±  34.481  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   285.176 ±  29.879  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   291.168 ±  28.558  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20  1713.633 ±  83.573  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20  1627.350 ±  87.380  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20  1764.509 ±  82.723  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20  1699.085 ±  48.225  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20  1139.104 ±  73.807  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20  1113.275 ±  87.073  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20  1127.357 ±  90.888  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20  1122.686 ± 116.448  ops/s

@zhicwu zhicwu merged commit 754cb32 into ClickHouse:develop Dec 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1002 issue instead of 81 Incorrect parsing of complex types (Arrays and Tuples)
1 participant