2020-12-07 - KDB Project Call Meeting Minutes

[alvin-c-shih]

Date

Monday, December 7, 2020 - 10am EST / 3pm BST

// First Monday of every month

Attendees

Fullname	Affiliation	GitHub Username
Timofey Beletskiy	JP Morgan
Aaron Davies	Morgan Stanley	@adavies42
Alvin Shih	Morgan Stanley	@alvin-c-shih
Peter Gyorok	Morgan Stanley	@gyorokpeter
Rob Underwood	FINOS	@brooklynrob
Jonny Press	AquaQ Analytics	@jonnypress
Alok Dutt	Citi

Minutes

• (5 min) Convene, roll call, welcome new people.
• (5 min) Approve previous meeting minutes.
  • Approved by acclamation.
• (5 min) Scheduling tweaks.
  • Moving time of meeting to 9:30 approved by acclamation.
• (5 min) Kdb Studio updates.
  • Updated README about icons.
  • Download instructions clarified.
  • Small dataset to replicate graph for screen shot added.
  • Still looking for help with generating launcher using javaw.exe instead of java.exe.
• (5 min) dep.q as a first iteration of FINOS module loader. add module loader #33
  • GitHub project will be reorganized to have a q/ directory for compatible modules.
    • More clearly distinguish from other contributions like technical-documentation/.
  • Won't worry about layout for C/C++ plugins until some are contributed.
    • Look to Python / Perl for inspiration when the time comes.
• (10 min) PyQ wrapper.
  • Several hassles related to HDBs / splayed tables.
  • Suggestions on how attributes / link columns can be meaningful in Python?
    • Will be easier if we have a script to generate sample HDB that makes the issues reproducible.
  • Start with a finos.q wrapper for pyq.q that detects HDB tables and reformulates as functional form of select.
  • Will check with Aaron Williamson whether can fork Bottleneck which is licensed under BSD-2.
    • It's what Pandas uses for NaN handling. Create a variant that also understands kdb nulls and infinities.
    • https://pypi.org/project/Bottleneck/
    • Inconsistencies in nansum for float32 dtype compared to numpy pydata/bottleneck#193
• (5 min) Vaex mentioned both by AquaQ and Kx.
  • https://www.aquaq.co.uk/q/comparing-columnar-data-formats-arrow-vaex-and-kdb/
  • https://www.linkedin.com/pulse/python-data-analysis-really-simple-ferenc-bodon-ph-d-/
  • Vaex currently V3.0.0. V4.0.0 currently in Alpha.
    • https://pypi.org/project/vaex/
    • https:/vaexio/vaex/releases/tag/core-v4.0.0-alpha.9
  • No push from users as of yet. Revisit later.
• (5 min) BigQuery
  • Some looking at hybrid model for 2021 : shallow history kdb / deep history BigQuery.
  • Was hoping for transfers of gigabytes / second from cloud provider, but seems to max out at 300 megabytes / second.
  • Small tables don't work well due to slow metadata queries.
• (5 min) AOB, Q&A, Adjourn.
  • Stories about Lustre filesystem for kdb on premises and in cloud welcome.
  • Tools for migrating to kdb+ 4.0?
  • Interoperability with Hadoop?
  • Utilities for accessing REST data sources?
    • Abstraction that might also work for on-disk and in-memory data?

WebEx info

Webex

• https://finos.webex.com/finos/j.php?MTID=mfc9190e818fd85f9e3ece5166f23d920

Dial-in and Project Materials

• US: +1-415-655-0003 US Toll
• UK: +44-20319-88141 UK Toll
• Access code: 663 746 304
• GitHub Repos:
  • https:/finos/kdb
  • https:/finos/kdb-studio
• Wiki Page:
  • https://finosfoundation.atlassian.net/wiki/spaces/DT/pages/329383945/kdb+Project
• KDB Metrics on FINOS
  • https://metrics.finos.org/goto/033b15c80f0d08cab70f40f4358932f6

[brooklynrob]

I am here.

[brooklynrob]

Sounds like decision was made to permanently move time to 9:30am ET, correct @alvin-c-shih? (FYI @mcleo-d @aitana16 @maoo)

[adavies42]

Aaron Davies, Morgan Stanley.

[brooklynrob]

Here are the people on the call

[tbel]

Tim Beletskiy, JPMC (logged in anyway)

[alvin-c-shih]

Sounds like decision was made to permanently move time to 9:30am ET, correct @alvin-c-shih? (FYI @mcleo-d @aitana16 @maoo)

Yes. Thanks!

[brooklynrob]

Here's the issue I crated related to evaluating how to use/incorporate Bottleneck within the Kdb project --> #37

[agitana]

Sounds like decision was made to permanently move time to 9:30am ET, correct @alvin-c-shih? (FYI @mcleo-d @aitana16 @maoo)

Yes. Thanks!

Thanks, I've adjusted the calendar invite and webex details accordingly. The next meeting will be on January 4th at 9:30am ET. / cc @brooklynrob @alvin-c-shih