TimescaleDB bridge - storing timeseries ngsild entity data in tsdb su…

…ccessfully

 - Related to epic #428

KafkaBridge/Dockerfile

@@ -2,6 +2,7 @@ FROM node:16
 COPY alerta/ /opt/alerta/
 COPY debeziumBridge/ /opt/debeziumBridge/
 COPY ngsildUpdates/ /opt/ngsildUpdates/
+COPY timescaledb/ /opt/timescaledb/
 ADD package.json /opt/package.json
 ADD package-lock.json package-lock.json
 COPY lib/ /opt/lib/

KafkaBridge/README.md

@@ -3,6 +3,7 @@
 1. Alerta bridge, a service which listens at a specific Kafka topic and forwards the data to Alerta.
 2. NgsildUpdates bride, a service which listens at a specific Kafka topic and forwards data to the scorpio/Ngsild broker.
 3. DebeziumBridge, a service which listens to debezium updates on ngsild entities and forward them to specific topics.
+4. Timescaledb Bridge, a service which listens at specific Kafka topic and forwards data to timescaledb(tsdb) running in postgres database
 
 ## Alerta Bridge
 
@@ -74,4 +75,22 @@ cat ngsildUpdates.json | tr -d '\n' | kafkacat -P -t iff.alerts -b my-cluster-ka
 ## Debezium Bridge
 
 The debezium bridge maps the updates from the NGSILD entity tabe to StreamingSQL tables.
-Concept TBD.
+Concept TBD.
+
+## TimescaledDB Bridge
+
+This service which listens at specific Kafka topic (iff.ngsild.attributes) and forwards NGSI-LD data to timescaledb(tsdb) running in postgres database.
+- Currently it is disabled by default. To enable please remove timescaledb deployment file link from .helmignore of kafkabridge charts
+
+* entities: Array of valid [NGSI-LD](https://www.etsi.org/deliver/etsi_gs/CIM/001_099/009/01.05.01_60/gs_CIM009v010501p.pdf) entities
+
+Data stored in the hypertable "entityhistories" of timescaledb database "tsdb" in below format:
+
+```
+tsdb=# select * from entityhistories;
+ observedAt | modifiedAt | entityId | attributeId | attributeType | datasetId | nodeType | value | index 
+----------------------------+----------------------------+--------------------+--------------------------------------------------+-------------------------------------------+---------------------------------------------------------------------+----------+--------------------------------------------------+-------
+ 2023-07-18 09:40:24.22+00 | 2023-07-18 09:40:24.22+00 | urn:plasmacutter:1 | https://industry-fusion.com/types/v0.9/state | https://uri.etsi.org/ngsi-ld/Property | urn:plasmacutter:1\https://industry-fusion.com/types/v0.9/state | @value | https://industry-fusion.com/types/v0.9/state_OFF | 0
+ 2023-07-18 09:38:15.559+00 | 2023-07-18 09:38:15.559+00 | urn:plasmacutter:1 | https://industry-fusion.com/types/v0.9/hasFilter | https://uri.etsi.org/ngsi-ld/Relationship | urn:plasmacutter:1\https://industry-fusion.com/types/v0.9/hasFilter | @id | urn:filter:1 | 0
+ 
+```

KafkaBridge/config/config.json

@@ -43,5 +43,14 @@
  },
  "bridgeCommon": {
  "kafkaSyncOnAttribute": "https://industry-fusion.com/types/v0.9/metadata/kafkaSyncOn"
+ },
+ "timescaledb": {
+ "topic": "iff.ngsild.attributes",
+ "PGSSLMODE": "require",
+ "dbname": "tsdb",
+ "hostname": "acid-cluster.iff.svc.cluster.local",
+ "port": "5432",
+ "username": "ngb",
+ "password": "POSTGRES_PASSWORD"
  }
-}
+}

KafkaBridge/package.json

@@ -12,6 +12,8 @@
  "dependencies": {
  "@comunica/actor-init-sparql-file": "1.22.3",
  "kafkajs": "^1.16.0",
+ "pg": "^8.2.1",
+ "sequelize": "^6.5.0",
  "keycloak-connect": "^17.0.1",
  "openid-client": "^5.1.2",
  "underscore": "^1.13.1",