Skip to content

fillmore-labs/kafka-sensors-talk

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Kafka Serialization Talk (WIP)

Chapter 01: Introduction

  • No code, only whiteboard

  • Model the data

  • API first (sort of)

Chapter 02: Model & Business Logic

  • Implement model and business logic

  • Independent of any other part of the program

bazel test //chapter02/...

Chapter 03: JSON Serialization

  • Implement JSON API

  • Independent of our model

  • Can run integration tests with external sources

  • Can produce sample data for external clients

bazel test //chapter03/...

Chapter 04: Integrate JSON Serialization with Our Model

  • Glue code between external API and model (ports and adapters)

  • Must be adapated for every change in API or model

  • Is usually pretty simple

  • Concrete serialization format does not show up in our test

bazel test //chapter04/...

Chapter 05: Integrate Business Logic with Kafka Streams

  • Use our own embedded Kafka 3.0 Server, because we can

  • Define a typesafe Kafka Streams topology

bazel test //chapter05/...

Chapter 06: Our First main: A Complete Application

  • So far, we have been driven by tests

  • Run a local Kafka server

bazel run //:kafka-sensors

Chapter 07: More Serialization Formats

  • Protocol Buffers

  • Apache Thrift

  • Alternate JSON implementation: GSON

  • Serializing the model with mix-ins, useful for prototyping

  • Because we can: Amazon Ion

  • Same pattern as in Chapters 03 & 04

bazel test //chapter07/...

Chapter 08: Apache Avro

  • History: ApacheCon09: Avro

  • Apache Hadoop, Apache Pig, Apache Hive

  • Dynamic format

  • Writer and reader schema required

bazel test //chapter08/...

Chapter 09: Confluent Variants

bazel test //chapter09/...

Chapter 10: Benchmarks

  • Maybe not the most important part, but interesting

  • Some implementations are slow

  • Investigate

bazel run //:bench10

bazel run //:bench10 -- -p "format=avro-specific,avro-generic" \
  -prof "async:output=flamegraph;direction=forward" "Bench\\.serialize"

bazel run //:bench10 -- -p "format=gson,json" \
  -prof "async:output=flamegraph;direction=forward" "Bench\\.deserialize"

Chapter 11: Revised Serialization Implementations

  • Make serializers faster

bazel test //chapter11/...
bazel run //:benchmark

Chapter 12: Wrap up

  • Comparison of formats, what to use when

  • Used techniques

    • Value objects

    • Hexagonal Architecture

    • Development driven by tests

  • Decoupled development enables separate testing and benchmarking

  • Early integration tests possible

  • Slow implementations are not hidden in the business logic

  • Separate input and business rules validation