How we move the data becomes nearly as important as the data itself. Publish/subscribe messaging is a pattern that is characterized by the senders (publishers) of a piece of data (message) and consumers of the message that are loosely decoupled from each other. Pub/sub systems like Kafka are often designed with the help of an intermediary broker to orchestrate this pattern to enable diverse use cases.
A message is simply an array of bytes as far as Kafka is concerned, so the data contained within it does not have a specific format or meaning to Kafka. A message can have an optional bit of metadata, which is referred to as a key. Keys are used when messages are to be written to partitions in a more controlled manner. For efficiency, messages are written into Kafka in batches. A batch is just a collection of messages, all of which are being produced to the same topic and partition to avoid individual roundtrips across the network for each message. …
In concurrent applications a thread might fail and die due to uncaught runtime exceptions even without noticing since the application may continue to work. Losing one consumer thread from a thread pool can be tolerable, but losing a single dispatcher thread can degrade application workflow.
There are four alternative approaches in Java to get notified when a task fails due to an exception, so that you can log or take some recovery action.
In the solution below, run() method of a thread is structured with try-catch block and if a task throws an unchecked exception, it allows the thread to die. …
The OMG Data Distribution Service for Real-Time Systems (DDS) is an open international middleware standard that supports data-centric publish-subscribe communications that aims to enable scalable, real-time, dependable, high-performance and interoperable data exchanges for large-scale distributed systems.
Mil-DDS Core Messaging Suite
The MilSoft DDS (Mil-DDS) product supports the following technical specifications at minimum:
The OMG Data Distribution Service for Real-Time Systems (DDS) is an open international middleware standard that supports data-centric publish-subscribe communications for large-scale distributed systems.
DDS is a protocol for the Internet of Things (IoT), which enables network interoperability for connected machines, enterprise systems, and mobile devices. It provides scalability, performance, and Quality of Service required to support IoT applications.
DDS can be deployed in platforms ranging from low-footprint devices to the Cloud and supports efficient bandwidth usage as well as agile orchestration of system components. It provides a global data space for analytics and enables flexible real-time system integration.
Speakers: Huseyin Kutluca and Abdullah Ozturk
Location: ODTU Teknokent — Ikizler Conference Hall, Ankara, TURKEY
Date/Time: 14th of April, 2015 (13:30–17:30)
Originally published at tech.aozturk.me on April 2, 2015.
Traditional software solutions use managed servers and containers, and scale via larger servers and multi-threading.
Today’s systems demand new requirements, because they are deployed on everything (mobile, cloud clusters etc.), have to benefit from multicore processors, show low response time (milli, or micro-second), are 100% uptime (highly available), and deal with big data (petabytes).
Coherent approach to systems architecture is needed. Reactive Systems meet the following requirements;
Reactive Systems are more flexible, loosely-coupled and scalable. They are easier to develop and amenable to change. They are significantly more tolerant of failure and when failure occurs they meet it with elegance. …
I’ve recently open-sourced my side project on GitHub that is Asynchronous Messaging Service (AMS).
In summary, it is a message passing framework based on ZeroMQ for low latency distributed systems seeking loose-coupling, fault-tolerance, dynamic discovery and scalability.
Asynchronous message passing systems deliver a message from sender to receiver, without waiting for the receiver to be ready. The advantage of asynchronous communication is that the sender and receiver can overlap their computation because they do not wait for each other.
The reactor used in AMS handles requests delivered concurrently by multiple event resources. Message dispatcher handles registering and unregistering of application-defined message handlers, and dispatches messages from the demultiplexer to the associated handlers. …
Below, you can find my presentation about Internet of Things (IoT) and how IoT world can benefit from The Data Distribution Service for Real-Time Systems (DDS), which is an Object Management Group (OMG) machine-to-machine (M2M) middleware standard that aims to enable scalable, real-time, dependable, high-performance and interoperable data exchanges between publishers and subscribers.
I presented it in Career Forum ’14 hosted by Bilkent IEEE on the 22th of Dec 2014, and in BYK’14 hosted by Hacettepe ACM on the 24th of Dec 2014.
Appreciate both university groups for their great organizations and welcomeness.
RocksDB is an embeddable persistent key-value store for fast storage open-sourced by Facebook.
RocksDB builds on LevelDB (created by Google) to be scalable to run on servers with many CPU cores, to efficiently use fast storage and to support IO-bound, in-memory and write-once workloads. However, RocksDB excels when the data stored is larger than the size of RAM on the target machine.
RocksDB solidly outperformed LevelDB for these IO bound workload according to the benchmarks conducted by Facebook:
Thus, RocksDB can be used by applications that need low latency database accesses. You can install RocksDB and try it on your local machine by following the steps here. …