Guidelines on how to achieve real-time data and stream processing at scale

Meet Kafka

How we move the data becomes nearly as important as the data itself. Publish/subscribe messaging is a pattern that is characterized by the senders (publishers) of a piece of data (message) and consumers of the message that are loosely decoupled from each other. Pub/sub systems like Kafka are often designed with the help of an intermediary broker to orchestrate this pattern to enable diverse use cases.

A message is simply an array of bytes as far as Kafka is concerned, so the data contained within it does not have a specific format or meaning to Kafka. A message can have…

(Updated: May, 2020) How to perform at your peak with the help of mobile apps

Energy, not time, is the fundamental source of efficiency and high performance.

Photo by Peter Conlan on Unsplash

1. Use apps to remind you to take a break

In concurrent applications a thread might fail and die due to uncaught runtime exceptions even without noticing since the application may continue to work. Losing one consumer thread from a thread pool can be tolerable, but losing a single dispatcher thread can degrade application workflow.

There are four alternative approaches in Java to get notified when a task fails due to an exception, so that you can log or take some recovery action.

Proactive Approach

In the solution below, run() method of a thread is structured with try-catch block and if a task throws an unchecked exception, it allows the…

Presentation based on Mil-DDS IoT Suite

Originally published at on May 24, 2015.

The OMG Data Distribution Service for Real-Time Systems (DDS) is an open international middleware standard that supports data-centric publish-subscribe communications that aims to enable scalable, real-time, dependable, high-performance and interoperable data exchanges for large-scale distributed systems.

Mil-DDS Core Messaging Suite

The MilSoft DDS (Mil-DDS) product supports the following technical specifications at minimum:

  • OMG DDS v1.2 compliance (Minimum Profile, Persistence Profile, Content Subscription Profile, Ownership Profile)
  • OMG Interoperability Wire Protocol (RTPS) v2.1 compliance
  • Operating Systems (Windows all, Linux — tested on CentOS, Fedora, Ubuntu, UNIX — tested on Solaris)
  • Processors (x86, x64, ARM, SPARC)
  • Programming Languages (C++, C#/.NET, Java)
  • Transports (UDPv4…

The OMG Data Distribution Service for Real-Time Systems (DDS) is an open international middleware standard that supports data-centric publish-subscribe communications for large-scale distributed systems.

DDS is a protocol for the Internet of Things (IoT), which enables network interoperability for connected machines, enterprise systems, and mobile devices. It provides scalability, performance, and Quality of Service required to support IoT applications.

DDS can be deployed in platforms ranging from low-footprint devices to the Cloud and supports efficient bandwidth usage as well as agile orchestration of system components. It provides a global data space for analytics and enables flexible real-time system integration.

Speakers: Huseyin Kutluca and Abdullah Ozturk

Location: ODTU Teknokent — Ikizler Conference Hall, Ankara, TURKEY

Date/Time: 14th of April, 2015 (13:30–17:30)

Program: (below)

Mil-DDS Workshop Flyer

Originally published at on April 2, 2015.

Traditional software solutions use managed servers and containers, and scale via larger servers and multi-threading.

Today’s systems demand new requirements, because they are deployed on everything (mobile, cloud clusters etc.), have to benefit from multicore processors, show low response time (milli, or micro-second), are 100% uptime (highly available), and deal with big data (petabytes).

Going Reactive

Coherent approach to systems architecture is needed. Reactive Systems meet the following requirements;

  • Responsive,
  • Resilient,
  • Elastic,
  • and Message Driven.

Reactive Systems are more flexible, loosely-coupled and scalable. They are easier to develop and amenable to change. They are significantly more tolerant of failure and when failure…

I’ve recently open-sourced my side project on GitHub that is Asynchronous Messaging Service (AMS).

In summary, it is a message passing framework based on ZeroMQ for low latency distributed systems seeking loose-coupling, fault-tolerance, dynamic discovery and scalability.



Asynchronous message passing systems deliver a message from sender to receiver, without waiting for the receiver to be ready. The advantage of asynchronous communication is that the sender and receiver can overlap their computation because they do not wait for each other.

The reactor used in AMS handles requests delivered concurrently by multiple event resources. Message dispatcher handles registering and unregistering of application-defined…

Below, you can find my presentation about Internet of Things (IoT) and how IoT world can benefit from The Data Distribution Service for Real-Time Systems (DDS), which is an Object Management Group (OMG) machine-to-machine (M2M) middleware standard that aims to enable scalable, real-time, dependable, high-performance and interoperable data exchanges between publishers and subscribers.

I presented it in Career Forum ’14 hosted by Bilkent IEEE on the 22th of Dec 2014, and in BYK’14 hosted by Hacettepe ACM on the 24th of Dec 2014.

Appreciate both university groups for their great organizations and welcomeness.

RocksDB is an embeddable persistent key-value store for fast storage open-sourced by Facebook.

RocksDB builds on LevelDB (created by Google) to be scalable to run on servers with many CPU cores, to efficiently use fast storage and to support IO-bound, in-memory and write-once workloads. However, RocksDB excels when the data stored is larger than the size of RAM on the target machine.

RocksDB solidly outperformed LevelDB for these IO bound workload according to the benchmarks conducted by Facebook:

  • Bulk Load of keys in Random Order
  • Bulk Load of keys in Sequential Order
  • Write Performance
  • Read Performance

Thus, RocksDB can be…

Abdullah Ozturk - Blog

Software engineer. Distributed systems enthusiast. #data, #iot, #mobile, #scalability, #cplusplus, #java

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store