Guidelines on how to achieve real-time data and stream processing at scale

Meet Kafka

How we move the data becomes nearly as important as the data itself. Publish/subscribe messaging is a pattern that is characterized by the senders (publishers) of a piece of data (message) and consumers of the message that are loosely decoupled from each other. Pub/sub systems like Kafka are often designed with the help of an intermediary broker to orchestrate this pattern to enable diverse use cases.

Image for post
Image for post

A message is simply an array of bytes as far as Kafka is concerned, so the data contained within it does not have a specific format or meaning to Kafka. A message can have an optional bit of metadata, which is referred to as a key. Keys are used when messages are to be written to partitions in a more controlled manner. For efficiency, messages are written into Kafka in batches. A batch is just a collection of messages, all of which are being produced to the same topic and partition to avoid individual roundtrips across the network for each message. …


(Updated: May, 2020) How to perform at your peak with the help of mobile apps

Energy, not time, is the fundamental source of efficiency and high performance.

Image for post
Image for post
Photo by Peter Conlan on Unsplash

1. Use apps to remind you to take a break

Image for post
Image for post

The recovery is an integral part of sustaining high levels of energy and thus high performance. Have a life-sustaining rhythm and disengage at regular intervals i.e. rest every 90 to 120 minutes.

Image for post
Image for post
Time Out: Break Reminders

2. Use apps to track hydration and water drinking

Image for post
Image for post

Drink enough water. Mild dehydration (fluid loss of 1–3%) can impair energy levels and mood, and lead to major reductions in memory and brain performance.

Image for post
Image for post
My Water: Drink Reminder

3. Use apps for sleep tracking

Image for post
Image for post

Take optimum amount of sleep. Having not enough sleep impairs your creativity, problem-solving, and memory that won’t help your productivity.

Image for post
Image for post
Pillow: Automatic Sleep Tracker

4. Use apps to track your workouts (gym, or home exercises)

Image for post
Image for post

Work out regularly, and incorporate small exercises into your everyday lives.

  • Exercise is shown to reduce stress, combat fatigue, improve performance and lead to fewer workday absences. …


In concurrent applications a thread might fail and die due to uncaught runtime exceptions even without noticing since the application may continue to work. Losing one consumer thread from a thread pool can be tolerable, but losing a single dispatcher thread can degrade application workflow.

There are four alternative approaches in Java to get notified when a task fails due to an exception, so that you can log or take some recovery action.

Proactive Approach

In the solution below, run() method of a thread is structured with try-catch block and if a task throws an unchecked exception, it allows the thread to die. …


Presentation based on Mil-DDS IoT Suite

Originally published at tech.aozturk.me on May 24, 2015.


The OMG Data Distribution Service for Real-Time Systems (DDS) is an open international middleware standard that supports data-centric publish-subscribe communications that aims to enable scalable, real-time, dependable, high-performance and interoperable data exchanges for large-scale distributed systems.

Image for post
Image for post

Mil-DDS Core Messaging Suite

The MilSoft DDS (Mil-DDS) product supports the following technical specifications at minimum:

  • OMG DDS v1.2 compliance (Minimum Profile, Persistence Profile, Content Subscription Profile, Ownership Profile)
  • OMG Interoperability Wire Protocol (RTPS) v2.1 compliance
  • Operating Systems (Windows all, Linux — tested on CentOS, Fedora, Ubuntu, UNIX — tested on Solaris)
  • Processors (x86, x64, ARM, SPARC)
  • Programming Languages (C++, C#/.NET, Java)
  • Transports (UDPv4 Unicast &…


The OMG Data Distribution Service for Real-Time Systems (DDS) is an open international middleware standard that supports data-centric publish-subscribe communications for large-scale distributed systems.

DDS is a protocol for the Internet of Things (IoT), which enables network interoperability for connected machines, enterprise systems, and mobile devices. It provides scalability, performance, and Quality of Service required to support IoT applications.

DDS can be deployed in platforms ranging from low-footprint devices to the Cloud and supports efficient bandwidth usage as well as agile orchestration of system components. It provides a global data space for analytics and enables flexible real-time system integration.

Speakers: Huseyin Kutluca and Abdullah Ozturk

Location: ODTU Teknokent — Ikizler Conference Hall, Ankara, TURKEY

Date/Time: 14th of April, 2015 (13:30–17:30)

Program: (below)

Image for post
Image for post
Mil-DDS Workshop Flyer

Originally published at tech.aozturk.me on April 2, 2015.


Traditional software solutions use managed servers and containers, and scale via larger servers and multi-threading.

Today’s systems demand new requirements, because they are deployed on everything (mobile, cloud clusters etc.), have to benefit from multicore processors, show low response time (milli, or micro-second), are 100% uptime (highly available), and deal with big data (petabytes).

Going Reactive

Coherent approach to systems architecture is needed. Reactive Systems meet the following requirements;

  • Responsive,
  • Resilient,
  • Elastic,
  • and Message Driven.
Image for post
Image for post

Reactive Systems are more flexible, loosely-coupled and scalable. They are easier to develop and amenable to change. They are significantly more tolerant of failure and when failure occurs they meet it with elegance. …


I’ve recently open-sourced my side project on GitHub that is Asynchronous Messaging Service (AMS).

In summary, it is a message passing framework based on ZeroMQ for low latency distributed systems seeking loose-coupling, fault-tolerance, dynamic discovery and scalability.

Features

Asynchronous

Asynchronous message passing systems deliver a message from sender to receiver, without waiting for the receiver to be ready. The advantage of asynchronous communication is that the sender and receiver can overlap their computation because they do not wait for each other.

The reactor used in AMS handles requests delivered concurrently by multiple event resources. Message dispatcher handles registering and unregistering of application-defined message handlers, and dispatches messages from the demultiplexer to the associated handlers. …


Below, you can find my presentation about Internet of Things (IoT) and how IoT world can benefit from The Data Distribution Service for Real-Time Systems (DDS), which is an Object Management Group (OMG) machine-to-machine (M2M) middleware standard that aims to enable scalable, real-time, dependable, high-performance and interoperable data exchanges between publishers and subscribers.

I presented it in Career Forum ’14 hosted by Bilkent IEEE on the 22th of Dec 2014, and in BYK’14 hosted by Hacettepe ACM on the 24th of Dec 2014.

Image for post
Image for post

Appreciate both university groups for their great organizations and welcomeness.

Image for post
Image for post

Updated the presentation based on Mil-DDS IoT Suite.

Originally published at tech.aozturk.me on July 10, 2014.


RocksDB is an embeddable persistent key-value store for fast storage open-sourced by Facebook.

RocksDB builds on LevelDB (created by Google) to be scalable to run on servers with many CPU cores, to efficiently use fast storage and to support IO-bound, in-memory and write-once workloads. However, RocksDB excels when the data stored is larger than the size of RAM on the target machine.

RocksDB solidly outperformed LevelDB for these IO bound workload according to the benchmarks conducted by Facebook:

  • Bulk Load of keys in Random Order
  • Bulk Load of keys in Sequential Order
  • Write Performance
  • Read Performance

Thus, RocksDB can be used by applications that need low latency database accesses. You can install RocksDB and try it on your local machine by following the steps here. …

About

Abdullah Ozturk - Blog

Software development engineer. Distributed systems enthusiast. #data, #iot, #mobile, #scalability, #cplusplus, #java https://github.com/aozturk

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store