Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: HADOOP in Production


Copy link to this message
-
RE: HADOOP in Production
Good points.

I'm not trying to be exhaustive in the discussion of real time systems.
My only intent was to point out the difference between real time and fast response.
There are lots of real time requirements that do not require a particularly fast response but the response needs to be on time.  Timing errors can introduce noise at a pretty astounding rate.
Real fast often uses the same technology as real time but the consequences of missing a deadline are different.

As for the high velocity traders, I'm an advocate of the Robin Hood Tax.
A little friction might help the world.

Hank Cohen

From: Ted Dunning [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, October 02, 2012 4:13 PM
To: [EMAIL PROTECTED]
Subject: Re: HADOOP in Production
On Tue, Oct 2, 2012 at 7:05 PM, Hank Cohen <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
There is an important difference between real time and real fast

Real time means that system response must meet a fixed schedule.
Real fast just means sooner is better.

Good thought, but real-time can also include a fixed schedule and a specified list of exceptional conditions which would prevent meeting the schedule.

It may also include a fixed schedule that must be met some fraction of the time (usually very near 100% of the time).

Without providing exceptions, you basically force the designer to lie about how reliable their system is.

Real time systems always have hard schedules. The schedule could be in microseconds to control a laser for making masks for semiconductor manufacturing, milliseconds to control the ignition in your car or flight controls on an F-22 or  seconds for even slower moving processes.  In real time system missing the schedule can mean that very bad things happen: planes fall from the sky, your laser printer fries it's imaging drum, factories explode etc.

It can mean that.  But if you specify the exceptional situations you can specifically mitigate for them.

Most transaction processing is happy with real fast
The folks doing high velocity trading are pretty close to real time but they probably will be happy with real fast.
If real fast systems miss a schedule then someone loses money.

Yeah.  And if you talk to these guys, they know the difference and ask for real-time.
The reason that RTOS type operating systems are popular for real time applications is that they don't allow operations to spend indeterminate amounts of time in uninterruptable states.  Java will never qualify as a real time system because it has garbage collection and garbage collection can lock up a system for an indefinite amount of time while it goes through marking and counting.

You are behind the times on a few counts.

- Java's collectors don't "count".

- Java can be real-time:

http://rtjava.blogspot.co.uk/2009/07/real-time-java-vms.html

- Garbage collection can be deterministic and real-time:

http://www.ibm.com/developerworks/java/library/j-rtj4/index.html
http://docs.oracle.com/cd/E13221_01/wlrt/docs30/intro_wlrt/tuning.html

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB