Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Fwd: Introducing Watchtower - Light Table for Pig

Copy link to this message
Fwd: Introducing Watchtower - Light Table for Pig
Awesome development from our pals at Mortar Data!

Russell Jurney http://datasyndrome.com

Begin forwarded message:

*From:* Scott from Mortar Data <[EMAIL PROTECTED]>
*Date:* August 7, 2013, 8:55:23 AM PDT
*To:* Russell <[EMAIL PROTECTED]>
*Subject:* *Introducing Watchtower - Light Table for Pig*
*Reply-To:* Scott from Mortar Data <[EMAIL PROTECTED]>

 Introducing Watchtower - Light Table for Pig

Introducing Watchtower - Light Table for Pig

There are no two ways around it, Hadoop development iterations are slow.
Traditional programmers have always had the benefit of re-compiling their
app, running it, and seeing the results within seconds. They have near
instant validation that what they’re building is actually working. When
you’re working with Hadoop, dealing with gigabytes of data, your
development iteration time is more like hours.

Inspired by the amazing real-time feedback experience of Light
we’ve built Mortar Watchtower to bring back that almost instant iteration
cycle developers are used to. Not only that, Watchtower also helps surface
the semantics of your Pig scripts, to give you insight into how your
scripts are working, not just that they are working.
 Instant Feedback

Watchtower is a daemon that sits in the background, continuously flowing a
sample of your data through your script while your work. It captures what
your data looks like, and shows how it mutates at each step, directly
inline with your script.

[image: image]
 Better Error Handling

Watchtower helps flush out the errors in your script. A common problem for
Pig developers is making sure your UDFs don’t have errors. When we’re
talking about a job that can take eight hours to run, finding an error in
your UDF that runs at the end of your execution can be painful. Since
Watchtower is constantly sending data through your entire script, errors
are surfaced and displayed instantly. No longer will you be surprised when
at the eleventh hour your Hadoop job fails.

[image: image]
 Watchtower Sees Everything

Watchtower is always looking for changes in your Mortar Project. If you
edit anything that alters the functionality of your script (including data
samples, UDFs, or parameters), Watchtower will know, start recompiling the
results, and almost instantly feed the results back to you. All you have to
do is save.
 Getting Started is Easy

Best of all Watchtower is incredibly simple to get started with. If you’re
already setup with Mortar Development Framework then all you have to do is:

$ mortar plugins:install https://github.com/mortardata/watchtower.git

 Then start watching a script. For example:

$ git clone https://github.com/mortardata/mortar-examples.git
$ cd mortar-examples
$ mortar watch pigscripts/coffee_tweets.pig \
  -f params/coffee_tweets/local.large.params

 We hope you love getting real-time feedback on your Pig scripts. Enjoy!
want to make our engineers cry? unsubscribe
   update subscription