Hi all,

Something has changed in how OSX and java handles IPv6, and now you
will get a log like:

2012-07-31 18:21:39,824 INFO org.apache.hadoop.hbase.master.HMaster:
Server active/primary master; 0:0:0:0:0:0:0:0%0,
59736,1343784093521, sessionid=0x138dfc604160000, cluster-up flag was=false
2012-07-31 18:21:39,832 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
globalMemStoreLimit=397.6m, global
MemStoreLimitLowMark=347.9m, maxHeap=994.1m
2012-07-31 18:21:39,847 FATAL org.apache.hadoop.hbase.master.HMaster:
Master server abort: loaded coprocessors are: [
]
2012-07-31 18:21:39,847 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 16mins,
40sec
2012-07-31 18:21:39,848 FATAL org.apache.hadoop.hbase.master.HMaster:
Unhandled exception. Starting shutdown.
java.util.UnknownFormatConversionException: Conversion = '0'
        at java.util.Formatter.checkText(Formatter.java:2503)
        at java.util.Formatter.parse(Formatter.java:2467)
        at java.util.Formatter.format(Formatter.java:2414)
        at java.util.Formatter.format(Formatter.java:2367)
        at java.lang.String.format(String.java:2769)
        at com.google.common.util.concurrent.ThreadFactoryBuilder.setNameFormat(ThreadFactoryBuilder.java:68)
        at org.apache.hadoop.hbase.executor.ExecutorService$Executor.<init>(ExecutorService.java:299)
        at org.apache.hadoop.hbase.executor.ExecutorService.startExecutorService(ExecutorService.java:185)
        at org.apache.hadoop.hbase.executor.ExecutorService.startExecutorService(ExecutorService.java:227)
        at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:821)
        at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:507)
        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:344)
        at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.run(HMasterCommandLine.java:220)
        at java.lang.Thread.run(Thread.java:680)
2012-07-31 18:21:39,908 INFO org.apache.hadoop.hbase.master.HMaster: Aborting

As you can see, the hostname is now "0:0:0:0:0:0:0:0%0".  The code
uses this as the prefix to a string, then that string is passed into a
formatter.  Hence it is confused by %0, and aborts in a weird place
causing the entire thing to crash n burn.

As I see it there are several solutions:
- s/%/_/ for the hostname at some point
- attempt not to use ipv6

the latter seems hacky though, surely someone would want to support ipv6?

-Djava.net.preferIPv4Stack=true ?

Does that still work?

On Tue, Jul 31, 2012 at 6:24 PM, Ryan Rawson <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> Something has changed in how OSX and java handles IPv6, and now you
> will get a log like:
>
> 2012-07-31 18:21:39,824 INFO org.apache.hadoop.hbase.master.HMaster:
> Server active/primary master; 0:0:0:0:0:0:0:0%0,
> 59736,1343784093521, sessionid=0x138dfc604160000, cluster-up flag was=false
> 2012-07-31 18:21:39,832 INFO
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
> globalMemStoreLimit=397.6m, global
> MemStoreLimitLowMark=347.9m, maxHeap=994.1m
> 2012-07-31 18:21:39,847 FATAL org.apache.hadoop.hbase.master.HMaster:
> Master server abort: loaded coprocessors are: [
> ]
> 2012-07-31 18:21:39,847 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 16mins,
> 40sec
> 2012-07-31 18:21:39,848 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unhandled exception. Starting shutdown.
> java.util.UnknownFormatConversionException: Conversion = '0'
>         at java.util.Formatter.checkText(Formatter.java:2503)
>         at java.util.Formatter.parse(Formatter.java:2467)
>         at java.util.Formatter.format(Formatter.java:2414)
>         at java.util.Formatter.format(Formatter.java:2367)
>         at java.lang.String.format(String.java:2769)
>         at com.google.common.util.concurrent.ThreadFactoryBuilder.setNameFormat(ThreadFactoryBuilder.java:68)
>         at org.apache.hadoop.hbase.executor.ExecutorService$Executor.<init>(ExecutorService.java:299)
>         at org.apache.hadoop.hbase.executor.ExecutorService.startExecutorService(ExecutorService.java:185)
>         at org.apache.hadoop.hbase.executor.ExecutorService.startExecutorService(ExecutorService.java:227)
>         at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:821)
>         at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:507)
>         at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:344)
>         at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.run(HMasterCommandLine.java:220)
>         at java.lang.Thread.run(Thread.java:680)
> 2012-07-31 18:21:39,908 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
>
> As you can see, the hostname is now "0:0:0:0:0:0:0:0%0".  The code
> uses this as the prefix to a string, then that string is passed into a
> formatter.  Hence it is confused by %0, and aborts in a weird place
> causing the entire thing to crash n burn.
>
> As I see it there are several solutions:
> - s/%/_/ for the hostname at some point
> - attempt not to use ipv6
>
> the latter seems hacky though, surely someone would want to support ipv6?

--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)

I shall try that.  I submitted a patch too that quashes the extra %
where it is causing problems.

On Tue, Jul 31, 2012 at 6:28 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
> -Djava.net.preferIPv4Stack=true ?
>
> Does that still work?
>
> On Tue, Jul 31, 2012 at 6:24 PM, Ryan Rawson <[EMAIL PROTECTED]> wrote:
>> Hi all,
>>
>> Something has changed in how OSX and java handles IPv6, and now you
>> will get a log like:
>>
>> 2012-07-31 18:21:39,824 INFO org.apache.hadoop.hbase.master.HMaster:
>> Server active/primary master; 0:0:0:0:0:0:0:0%0,
>> 59736,1343784093521, sessionid=0x138dfc604160000, cluster-up flag was=false
>> 2012-07-31 18:21:39,832 INFO
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
>> globalMemStoreLimit=397.6m, global
>> MemStoreLimitLowMark=347.9m, maxHeap=994.1m
>> 2012-07-31 18:21:39,847 FATAL org.apache.hadoop.hbase.master.HMaster:
>> Master server abort: loaded coprocessors are: [
>> ]
>> 2012-07-31 18:21:39,847 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 16mins,
>> 40sec
>> 2012-07-31 18:21:39,848 FATAL org.apache.hadoop.hbase.master.HMaster:
>> Unhandled exception. Starting shutdown.
>> java.util.UnknownFormatConversionException: Conversion = '0'
>>         at java.util.Formatter.checkText(Formatter.java:2503)
>>         at java.util.Formatter.parse(Formatter.java:2467)
>>         at java.util.Formatter.format(Formatter.java:2414)
>>         at java.util.Formatter.format(Formatter.java:2367)
>>         at java.lang.String.format(String.java:2769)
>>         at com.google.common.util.concurrent.ThreadFactoryBuilder.setNameFormat(ThreadFactoryBuilder.java:68)
>>         at org.apache.hadoop.hbase.executor.ExecutorService$Executor.<init>(ExecutorService.java:299)
>>         at org.apache.hadoop.hbase.executor.ExecutorService.startExecutorService(ExecutorService.java:185)
>>         at org.apache.hadoop.hbase.executor.ExecutorService.startExecutorService(ExecutorService.java:227)
>>         at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:821)
>>         at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:507)
>>         at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:344)
>>         at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.run(HMasterCommandLine.java:220)
>>         at java.lang.Thread.run(Thread.java:680)
>> 2012-07-31 18:21:39,908 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
>>
>> As you can see, the hostname is now "0:0:0:0:0:0:0:0%0".  The code
>> uses this as the prefix to a string, then that string is passed into a
>> formatter.  Hence it is confused by %0, and aborts in a weird place
>> causing the entire thing to crash n burn.
>>
>> As I see it there are several solutions:
>> - s/%/_/ for the hostname at some point
>> - attempt not to use ipv6
>>
>> the latter seems hacky though, surely someone would want to support ipv6?
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein (via Tom White)

Did this fix end up working?  I'm hesitant to upgrade to 10.8 if I'm going
to run into this issue.  I'm running the CDH3 jars locally to mirror my
production cluster.

On Tue, Jul 31, 2012 at 9:41 PM, Ryan Rawson <[EMAIL PROTECTED]> wrote:

> I shall try that.  I submitted a patch too that quashes the extra %
> where it is causing problems.
>
>
>
> On Tue, Jul 31, 2012 at 6:28 PM, Andrew Purtell <[EMAIL PROTECTED]>
> wrote:
> > -Djava.net.preferIPv4Stack=true ?
> >
> > Does that still work?
> >
> > On Tue, Jul 31, 2012 at 6:24 PM, Ryan Rawson <[EMAIL PROTECTED]> wrote:
> >> Hi all,
> >>
> >> Something has changed in how OSX and java handles IPv6, and now you
> >> will get a log like:
> >>
> >> 2012-07-31 18:21:39,824 INFO org.apache.hadoop.hbase.master.HMaster:
> >> Server active/primary master; 0:0:0:0:0:0:0:0%0,
> >> 59736,1343784093521, sessionid=0x138dfc604160000, cluster-up flag
> was=false
> >> 2012-07-31 18:21:39,832 INFO
> >> org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
> >> globalMemStoreLimit=397.6m, global
> >> MemStoreLimitLowMark=347.9m, maxHeap=994.1m
> >> 2012-07-31 18:21:39,847 FATAL org.apache.hadoop.hbase.master.HMaster:
> >> Master server abort: loaded coprocessors are: [
> >> ]
> >> 2012-07-31 18:21:39,847 INFO
> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 16mins,
> >> 40sec
> >> 2012-07-31 18:21:39,848 FATAL org.apache.hadoop.hbase.master.HMaster:
> >> Unhandled exception. Starting shutdown.
> >> java.util.UnknownFormatConversionException: Conversion = '0'
> >>         at java.util.Formatter.checkText(Formatter.java:2503)
> >>         at java.util.Formatter.parse(Formatter.java:2467)
> >>         at java.util.Formatter.format(Formatter.java:2414)
> >>         at java.util.Formatter.format(Formatter.java:2367)
> >>         at java.lang.String.format(String.java:2769)
> >>         at
> com.google.common.util.concurrent.ThreadFactoryBuilder.setNameFormat(ThreadFactoryBuilder.java:68)
> >>         at
> org.apache.hadoop.hbase.executor.ExecutorService$Executor.<init>(ExecutorService.java:299)
> >>         at
> org.apache.hadoop.hbase.executor.ExecutorService.startExecutorService(ExecutorService.java:185)
> >>         at
> org.apache.hadoop.hbase.executor.ExecutorService.startExecutorService(ExecutorService.java:227)
> >>         at
> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:821)
> >>         at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:507)
> >>         at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:344)
> >>         at
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.run(HMasterCommandLine.java:220)
> >>         at java.lang.Thread.run(Thread.java:680)
> >> 2012-07-31 18:21:39,908 INFO org.apache.hadoop.hbase.master.HMaster:
> Aborting
> >>
> >> As you can see, the hostname is now "0:0:0:0:0:0:0:0%0".  The code
> >> uses this as the prefix to a string, then that string is passed into a
> >> formatter.  Hence it is confused by %0, and aborts in a weird place
> >> causing the entire thing to crash n burn.
> >>
> >> As I see it there are several solutions:
> >> - s/%/_/ for the hostname at some point
> >> - attempt not to use ipv6
> >>
> >> the latter seems hacky though, surely someone would want to support
> ipv6?
> >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein (via Tom White)
>

Bryan,

I believe running with "-Djava.net.preferIPv4Stack=true" should work just fine.

On Thu, Aug 9, 2012 at 1:17 AM, Bryan Beaudreault
<[EMAIL PROTECTED]> wrote:
> Did this fix end up working?  I'm hesitant to upgrade to 10.8 if I'm going
> to run into this issue.  I'm running the CDH3 jars locally to mirror my
> production cluster.
>
> On Tue, Jul 31, 2012 at 9:41 PM, Ryan Rawson <[EMAIL PROTECTED]> wrote:
>
>> I shall try that.  I submitted a patch too that quashes the extra %
>> where it is causing problems.
>>
>>
>>
>> On Tue, Jul 31, 2012 at 6:28 PM, Andrew Purtell <[EMAIL PROTECTED]>
>> wrote:
>> > -Djava.net.preferIPv4Stack=true ?
>> >
>> > Does that still work?
>> >
>> > On Tue, Jul 31, 2012 at 6:24 PM, Ryan Rawson <[EMAIL PROTECTED]> wrote:
>> >> Hi all,
>> >>
>> >> Something has changed in how OSX and java handles IPv6, and now you
>> >> will get a log like:
>> >>
>> >> 2012-07-31 18:21:39,824 INFO org.apache.hadoop.hbase.master.HMaster:
>> >> Server active/primary master; 0:0:0:0:0:0:0:0%0,
>> >> 59736,1343784093521, sessionid=0x138dfc604160000, cluster-up flag
>> was=false
>> >> 2012-07-31 18:21:39,832 INFO
>> >> org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
>> >> globalMemStoreLimit=397.6m, global
>> >> MemStoreLimitLowMark=347.9m, maxHeap=994.1m
>> >> 2012-07-31 18:21:39,847 FATAL org.apache.hadoop.hbase.master.HMaster:
>> >> Master server abort: loaded coprocessors are: [
>> >> ]
>> >> 2012-07-31 18:21:39,847 INFO
>> >> org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 16mins,
>> >> 40sec
>> >> 2012-07-31 18:21:39,848 FATAL org.apache.hadoop.hbase.master.HMaster:
>> >> Unhandled exception. Starting shutdown.
>> >> java.util.UnknownFormatConversionException: Conversion = '0'
>> >>         at java.util.Formatter.checkText(Formatter.java:2503)
>> >>         at java.util.Formatter.parse(Formatter.java:2467)
>> >>         at java.util.Formatter.format(Formatter.java:2414)
>> >>         at java.util.Formatter.format(Formatter.java:2367)
>> >>         at java.lang.String.format(String.java:2769)
>> >>         at
>> com.google.common.util.concurrent.ThreadFactoryBuilder.setNameFormat(ThreadFactoryBuilder.java:68)
>> >>         at
>> org.apache.hadoop.hbase.executor.ExecutorService$Executor.<init>(ExecutorService.java:299)
>> >>         at
>> org.apache.hadoop.hbase.executor.ExecutorService.startExecutorService(ExecutorService.java:185)
>> >>         at
>> org.apache.hadoop.hbase.executor.ExecutorService.startExecutorService(ExecutorService.java:227)
>> >>         at
>> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:821)
>> >>         at
>> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:507)
>> >>         at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:344)
>> >>         at
>> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.run(HMasterCommandLine.java:220)
>> >>         at java.lang.Thread.run(Thread.java:680)
>> >> 2012-07-31 18:21:39,908 INFO org.apache.hadoop.hbase.master.HMaster:
>> Aborting
>> >>
>> >> As you can see, the hostname is now "0:0:0:0:0:0:0:0%0".  The code
>> >> uses this as the prefix to a string, then that string is passed into a
>> >> formatter.  Hence it is confused by %0, and aborts in a weird place
>> >> causing the entire thing to crash n burn.
>> >>
>> >> As I see it there are several solutions:
>> >> - s/%/_/ for the hostname at some point
>> >> - attempt not to use ipv6
>> >>
>> >> the latter seems hacky though, surely someone would want to support
>> ipv6?
>> >
>> >
>> >
>> > --
>> > Best regards,
>> >
>> >    - Andy
>> >
>> > Problems worthy of attack prove their worth by hitting back. - Piet
>> > Hein (via Tom White)
>>

--
Harsh J

On Sun, Aug 12, 2012 at 3:57 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> Bryan,
>
> I believe running with "-Djava.net.preferIPv4Stack=true" should work just fine.
>

I can add to the ref guide a macosx section if above works for you Bryan.
St.Ack