Flume, mail # user - flume 0.9.4 windows version out of memory

flume 0.9.4 windows version out of memory

周梦想 2013-01-25, 07:42

when i start the flume node service,the agent is OPENING.
Hello,

I need to gather logs from windows 7, but the flume master and collector is
running on linux. I download 0.9.4 exe file from flume official web.

When I start the node from windows, the status is OPENING.

zhouhh.TKOffice.localzhouhh.TKOffice.local zhouhh.TKOffice.localOPENINGFri
Jan 25 11:45:01 CST 2013 4

And the log reports OutOfMemoryError.

===============D:\Program Files\Cloudera\Flume 0.9.4\log\flumenode-stdout.2013-01-25.log

2013-01-25 15:31:03 Commons Daemon procrun stdout initializedListening for
transport dt_socket at address: 8888
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid8100.hprof ...
Heap dump file created [13270531 bytes in 0.115 secs]

==============D:\Program Files\Cloudera\Flume 0.9.4\log\flumenode-stderr.2013-01-25.log

2013-01-25 15:31:03 Commons Daemon procrun stderr initialized
13/01/25 15:31:03 INFO agent.FlumeNode: Flume 0.9.4
13/01/25 15:31:03 INFO agent.FlumeNode: rev
8d0aa4be4ee50bb664cb0d1624e5634f46d7b62a
13/01/25 15:31:03 INFO agent.FlumeNode: Compiled on Mon Jun 20 14:46:53
PDT 2011
13/01/25 15:31:03 INFO agent.FlumeNode: System property
java.runtime.name=Java(TM)
SE Runtime Environment
13/01/25 15:31:03 INFO agent.FlumeNode: System property
sun.boot.library.path=C:\PROGRA~1\Java\jre7\bin
13/01/25 15:31:03 INFO agent.FlumeNode: System property
java.vm.version=23.3-b01
13/01/25 15:31:03 INFO agent.FlumeNode: System property
java.vm.vendor=Oracle Corporation
13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vendor.urlhttp://java.oracle.com/
13/01/25 15:31:03 INFO agent.FlumeNode: System property path.separator=;
13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vm.name=Java
HotSpot(TM) Client VM
13/01/25 15:31:03 INFO agent.FlumeNode: System property file.encoding.pkgsun.io
13/01/25 15:31:03 INFO agent.FlumeNode: System property user.country=CN
13/01/25 15:31:03 INFO agent.FlumeNode: System property user.script13/01/25 15:31:03 INFO agent.FlumeNode: System property
sun.os.patch.level=Service Pack 1
13/01/25 15:31:03 INFO agent.FlumeNode: System property
java.vm.specification.name=Java Virtual Machine Specification
13/01/25 15:31:03 INFO agent.FlumeNode: System property
user.dir=C:\Windows\system32
13/01/25 15:31:03 INFO agent.FlumeNode: System property
java.runtime.version=1.7.0_07-b11
13/01/25 15:31:03 INFO agent.FlumeNode: System property flume13/01/25 15:31:03 INFO agent.FlumeNode: System property
java.awt.graphicsenv=sun.awt.Win32GraphicsEnvironment
13/01/25 15:31:03 INFO agent.FlumeNode: System property
java.endorsed.dirs=C:\PROGRA~1\Java\jre7\lib\endorsed
13/01/25 15:31:03 INFO agent.FlumeNode: System property os.arch=x86
13/01/25 15:31:03 INFO agent.FlumeNode: System property
java.io.tmpdir=C:\Windows\TEMP\
13/01/25 15:31:03 INFO agent.FlumeNode: System property line.separator
13/01/25 15:31:03 INFO agent.FlumeNode: System property
java.vm.specification.vendor=Oracle Corporation
13/01/25 15:31:03 INFO agent.FlumeNode: System property user.variant13/01/25 15:31:03 INFO agent.FlumeNode: System property os.name=Windows 7
13/01/25 15:31:03 INFO agent.FlumeNode: System property
flume.home=d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..
13/01/25 15:31:03 INFO agent.FlumeNode: System property sun.jnu.encoding=GBK
13/01/25 15:31:03 INFO agent.FlumeNode: System property
java.library.path=d:\PROGRA~1\Cloudera\FLUME0~1.4\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program
Files\Common
Files\NetSarang;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program
Files\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files\Microsoft SQL
Server\100\DTS\Binn\;C:\Program Files\Common Files\Thunder
Network\KanKan\Codecs;d:\php;d:\php\ext;D:\Program Files\MySQL\MySQL Server
5.5\bin;D:\mysql\bin;D:\wxWidgets-2.9.1\include;D:\wxWidgets-2.9.1\include\msvc;W:\Server\include;W:\Base\include;W:\Client\include;d:\Program
Files\Lua\5.1;d:\Program Files\Lua\5.1\clibs;D:\Program
Files\TortoiseSVN\bin;d:\Program Files\Git\cmd;d:\Program
Files\Git\bin;D:\Program Files\TortoiseGit\bin;D:\docbook\bin;;.
13/01/25 15:31:03 INFO agent.FlumeNode: System property
java.specification.name=Java Platform API Specification
13/01/25 15:31:03 INFO agent.FlumeNode: System property
java.class.version=51.0
13/01/25 15:31:03 INFO agent.FlumeNode: System property
sun.management.compiler=HotSpot Client Compiler
13/01/25 15:31:03 INFO agent.FlumeNode: System property os.version=6.1
13/01/25 15:31:03 INFO agent.FlumeNode: System property user.home=C:\
13/01/25 15:31:03 INFO agent.FlumeNode: System property
user.timezone=Asia/Shanghai
13/01/25 15:31:03 INFO agent.FlumeNode: System property
java.awt.printerjob=sun.awt.windows.WPrinterJob
13/01/25 15:31:03 INFO agent.FlumeNode: System property file.encoding=GBK
13/01/25 15:31:03 INFO agent.FlumeNode: System property
java.specification.version=1.7
13/01/25 15:31:03 INFO agent.FlumeNode: System property
java.class.path=d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\build\classes;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\activation-1.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\ahocorasick-2.x.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\ant-1.6.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\antlr-2.7.7.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\antlr-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\antlr-runtime-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\asm-3.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\automaton-1.0.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-1.5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-compiler-1.5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-ipc-1.5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-cli-1.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-codec-1.3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-collections-3.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-el-1.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-httpclient-3.1.jar;d:\PROGRA~1\

Re: flume 0.9.4 windows version out of memory

周梦想 2013-01-25, 08:34

I found that if I change windows agent configure
fromtail("c:\\test.txt") agentSink("hadoop48",35853)
;to
tail("c:\\test.txt")agentDFOSink("hadoop48",35853) ;
then the agent changes status from OPENING to ACTIVE.
my windows agent is :192.168.20.81, while linux collector is : 192.168.10.48
another linux agent is 192.168.10.46. 10.46 have no problem of agentSink.

Thanks!
Andy.

2013/1/25 周梦想 <[EMAIL PROTECTED]>

> when i start the flume node service,the agent is OPENING.
> Hello,
>
> I need to gather logs from windows 7, but the flume master and collector
> is running on linux. I download 0.9.4 exe file from flume official web.
>
> When I start the node from windows, the status is OPENING.
>
> zhouhh.TKOffice.localzhouhh.TKOffice.local zhouhh.TKOffice.localOPENINGFri
> Jan 25 11:45:01 CST 2013 4
>
> And the log reports OutOfMemoryError.
>
> ===============> D:\Program Files\Cloudera\Flume 0.9.4\log\flumenode-stdout.2013-01-25.log
>
> 2013-01-25 15:31:03 Commons Daemon procrun stdout initializedListening for
> transport dt_socket at address: 8888
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid8100.hprof ...
> Heap dump file created [13270531 bytes in 0.115 secs]
>
> ==============> D:\Program Files\Cloudera\Flume 0.9.4\log\flumenode-stderr.2013-01-25.log
>
> 2013-01-25 15:31:03 Commons Daemon procrun stderr initialized
> 13/01/25 15:31:03 INFO agent.FlumeNode: Flume 0.9.4
> 13/01/25 15:31:03 INFO agent.FlumeNode: rev
> 8d0aa4be4ee50bb664cb0d1624e5634f46d7b62a
> 13/01/25 15:31:03 INFO agent.FlumeNode: Compiled on Mon Jun 20 14:46:53
> PDT 2011
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property java.runtime.name=Java(TM)
> SE Runtime Environment
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> sun.boot.library.path=C:\PROGRA~1\Java\jre7\bin
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.vm.version=23.3-b01
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.vm.vendor=Oracle Corporation
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vendor.url> http://java.oracle.com/
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property path.separator=;
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vm.name=Java
> HotSpot(TM) Client VM
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property file.encoding.pkg> sun.io
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.country=CN
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.script> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> sun.os.patch.level=Service Pack 1
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.vm.specification.name=Java Virtual Machine Specification
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> user.dir=C:\Windows\system32
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.runtime.version=1.7.0_07-b11
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property flume> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.awt.graphicsenv=sun.awt.Win32GraphicsEnvironment
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.endorsed.dirs=C:\PROGRA~1\Java\jre7\lib\endorsed
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property os.arch=x86
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.io.tmpdir=C:\Windows\TEMP\
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property line.separator>
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.vm.specification.vendor=Oracle Corporation
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.variant> 13/01/25 15:31:03 INFO agent.FlumeNode: System property os.name=Windows 7
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> flume.home=d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> sun.jnu.encoding=GBK
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.library.path=d:\PROGRA~1\Cloudera\FLUME0~1.4\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program

Re: flume 0.9.4 windows version out of memory

Jeong-shik Jang 2013-01-25, 09:21

Hi Andy,

Seeing your log, out of memory error had occurred during opening stage
of the logical node, and so was the last status of the logical node
reported to master; no more status was reported to master due to the
error since then, so it stays as OPENNING in master.
Your error seems to be related to ack rolling because agentSink is
agentE2ESink; agentDFO is not related to ack but still WAL for disk
failover, though.

I remember you can change java memory option in flume procrun tray
(flumenodew.exe) icon menu; one looks just like Apache web server tray icon.
You need to restart(stop and start) service through the tray menu once
you make the change.

-JS

On 1/25/13 5:34 PM, 周梦想 wrote:
> I found that if I change windows agent configure from
> tail("c:\\test.txt") agentSink("hadoop48",35853) ;
>
> to
> tail("c:\\test.txt") agentDFOSink("hadoop48",35853) ;
>
>
> then the agent changes status from OPENING to ACTIVE.
> my windows agent is :192.168.20.81, while linux collector is :
> 192.168.10.48
> another linux agent is 192.168.10.46. 10.46 have no problem of agentSink.
>
> Thanks!
> Andy.
>
> 2013/1/25 周梦想 <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
>
> when i start the flume node service,the agent is OPENING.
> Hello,
>
> I need to gather logs from windows 7, but the flume master and
> collector is running on linux. I download 0.9.4 exe file from
> flume official web.
>
> When I start the node from windows, the status is OPENING.
>
> zhouhh.TKOffice.local zhouhh.TKOffice.local
> zhouhh.TKOffice.local OPENING Fri Jan 25 11:45:01 CST 2013 4
>
>
> And the log reports OutOfMemoryError.
>
> ===============> D:\Program Files\Cloudera\Flume
> 0.9.4\log\flumenode-stdout.2013-01-25.log
>
> 2013-01-25 15:31:03 Commons Daemon procrun stdout
> initializedListening for transport dt_socket at address: 8888
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid8100.hprof ...
> Heap dump file created [13270531 bytes in 0.115 secs]
>
> ==============> D:\Program Files\Cloudera\Flume
> 0.9.4\log\flumenode-stderr.2013-01-25.log
>
> 2013-01-25 15:31:03 Commons Daemon procrun stderr initialized
> 13/01/25 15:31:03 INFO agent.FlumeNode: Flume 0.9.4
> 13/01/25 15:31:03 INFO agent.FlumeNode: rev
> 8d0aa4be4ee50bb664cb0d1624e5634f46d7b62a
> 13/01/25 15:31:03 INFO agent.FlumeNode: Compiled on Mon Jun 20
> 14:46:53 PDT 2011
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.runtime.name <http://java.runtime.name>=Java(TM) SE Runtime
> Environment
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> sun.boot.library.path=C:\PROGRA~1\Java\jre7\bin
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.vm.version=23.3-b01
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.vm.vendor=Oracle Corporation
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.vendor.url=http://java.oracle.com/
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> path.separator=;
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.vm.name <http://java.vm.name>=Java HotSpot(TM) Client VM
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> file.encoding.pkg=sun.io <http://sun.io>
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> user.country=CN
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.script> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> sun.os.patch.level=Service Pack 1
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.vm.specification.name
> <http://java.vm.specification.name>=Java Virtual Machine Specification
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> user.dir=C:\Windows\system32
> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
> java.runtime.version=1.7.0_07-b11
Jeong-shik Jang / [EMAIL PROTECTED]
Gruter, Inc., R&D Team Leader
www.gruter.com
Enjoy Connecting

Re: flume 0.9.4 windows version out of memory

周梦想 2013-01-28, 04:05

Thank you, Jeong-shik!
I also consider that the OPENNING status related to ack rolling. Because it
happened on Windows, not on Linux although I closed the firewall of
windows. But I don't know why, maybe there are some anti-virus software do
some thing to stop ack.

The flume node tray icon can't modify the parameter of java VM. It's read
only.

Best regards!
Andy

2013/1/25 Jeong-shik Jang <[EMAIL PROTECTED]>

> Hi Andy,
>
> Seeing your log, out of memory error had occurred during opening stage of
> the logical node, and so was the last status of the logical node reported
> to master; no more status was reported to master due to the error since
> then, so it stays as OPENNING in master.
> Your error seems to be related to ack rolling because agentSink is
> agentE2ESink; agentDFO is not related to ack but still WAL for disk
> failover, though.
>
> I remember you can change java memory option in flume procrun tray
> (flumenodew.exe) icon menu; one looks just like Apache web server tray icon.
> You need to restart(stop and start) service through the tray menu once you
> make the change.
>
> -JS
>
>
> On 1/25/13 5:34 PM, 周梦想 wrote:
>
> I found that if I change windows agent configure from
> tail("c:\\test.txt") agentSink("hadoop48",35853) ; to
> tail("c:\\test.txt") agentDFOSink("hadoop48",35853) ;
> then the agent changes status from OPENING to ACTIVE.
> my windows agent is :192.168.20.81, while linux collector is :
> 192.168.10.48
> another linux agent is 192.168.10.46. 10.46 have no problem of agentSink.
>
> Thanks!
> Andy.
>
> 2013/1/25 周梦想 <[EMAIL PROTECTED]>
>
>> when i start the flume node service,the agent is OPENING.
>> Hello,
>>
>> I need to gather logs from windows 7, but the flume master and
>> collector is running on linux. I download 0.9.4 exe file from flume
>> official web.
>>
>> When I start the node from windows, the status is OPENING.
>>
>> zhouhh.TKOffice.local zhouhh.TKOffice.local zhouhh.TKOffice.local
>> OPENING Fri Jan 25 11:45:01 CST 2013 4
>>
>> And the log reports OutOfMemoryError.
>>
>> ===============>> D:\Program Files\Cloudera\Flume 0.9.4\log\flumenode-stdout.2013-01-25.log
>>
>> 2013-01-25 15:31:03 Commons Daemon procrun stdout initializedListening
>> for transport dt_socket at address: 8888
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid8100.hprof ...
>> Heap dump file created [13270531 bytes in 0.115 secs]
>>
>> ==============>> D:\Program Files\Cloudera\Flume 0.9.4\log\flumenode-stderr.2013-01-25.log
>>
>> 2013-01-25 15:31:03 Commons Daemon procrun stderr initialized
>> 13/01/25 15:31:03 INFO agent.FlumeNode: Flume 0.9.4
>> 13/01/25 15:31:03 INFO agent.FlumeNode: rev
>> 8d0aa4be4ee50bb664cb0d1624e5634f46d7b62a
>> 13/01/25 15:31:03 INFO agent.FlumeNode: Compiled on Mon Jun 20 14:46:53
>> PDT 2011
>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property java.runtime.name=Java(TM)
>> SE Runtime Environment
>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
>> sun.boot.library.path=C:\PROGRA~1\Java\jre7\bin
>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
>> java.vm.version=23.3-b01
>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
>> java.vm.vendor=Oracle Corporation
>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vendor.url>> http://java.oracle.com/
>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property path.separator=;
>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property java.vm.name=Java
>> HotSpot(TM) Client VM
>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property file.encoding.pkg>> sun.io
>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.country=CN
>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property user.script>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
>> sun.os.patch.level=Service Pack 1
>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
>> java.vm.specification.name=Java Virtual Machine Specification
>> 13/01/25 15:31:03 INFO agent.FlumeNode

Re: flume 0.9.4 windows version out of memory

Jeong-shik Jang 2013-01-28, 06:49

How about editing registry using regedit?

1. locate memory option parameters in registry:
HKEY_LOCAL_MACHINE > SOFTWARE > Apache Software Foundation > Procrun 2.0
> FlumeNode > Parameters > Java

2. Increase values in JvmMs and JvmMx

3. Stop/Start FlumeNode Service via Flume tray icon menu.

-JS

On 1/28/13 1:05 PM, 周梦想 wrote:
> Thank you, Jeong-shik!
> I also consider that the OPENNING status related to ack rolling.
> Because it happened on Windows, not on Linux although I closed the
> firewall of windows. But I don't know why, maybe there are some
> anti-virus software do some thing to stop ack.
>
> The flume node tray icon can't modify the parameter of java VM. It's
> read only.
>
> Best regards!
> Andy
>
> 2013/1/25 Jeong-shik Jang <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
>
> Hi Andy,
>
> Seeing your log, out of memory error had occurred during opening
> stage of the logical node, and so was the last status of the
> logical node reported to master; no more status was reported to
> master due to the error since then, so it stays as OPENNING in master.
> Your error seems to be related to ack rolling because agentSink is
> agentE2ESink; agentDFO is not related to ack but still WAL for
> disk failover, though.
>
> I remember you can change java memory option in flume procrun tray
> (flumenodew.exe) icon menu; one looks just like Apache web server
> tray icon.
> You need to restart(stop and start) service through the tray menu
> once you make the change.
>
> -JS
>
>
> On 1/25/13 5:34 PM, 周梦想 wrote:
>> I found that if I change windows agent configure from
>> tail("c:\\test.txt") agentSink("hadoop48",35853) ;
>>
>> to
>> tail("c:\\test.txt") agentDFOSink("hadoop48",35853) ;
>>
>>
>> then the agent changes status from OPENING to ACTIVE.
>> my windows agent is :192.168.20.81, while linux collector is :
>> 192.168.10.48
>> another linux agent is 192.168.10.46. 10.46 have no problem of
>> agentSink.
>>
>> Thanks!
>> Andy.
>>
>> 2013/1/25 周梦想 <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
>>
>> when i start the flume node service,the agent is OPENING.
>> Hello,
>>
>> I need to gather logs from windows 7, but the flume master
>> and collector is running on linux. I download 0.9.4 exe file
>> from flume official web.
>>
>> When I start the node from windows, the status is OPENING.
>>
>> zhouhh.TKOffice.local zhouhh.TKOffice.local
>> zhouhh.TKOffice.local OPENING Fri Jan 25 11:45:01 CST 2013 4
>>
>>
>> And the log reports OutOfMemoryError.
>>
>> ===============>> D:\Program Files\Cloudera\Flume
>> 0.9.4\log\flumenode-stdout.2013-01-25.log
>>
>> 2013-01-25 15:31:03 Commons Daemon procrun stdout
>> initializedListening for transport dt_socket at address: 8888
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid8100.hprof ...
>> Heap dump file created [13270531 bytes in 0.115 secs]
>>
>> ==============>> D:\Program Files\Cloudera\Flume
>> 0.9.4\log\flumenode-stderr.2013-01-25.log
>>
>> 2013-01-25 15:31:03 Commons Daemon procrun stderr initialized
>> 13/01/25 15:31:03 INFO agent.FlumeNode: Flume 0.9.4
>> 13/01/25 15:31:03 INFO agent.FlumeNode: rev
>> 8d0aa4be4ee50bb664cb0d1624e5634f46d7b62a
>> 13/01/25 15:31:03 INFO agent.FlumeNode: Compiled on Mon Jun
>> 20 14:46:53 PDT 2011
>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
>> java.runtime.name <http://java.runtime.name>=Java(TM) SE
>> Runtime Environment
>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
>> sun.boot.library.path=C:\PROGRA~1\Java\jre7\bin
>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
>> java.vm.version=23.3-b01
>> 13/01/25 15

Jeong-shik Jang / [EMAIL PROTECTED]
Gruter, Inc., R&D Team Leader
www.gruter.com
Enjoy Connecting

Re: flume 0.9.4 windows version out of memory

Alexander Alten-Lorenz 2013-01-28, 07:07

+1

Thats the easiest option on Windows, unfortunately.

- Alex

On Jan 28, 2013, at 7:49 AM, Jeong-shik Jang <[EMAIL PROTECTED]> wrote:

> How about editing registry using regedit?
>
> 1. locate memory option parameters in registry:
> HKEY_LOCAL_MACHINE > SOFTWARE > Apache Software Foundation > Procrun 2.0 > FlumeNode > Parameters > Java
>
> 2. Increase values in JvmMs and JvmMx
>
> 3. Stop/Start FlumeNode Service via Flume tray icon menu.
>
> -JS
>
> On 1/28/13 1:05 PM, 周梦想 wrote:
>> Thank you, Jeong-shik!
>> I also consider that the OPENNING status related to ack rolling. Because it happened on Windows, not on Linux although I closed the firewall of windows. But I don't know why, maybe there are some anti-virus software do some thing to stop ack.
>>
>> The flume node tray icon can't modify the parameter of java VM. It's read only.
>>
>> Best regards!
>> Andy
>>
>> 2013/1/25 Jeong-shik Jang <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
>>
>> Hi Andy,
>>
>> Seeing your log, out of memory error had occurred during opening
>> stage of the logical node, and so was the last status of the
>> logical node reported to master; no more status was reported to
>> master due to the error since then, so it stays as OPENNING in master.
>> Your error seems to be related to ack rolling because agentSink is
>> agentE2ESink; agentDFO is not related to ack but still WAL for
>> disk failover, though.
>>
>> I remember you can change java memory option in flume procrun tray
>> (flumenodew.exe) icon menu; one looks just like Apache web server
>> tray icon.
>> You need to restart(stop and start) service through the tray menu
>> once you make the change.
>>
>> -JS
>>
>>
>> On 1/25/13 5:34 PM, 周梦想 wrote:
>>> I found that if I change windows agent configure from
>>> tail("c:\\test.txt") agentSink("hadoop48",35853) ;
>>>
>>> to
>>> tail("c:\\test.txt") agentDFOSink("hadoop48",35853) ;
>>>
>>>
>>> then the agent changes status from OPENING to ACTIVE.
>>> my windows agent is :192.168.20.81, while linux collector is :
>>> 192.168.10.48
>>> another linux agent is 192.168.10.46. 10.46 have no problem of
>>> agentSink.
>>>
>>> Thanks!
>>> Andy.
>>>
>>> 2013/1/25 周梦想 <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>
>>>
>>> when i start the flume node service,the agent is OPENING.
>>> Hello,
>>>
>>> I need to gather logs from windows 7, but the flume master
>>> and collector is running on linux. I download 0.9.4 exe file
>>> from flume official web.
>>>
>>> When I start the node from windows, the status is OPENING.
>>>
>>> zhouhh.TKOffice.local zhouhh.TKOffice.local
>>> zhouhh.TKOffice.local OPENING Fri Jan 25 11:45:01 CST 2013 4
>>>
>>>
>>> And the log reports OutOfMemoryError.
>>>
>>> ===============>>> D:\Program Files\Cloudera\Flume
>>> 0.9.4\log\flumenode-stdout.2013-01-25.log
>>>
>>> 2013-01-25 15:31:03 Commons Daemon procrun stdout
>>> initializedListening for transport dt_socket at address: 8888
>>> java.lang.OutOfMemoryError: Java heap space
>>> Dumping heap to java_pid8100.hprof ...
>>> Heap dump file created [13270531 bytes in 0.115 secs]
>>>
>>> ==============>>> D:\Program Files\Cloudera\Flume
>>> 0.9.4\log\flumenode-stderr.2013-01-25.log
>>>
>>> 2013-01-25 15:31:03 Commons Daemon procrun stderr initialized
>>> 13/01/25 15:31:03 INFO agent.FlumeNode: Flume 0.9.4
>>> 13/01/25 15:31:03 INFO agent.FlumeNode: rev
>>> 8d0aa4be4ee50bb664cb0d1624e5634f46d7b62a
>>> 13/01/25 15:31:03 INFO agent.FlumeNode: Compiled on Mon Jun
>>> 20 14:46:53 PDT 2011
>>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>> java.runtime.name <http://java.runtime.name>=Java(TM) SE
>>> Runtime Environment
>>> 13/01/25 15:31:03 INFO agent.FlumeNode
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: flume 0.9.4 windows version out of memory

周梦想 2013-01-28, 07:22

Sorry, I found in the flume0.9.4 tray icon menu I could edit the parameter,
and edit registry is also ok.
I add two parameter:
-XX:MaxDirectMemorySize=500m
-XX:MaxNewSize=500m
and change JvmMs to 500. It did not report out of memory again.

Thank you, Jeong-shik!
Andy

2013/1/28 Jeong-shik Jang <[EMAIL PROTECTED]>

> How about editing registry using regedit?
>
> 1. locate memory option parameters in registry:
> HKEY_LOCAL_MACHINE > SOFTWARE > Apache Software Foundation > Procrun 2.0 >
> FlumeNode > Parameters > Java
>
> 2. Increase values in JvmMs and JvmMx
>
> 3. Stop/Start FlumeNode Service via Flume tray icon menu.
>
> -JS
>
>
> On 1/28/13 1:05 PM, 周梦想 wrote:
>
> Thank you, Jeong-shik!
> I also consider that the OPENNING status related to ack rolling. Because
> it happened on Windows, not on Linux although I closed the firewall of
> windows. But I don't know why, maybe there are some anti-virus software do
> some thing to stop ack.
>
> The flume node tray icon can't modify the parameter of java VM. It's
> read only.
>
> Best regards!
> Andy
>
> 2013/1/25 Jeong-shik Jang <[EMAIL PROTECTED]>
>
>> Hi Andy,
>>
>> Seeing your log, out of memory error had occurred during opening stage of
>> the logical node, and so was the last status of the logical node reported
>> to master; no more status was reported to master due to the error since
>> then, so it stays as OPENNING in master.
>> Your error seems to be related to ack rolling because agentSink is
>> agentE2ESink; agentDFO is not related to ack but still WAL for disk
>> failover, though.
>>
>> I remember you can change java memory option in flume procrun tray
>> (flumenodew.exe) icon menu; one looks just like Apache web server tray icon.
>> You need to restart(stop and start) service through the tray menu once
>> you make the change.
>>
>> -JS
>>
>>
>> On 1/25/13 5:34 PM, 周梦想 wrote:
>>
>> I found that if I change windows agent configure from
>> tail("c:\\test.txt") agentSink("hadoop48",35853) ; to
>> tail("c:\\test.txt") agentDFOSink("hadoop48",35853) ;
>> then the agent changes status from OPENING to ACTIVE.
>> my windows agent is :192.168.20.81, while linux collector is :
>> 192.168.10.48
>> another linux agent is 192.168.10.46. 10.46 have no problem of
>> agentSink.
>>
>> Thanks!
>> Andy.
>>
>> 2013/1/25 周梦想 <[EMAIL PROTECTED]>
>>
>>> when i start the flume node service,the agent is OPENING.
>>> Hello,
>>>
>>> I need to gather logs from windows 7, but the flume master and
>>> collector is running on linux. I download 0.9.4 exe file from flume
>>> official web.
>>>
>>> When I start the node from windows, the status is OPENING.
>>>
>>> zhouhh.TKOffice.local zhouhh.TKOffice.local zhouhh.TKOffice.local
>>> OPENING Fri Jan 25 11:45:01 CST 2013 4
>>>
>>> And the log reports OutOfMemoryError.
>>>
>>> ===============>>> D:\Program Files\Cloudera\Flume 0.9.4\log\flumenode-stdout.2013-01-25.log
>>>
>>> 2013-01-25 15:31:03 Commons Daemon procrun stdout initializedListening
>>> for transport dt_socket at address: 8888
>>> java.lang.OutOfMemoryError: Java heap space
>>> Dumping heap to java_pid8100.hprof ...
>>> Heap dump file created [13270531 bytes in 0.115 secs]
>>>
>>> ==============>>> D:\Program Files\Cloudera\Flume 0.9.4\log\flumenode-stderr.2013-01-25.log
>>>
>>> 2013-01-25 15:31:03 Commons Daemon procrun stderr initialized
>>> 13/01/25 15:31:03 INFO agent.FlumeNode: Flume 0.9.4
>>> 13/01/25 15:31:03 INFO agent.FlumeNode: rev
>>> 8d0aa4be4ee50bb664cb0d1624e5634f46d7b62a
>>> 13/01/25 15:31:03 INFO agent.FlumeNode: Compiled on Mon Jun 20
>>> 14:46:53 PDT 2011
>>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>> java.runtime.name=Java(TM) SE Runtime Environment
>>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>> sun.boot.library.path=C:\PROGRA~1\Java\jre7\bin
>>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>> java.vm.version=23.3-b01
>>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>> java.vm.vendor=Oracle Corporation

Re: flume 0.9.4 windows version out of memory

周梦想 2013-01-28, 07:41

It report out of memory again after I change configure from agentDFOSink
to agentSink. and the state of flume agent node change to OPENNING again. :(

2013/1/28 周梦想 <[EMAIL PROTECTED]>

> Sorry, I found in the flume0.9.4 tray icon menu I could edit the
> parameter, and edit registry is also ok.
> I add two parameter:
> -XX:MaxDirectMemorySize=500m
> -XX:MaxNewSize=500m
> and change JvmMs to 500. It did not report out of memory again.
>
> Thank you, Jeong-shik!
> Andy
>
> 2013/1/28 Jeong-shik Jang <[EMAIL PROTECTED]>
>
>> How about editing registry using regedit?
>>
>> 1. locate memory option parameters in registry:
>> HKEY_LOCAL_MACHINE > SOFTWARE > Apache Software Foundation > Procrun 2.0
>> > FlumeNode > Parameters > Java
>>
>> 2. Increase values in JvmMs and JvmMx
>>
>> 3. Stop/Start FlumeNode Service via Flume tray icon menu.
>>
>> -JS
>>
>>
>> On 1/28/13 1:05 PM, 周梦想 wrote:
>>
>> Thank you, Jeong-shik!
>> I also consider that the OPENNING status related to ack rolling. Because
>> it happened on Windows, not on Linux although I closed the firewall of
>> windows. But I don't know why, maybe there are some anti-virus software do
>> some thing to stop ack.
>>
>> The flume node tray icon can't modify the parameter of java VM. It's
>> read only.
>>
>> Best regards!
>> Andy
>>
>> 2013/1/25 Jeong-shik Jang <[EMAIL PROTECTED]>
>>
>>> Hi Andy,
>>>
>>> Seeing your log, out of memory error had occurred during opening stage
>>> of the logical node, and so was the last status of the logical node
>>> reported to master; no more status was reported to master due to the error
>>> since then, so it stays as OPENNING in master.
>>> Your error seems to be related to ack rolling because agentSink is
>>> agentE2ESink; agentDFO is not related to ack but still WAL for disk
>>> failover, though.
>>>
>>> I remember you can change java memory option in flume procrun tray
>>> (flumenodew.exe) icon menu; one looks just like Apache web server tray icon.
>>> You need to restart(stop and start) service through the tray menu once
>>> you make the change.
>>>
>>> -JS
>>>
>>>
>>> On 1/25/13 5:34 PM, 周梦想 wrote:
>>>
>>> I found that if I change windows agent configure from
>>> tail("c:\\test.txt") agentSink("hadoop48",35853) ; to
>>> tail("c:\\test.txt") agentDFOSink("hadoop48",35853) ;
>>> then the agent changes status from OPENING to ACTIVE.
>>> my windows agent is :192.168.20.81, while linux collector is :
>>> 192.168.10.48
>>> another linux agent is 192.168.10.46. 10.46 have no problem of
>>> agentSink.
>>>
>>> Thanks!
>>> Andy.
>>>
>>> 2013/1/25 周梦想 <[EMAIL PROTECTED]>
>>>
>>>> when i start the flume node service,the agent is OPENING.
>>>> Hello,
>>>>
>>>> I need to gather logs from windows 7, but the flume master and
>>>> collector is running on linux. I download 0.9.4 exe file from flume
>>>> official web.
>>>>
>>>> When I start the node from windows, the status is OPENING.
>>>>
>>>> zhouhh.TKOffice.local zhouhh.TKOffice.local zhouhh.TKOffice.local
>>>> OPENING Fri Jan 25 11:45:01 CST 2013 4
>>>>
>>>> And the log reports OutOfMemoryError.
>>>>
>>>> ===============>>>> D:\Program Files\Cloudera\Flume
>>>> 0.9.4\log\flumenode-stdout.2013-01-25.log
>>>>
>>>> 2013-01-25 15:31:03 Commons Daemon procrun stdout
>>>> initializedListening for transport dt_socket at address: 8888
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> Dumping heap to java_pid8100.hprof ...
>>>> Heap dump file created [13270531 bytes in 0.115 secs]
>>>>
>>>> ==============>>>> D:\Program Files\Cloudera\Flume
>>>> 0.9.4\log\flumenode-stderr.2013-01-25.log
>>>>
>>>> 2013-01-25 15:31:03 Commons Daemon procrun stderr initialized
>>>> 13/01/25 15:31:03 INFO agent.FlumeNode: Flume 0.9.4
>>>> 13/01/25 15:31:03 INFO agent.FlumeNode: rev
>>>> 8d0aa4be4ee50bb664cb0d1624e5634f46d7b62a
>>>> 13/01/25 15:31:03 INFO agent.FlumeNode: Compiled on Mon Jun 20
>>>> 14:46:53 PDT 2011
>>>> 13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>> java.runtime.name=Java(TM) SE Runtime Environment