Hadoop, mail # user - Re: how to figure out the range of a split that failed?

how to figure out the range of a split that failed?

edward choi 2010-06-30, 00:30

hi,

I am running a mapreduce job on my hadoop cluster.

I am running a 10 gigabytes data and one tiny failed task crashes the whole
operation.
I am up to 98% complete and throwing away all the finished data seems just
like an awful waste.
I'd like to save the finished data and run again only the failed ones(the
remaining 2%).

Is there any way to figure out the range of the splits that failed?
I go to "localhost:50030" to see if I can find any useful information but I
must be looking at wrong places.

Could somebody help me with this problem?
Below is the log of a failed task. Any information I can use?

*syslog logs*

Records R/W=41707/41639
2010-06-30 07:35:30,530 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=41776/41726
2010-06-30 07:35:40,554 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=41865/41804
2010-06-30 07:35:50,559 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=41970/41932
2010-06-30 07:36:00,637 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=42073/42065
2010-06-30 07:36:10,772 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=42258/42196
2010-06-30 07:36:20,785 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=42318/42274
2010-06-30 07:36:30,985 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=42378/42351
2010-06-30 07:36:41,005 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=42442/42419
2010-06-30 07:36:51,149 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=42499/42484
2010-06-30 07:37:01,235 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=42559/42547
2010-06-30 07:37:11,242 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=42626/42611
2010-06-30 07:37:21,485 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=42769/42704
2010-06-30 07:37:31,617 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=42845/42782
2010-06-30 07:37:41,725 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=42915/42875
2010-06-30 07:37:51,733 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=42986/42949
2010-06-30 07:38:01,795 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=43070/43051
2010-06-30 07:38:11,849 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=43138/43136
2010-06-30 07:38:22,398 INFO org.apache.hadoop.streaming.PipeMapRed:
Records R/W=43258/43200
2010-06-30 07:38:31,642 INFO org.apache.hadoop.streaming.PipeMapRed:
MRErrorThread done
2010-06-30 07:38:31,643 INFO org.apache.hadoop.streaming.PipeMapRed:
MROutputThread done
2010-06-30 07:38:31,765 INFO org.apache.hadoop.streaming.PipeMapRed: log:null
R/W/S=43335/43271/0 in:7=43335/5885 [rec/s] out:7=43271/5885 [rec/s]
minRecWrittenToEnableSkip_=9223372036854775807 LOGNAME=null
HOST=null
USER=hadoop
HADOOP_USER=null
last Hadoop input: |null|
last tool output: |[B@d22860|
Date: Wed Jun 30 07:38:31 KST 2010
java.io.IOException: Broken pipe
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.streaming.PipeMapRed.write(PipeMapRed.java:635)
at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:105)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
2010-06-30 07:38:31,766 INFO org.apache.hadoop.streaming.PipeMapRed:
PipeMapRed failed!
2010-06-30 07:38:31,766 INFO org.apache.hadoop.streaming.PipeMapRed:
PipeMapRed failed!
2010-06-30 07:38:32,028 WARN org.apache.hadoop.mapred.TaskTracker:
Error running child
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess
failed with code 139
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
2010-06-30 07:38:32,029 INFO org.apache.hadoop.mapred.TaskRunner:
Runnning cleanup for the task

Re: how to figure out the range of a split that failed?

Hemanth Yamijala 2010-06-30, 04:23

Hi,

> I am running a mapreduce job on my hadoop cluster.
>
> I am running a 10 gigabytes data and one tiny failed task crashes the whole
> operation.
> I am up to 98% complete and throwing away all the finished data seems just
> like an awful waste.
> I'd like to save the finished data and run again only the failed ones(the
> remaining 2%).
>
> Is there any way to figure out the range of the splits that failed?
> I go to "localhost:50030" to see if I can find any useful information but I
> must be looking at wrong places.

Can you check the 'Skip Bad records' feature mentioned here and see if
that helps: http://hadoop.apache.org/common/docs/r0.20.1/mapred_tutorial.html#Skipping+Bad+Records
?

Thanks
Hemanth

>
> Could somebody help me with this problem?
>
>
> Below is the log of a failed task. Any information I can use?
>
> *syslog logs*
>
> Records R/W=41707/41639
> 2010-06-30 07:35:30,530 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=41776/41726
> 2010-06-30 07:35:40,554 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=41865/41804
> 2010-06-30 07:35:50,559 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=41970/41932
> 2010-06-30 07:36:00,637 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=42073/42065
> 2010-06-30 07:36:10,772 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=42258/42196
> 2010-06-30 07:36:20,785 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=42318/42274
> 2010-06-30 07:36:30,985 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=42378/42351
> 2010-06-30 07:36:41,005 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=42442/42419
> 2010-06-30 07:36:51,149 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=42499/42484
> 2010-06-30 07:37:01,235 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=42559/42547
> 2010-06-30 07:37:11,242 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=42626/42611
> 2010-06-30 07:37:21,485 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=42769/42704
> 2010-06-30 07:37:31,617 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=42845/42782
> 2010-06-30 07:37:41,725 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=42915/42875
> 2010-06-30 07:37:51,733 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=42986/42949
> 2010-06-30 07:38:01,795 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=43070/43051
> 2010-06-30 07:38:11,849 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=43138/43136
> 2010-06-30 07:38:22,398 INFO org.apache.hadoop.streaming.PipeMapRed:
> Records R/W=43258/43200
> 2010-06-30 07:38:31,642 INFO org.apache.hadoop.streaming.PipeMapRed:
> MRErrorThread done
> 2010-06-30 07:38:31,643 INFO org.apache.hadoop.streaming.PipeMapRed:
> MROutputThread done
> 2010-06-30 07:38:31,765 INFO org.apache.hadoop.streaming.PipeMapRed: log:null
> R/W/S=43335/43271/0 in:7=43335/5885 [rec/s] out:7=43271/5885 [rec/s]
> minRecWrittenToEnableSkip_=9223372036854775807 LOGNAME=null
> HOST=null
> USER=hadoop
> HADOOP_USER=null
> last Hadoop input: |null|
> last tool output: |[B@d22860|
> Date: Wed Jun 30 07:38:31 KST 2010
> java.io.IOException: Broken pipe
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.streaming.PipeMapRed.write(PipeMapRed.java:635)
> at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:105)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)

Re: how to figure out the range of a split that failed?

edward choi 2010-06-30, 05:20

Thanks for the quick response.
I know the SkipBadRecords feature but unfortunately I cannot use it since I
am running my job on Hadoop Streaming.
I had asked if there were any way to use SkipBadRecords in Hadoop Streaming
but never got an answer. I guess it is not possible at all.
Thanks for your concern.

2010/6/30 Hemanth Yamijala <[EMAIL PROTECTED]>

> Hi,
>
> > I am running a mapreduce job on my hadoop cluster.
> >
> > I am running a 10 gigabytes data and one tiny failed task crashes the
> whole
> > operation.
> > I am up to 98% complete and throwing away all the finished data seems
> just
> > like an awful waste.
> > I'd like to save the finished data and run again only the failed ones(the
> > remaining 2%).
> >
> > Is there any way to figure out the range of the splits that failed?
> > I go to "localhost:50030" to see if I can find any useful information but
> I
> > must be looking at wrong places.
>
> Can you check the 'Skip Bad records' feature mentioned here and see if
> that helps:
> http://hadoop.apache.org/common/docs/r0.20.1/mapred_tutorial.html#Skipping+Bad+Records
> ?
>
> Thanks
> Hemanth
>
> >
> > Could somebody help me with this problem?
> >
> >
> > Below is the log of a failed task. Any information I can use?
> >
> > *syslog logs*
> >
> > Records R/W=41707/41639
> > 2010-06-30 07:35:30,530 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=41776/41726
> > 2010-06-30 07:35:40,554 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=41865/41804
> > 2010-06-30 07:35:50,559 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=41970/41932
> > 2010-06-30 07:36:00,637 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=42073/42065
> > 2010-06-30 07:36:10,772 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=42258/42196
> > 2010-06-30 07:36:20,785 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=42318/42274
> > 2010-06-30 07:36:30,985 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=42378/42351
> > 2010-06-30 07:36:41,005 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=42442/42419
> > 2010-06-30 07:36:51,149 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=42499/42484
> > 2010-06-30 07:37:01,235 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=42559/42547
> > 2010-06-30 07:37:11,242 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=42626/42611
> > 2010-06-30 07:37:21,485 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=42769/42704
> > 2010-06-30 07:37:31,617 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=42845/42782
> > 2010-06-30 07:37:41,725 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=42915/42875
> > 2010-06-30 07:37:51,733 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=42986/42949
> > 2010-06-30 07:38:01,795 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=43070/43051
> > 2010-06-30 07:38:11,849 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=43138/43136
> > 2010-06-30 07:38:22,398 INFO org.apache.hadoop.streaming.PipeMapRed:
> > Records R/W=43258/43200
> > 2010-06-30 07:38:31,642 INFO org.apache.hadoop.streaming.PipeMapRed:
> > MRErrorThread done
> > 2010-06-30 07:38:31,643 INFO org.apache.hadoop.streaming.PipeMapRed:
> > MROutputThread done
> > 2010-06-30 07:38:31,765 INFO org.apache.hadoop.streaming.PipeMapRed:
> log:null
> > R/W/S=43335/43271/0 in:7=43335/5885 [rec/s] out:7=43271/5885 [rec/s]
> > minRecWrittenToEnableSkip_=9223372036854775807 LOGNAME=null
> > HOST=null
> > USER=hadoop
> > HADOOP_USER=null
> > last Hadoop input: |null|
> > last tool output: |[B@d22860|
> > Date: Wed Jun 30 07:38:31 KST 2010
> > java.io.IOException: Broken pipe
> > at java.io.FileOutputStream.writeBytes(Native Method)
> > at java.io.FileOutputStream.write(FileOutputStream.java:260)
> > at
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> > at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)

Re: how to figure out the range of a split that failed?

edward choi 2010-06-30, 11:35

Dear Sharad,

Oh my God, thank you Sharad. You are my savior. Though the example you've
given me is not the exact Hadoop Stream I was looking for, it sure shed
light on my problem.
Thanks again!!!
And for the people who are wondering how to enable SkipBadRecords feature in
Hadoop Streaming, refer to this site:
http://hadoop.apache.org/common/docs/current/mapred-default.html
Search for "mapred.skip.attempts.to.start.skipping" and you will get the
answer.

Sincerely, Ed

2010/6/30 Sharad Agarwal <[EMAIL PROTECTED]>

> edward choi wrote:
>
>> Thanks for the quick response.
>> I know the SkipBadRecords feature but unfortunately I cannot use it since
>> I
>> am running my job on Hadoop Streaming.
>> I had asked if there were any way to use SkipBadRecords in Hadoop
>> Streaming
>> but never got an answer. I guess it is not possible at all.
>> Thanks for your concern.
>>
>>
> SkipBadRecords feature can be used for streaming as well. Perhaps the best
> example is the testcase
> ->
> http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingBadRecords.java?view=markup
>
> Sharad
>

Re: how to figure out the range of a split that failed?

edward choi 2010-07-01, 05:15

Dear Sharad,

I have come across another problem. I hope you can help me with this too.
I am trying to use SkipBadRecords feature on Hadoop Streaming.
The streaming method I use is: "hadoop jar
$HADOOP_INSTALL/contrib/streaming/hadoop-*-streaming.jar"
But your example uses Java application which I cannot use because I am
trying to use a C++ application connecting it with Hadoop Streaming.

So what I am doing is:
hadoop jar $HADOOP_INSTALL/contrib/streaming/hadoop-*-streaming.jar -D
mapred.skip.mode.enabled=true -D mapred.skip.attempts.to.start.skipping=2 -D
mapred.skip.map.max.skip.records=Long.MAX_VALUE -D mapred.reduce.tasks=0
-file "..." -mapper "..." -input "..." -output "..."

Then I noticed that you have to set
"mapred.skip.map.auto.incr.proc.count=false" and increment
COUNTER_MAP_PROCESSED_RECORDS in your own application. I guess that you can
do this in your example, but I don't know how to do it using my way of
Hadoop Streaming. Could you enlighten me please?

Sincerely, Ed

2010/6/30 Sharad Agarwal <[EMAIL PROTECTED]>

> edward choi wrote:
>
>> Thanks for the quick response.
>> I know the SkipBadRecords feature but unfortunately I cannot use it since
>> I
>> am running my job on Hadoop Streaming.
>> I had asked if there were any way to use SkipBadRecords in Hadoop
>> Streaming
>> but never got an answer. I guess it is not possible at all.
>> Thanks for your concern.
>>
>>
> SkipBadRecords feature can be used for streaming as well. Perhaps the best
> example is the testcase
> ->
> http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingBadRecords.java?view=markup
>
> Sharad
>

Re: how to figure out the range of a split that failed?

edward choi 2010-07-05, 04:30

Thanks for the response. I went to the web page you told me and several
other pages that I found.
I am still not sure if I got it right.
If I am trying to increment COUNTER_MAP_PROCESS_RECORDS using Hadoop
Streaming, is the example below the way to do it? (assuming that I am using
c++)

example:
cerr << "reporter:counter:counters,linecount,1" << endl;

I am not sure how to use "reporter:counter:<group>,<counter>,<amount>".
I wrote the example by referring to "
https://issues.apache.org/jira/browse/HADOOP-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
"
Am I doing it right?

2010/7/1 Sharad Agarwal <[EMAIL PROTECTED]>

> edward choi wrote:
>
>>
>>
>> Then I noticed that you have to set
>> "mapred.skip.map.auto.incr.proc.count=false" and increment
>> COUNTER_MAP_PROCESSED_RECORDS in your own application. I guess that you can
>> do this in your example, but I don't know how to do it using my way of
>> Hadoop Streaming. Could you enlighten me please?
>>
>> You can refer
>
> http://hadoop.apache.org/common/docs/r0.20.0/streaming.html#How+do+I+update+counters+in+streaming+applications%3F
> for knowing how to increment counter in streaming.
>
>
> Sharad
>

Re: how to figure out the range of a split that failed?

edward choi 2010-07-06, 05:07

Thanks for the tip.
I actually already have tried your method. The command I wrote is like below

cerr << "reporter:counter:SkippingTaskCounters,MapProcessedRecords,1\n";

This actually produced some skipped records in skip folder. But the problem
is that the skipped records' text was all messed up. So I couldn't recycle
them. The broken text is at the end of this mail.
I don't know the reason. Maybe it's because I wrote some other information
on error stream(such as document ID. The command below is the one)

cerr << "Processing: " << docID << endl;

Anyway if I happen to have any progress, I will update through this post.

Broken Text:
SEQ#6;!org.apache.hadoop.io.LongWritable#25;org.apache.hadoop.io.Text#1;#1;*org.apache.hadoop.io.compress.DefaultCodec�9/#6;S#11;#2;#15;庇4z��#25;9/#6;S#11;#2;#15;庇4z��
x��#26;#17;#18;x�`#1;[#15;0텬�#6;,#1;w#14;x�c�c]#1;#7;r#2;8�$x�U]o#19;W#16;}i�H�Hi��Z�b+X�r�y접#16;�#4;"H��\#17;��탈�y��
6�%�b��EML#18;の]`[}Ø�}�則�霙�綴#23;�愁P�#18;<#18;#31;�q�賽��\��5~庾X�xp�c≠�=��<�嵬#14;p|3:#24;#28;#25;�B�A?-8�勁쫑�2��_�@��#18;F#A�6#16;
#15;G�}殘�蛛甦Q�$潾�,8#16;��#4;0鉛=�츈^曠�#29;�Ｇ�└&?日��;=~兪��묽��씔�nl�o]�판��M�d橫罹�|잣��C�#15;#4;噫�0#18;��#20;��p`��i#27;�U��z�4#20;t^d芚Mc}�qf��뎬��_^#6;朽c��V��#3;}S�V콸�Q�%κ쐴*�O_��s��:�kn��(��b��RX�+oh#1;S��8��梁#5;=㈍哥�0%DGw當�.
#7;�#20;3#1;V��y#6;�분�!A�0%�;��瀏��茁�'&?
�N#31;�jP��#8;��$�� %�m��밑!c�#8;�l#17;s��<``% ��廣#18;�>Y�i��#8;�璽#6;��Fb賃$x좃_�W�L
�\F#12;�g#7;i�Ix8#22;#25;��廟別<�毫�<u9�BI(5S��ｕ#21;�#6;�後#3;�
#��#14;#15;~��弛欒��鯖�`h��|�9�c�高#5;#16;�:課Y�Q��#31;銳�占�쇽#31;��p
-�cK��#20;qY#4;%�OR�#18;��55�pJ��潁~�#14;�Qt�>HywK�\Dz觴��U��(��/��渾�Re'%�s#30;0k咫키L�#11;%{#17;K$��U+��1�B5�#12;7j�-�f~��~頃�K�`c\#26;�G�+t#23;-��dJc|s�#11;�b�#16;#6;vWA�2�荳�f�X�却"��M��W�.��#23;��D=O~��#25;/��$
#20;s#14;��헷1M�e��#29;��긺 ^#3;�9��1�揷�]奸��;��0-�굇��
�#8;w욜=l泊�踰=c��u��瓣��S��#18;*��奈y1룡#18;bM]��X�씬�7h#27;짼Zx�\琅윰'��n?-�s#31;�?�#17;�q�弘벧MV�*距�#28;已희�岸侖:N#2;��刃#2;2�#20;
2010/7/6 Sharad Agarwal <[EMAIL PROTECTED]>

> to be precise you have to write on error stream ->
> for map:
> reporter:counter:SkippingTaskCounters,MapProcessedRecords,<count>
>
> for reduce:
> reporter:counter:SkippingTaskCounters,ReduceProcessedGroups,<count>
>
>
> edward choi wrote:
>
>> Thanks for the response. I went to the web page you told me and several
>> other pages that I found.
>> I am still not sure if I got it right.
>> If I am trying to increment COUNTER_MAP_PROCESS_RECORDS using Hadoop
>> Streaming, is the example below the way to do it? (assuming that I am using
>> c++)
>>
>> example:
>> cerr << "reporter:counter:counters,linecount,1" << endl;
>>
>>
>>
>