Hadoop, mail # user - Re: EOFException at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)......

EOFException at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)......

waqas latif 2012-05-25, 09:42

Hi Experts,

I am fairly new to hadoop MapR and I was trying to run a matrix
multiplication example presented by Mr. Norstadt under following link
http://www.norstad.org/matrix-multiply/index.html. I can run it
successfully with hadoop 0.20.2 but I tried to run it with hadoop 1.0.3 but
I am getting following error. Is it the problem with my hadoop
configuration or it is compatibility problem in the code which was written
in hadoop 0.20 by author.Also please guide me that how can I fix this error
in either case. Here is the error I am getting.

in thread "main" java.io.EOFException
   at java.io.DataInputStream.readFully(DataInputStream.java:180)
   at java.io.DataInputStream.readFully(DataInputStream.java:152)
   at
org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
   at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486)
   at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
   at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470)
   at TestMatrixMultiply.fillMatrix(TestMatrixMultiply.java:60)
   at TestMatrixMultiply.readMatrix(TestMatrixMultiply.java:87)
   at TestMatrixMultiply.checkAnswer(TestMatrixMultiply.java:112)
   at TestMatrixMultiply.runOneTest(TestMatrixMultiply.java:150)
   at TestMatrixMultiply.testRandom(TestMatrixMultiply.java:278)
   at TestMatrixMultiply.main(TestMatrixMultiply.java:308)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

Thanks in advance

Regards,
waqas

Re: EOFException at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)......

Marcos Ortiz 2012-05-25, 15:13

Regards, waqas. I think that you have to ask to MapR experts.
On 05/25/2012 05:42 AM, waqas latif wrote:
> Hi Experts,
>
> I am fairly new to hadoop MapR and I was trying to run a matrix
> multiplication example presented by Mr. Norstadt under following link
> http://www.norstad.org/matrix-multiply/index.html. I can run it
> successfully with hadoop 0.20.2 but I tried to run it with hadoop 1.0.3 but
> I am getting following error. Is it the problem with my hadoop
> configuration or it is compatibility problem in the code which was written
> in hadoop 0.20 by author.Also please guide me that how can I fix this error
> in either case. Here is the error I am getting.
The same code that you write for 0.20.2 should work in 1.0.3 too.

>
> in thread "main" java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470)
> at TestMatrixMultiply.fillMatrix(TestMatrixMultiply.java:60)
> at TestMatrixMultiply.readMatrix(TestMatrixMultiply.java:87)
> at TestMatrixMultiply.checkAnswer(TestMatrixMultiply.java:112)
> at TestMatrixMultiply.runOneTest(TestMatrixMultiply.java:150)
> at TestMatrixMultiply.testRandom(TestMatrixMultiply.java:278)
> at TestMatrixMultiply.main(TestMatrixMultiply.java:308)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> Thanks in advance
>
> Regards,
> waqas
Can you put here the completed log for this?
Best wishes

--
Marcos Luis Ortï¿½z Valmaseda
  Data Engineer&& Sr. System Administrator at UCI
  http://marcosluis2186.posterous.com
  http://www.linkedin.com/in/marcosluis2186
  Twitter: @marcosluis2186
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci

Re: EOFException at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)......

Harsh J 2012-05-25, 16:39

Waqas,

Can you ensure this file isn't empty (0 in size)?

On Fri, May 25, 2012 at 3:12 PM, waqas latif <[EMAIL PROTECTED]> wrote:
> Hi Experts,
>
> I am fairly new to hadoop MapR and I was trying to run a matrix
> multiplication example presented by Mr. Norstadt under following link
> http://www.norstad.org/matrix-multiply/index.html. I can run it
> successfully with hadoop 0.20.2 but I tried to run it with hadoop 1.0.3 but
> I am getting following error. Is it the problem with my hadoop
> configuration or it is compatibility problem in the code which was written
> in hadoop 0.20 by author.Also please guide me that how can I fix this error
> in either case. Here is the error I am getting.
>
> in thread "main" java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470)
> at TestMatrixMultiply.fillMatrix(TestMatrixMultiply.java:60)
> at TestMatrixMultiply.readMatrix(TestMatrixMultiply.java:87)
> at TestMatrixMultiply.checkAnswer(TestMatrixMultiply.java:112)
> at TestMatrixMultiply.runOneTest(TestMatrixMultiply.java:150)
> at TestMatrixMultiply.testRandom(TestMatrixMultiply.java:278)
> at TestMatrixMultiply.main(TestMatrixMultiply.java:308)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> Thanks in advance
>
> Regards,
> waqas

--
Harsh J

Re: EOFException at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)......

Kasi Subrahmanyam 2012-05-26, 05:33

Hi,
If you are using a custom writable object while passing data from the
mapper to the reducer make sure that the read fields and the write has the
same number of variables. It might be possible that you wrote datavtova
file using custom writable but later modified the custom writable (like
adding new attribute to the writable) which the old data doesn't have.

It might be a possibility is please check once

On Friday, May 25, 2012, waqas latif wrote:

> Hi Experts,
>
> I am fairly new to hadoop MapR and I was trying to run a matrix
> multiplication example presented by Mr. Norstadt under following link
> http://www.norstad.org/matrix-multiply/index.html. I can run it
> successfully with hadoop 0.20.2 but I tried to run it with hadoop 1.0.3 but
> I am getting following error. Is it the problem with my hadoop
> configuration or it is compatibility problem in the code which was written
> in hadoop 0.20 by author.Also please guide me that how can I fix this error
> in either case. Here is the error I am getting.
>
> in thread "main" java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470)
> at TestMatrixMultiply.fillMatrix(TestMatrixMultiply.java:60)
> at TestMatrixMultiply.readMatrix(TestMatrixMultiply.java:87)
> at TestMatrixMultiply.checkAnswer(TestMatrixMultiply.java:112)
> at TestMatrixMultiply.runOneTest(TestMatrixMultiply.java:150)
> at TestMatrixMultiply.testRandom(TestMatrixMultiply.java:278)
> at TestMatrixMultiply.main(TestMatrixMultiply.java:308)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> Thanks in advance
>
> Regards,
> waqas
>

Re: EOFException at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)......

Prashant Kommireddi 2012-05-27, 10:00

I have seen this issue with large file writes using SequenceFile writer.
Not found the same issue when testing with writing fairly small files ( <
1GB).

On Fri, May 25, 2012 at 10:33 PM, Kasi Subrahmanyam
<[EMAIL PROTECTED]>wrote:

> Hi,
> If you are using a custom writable object while passing data from the
> mapper to the reducer make sure that the read fields and the write has the
> same number of variables. It might be possible that you wrote datavtova
> file using custom writable but later modified the custom writable (like
> adding new attribute to the writable) which the old data doesn't have.
>
> It might be a possibility is please check once
>
> On Friday, May 25, 2012, waqas latif wrote:
>
> > Hi Experts,
> >
> > I am fairly new to hadoop MapR and I was trying to run a matrix
> > multiplication example presented by Mr. Norstadt under following link
> > http://www.norstad.org/matrix-multiply/index.html. I can run it
> > successfully with hadoop 0.20.2 but I tried to run it with hadoop 1.0.3
> but
> > I am getting following error. Is it the problem with my hadoop
> > configuration or it is compatibility problem in the code which was
> written
> > in hadoop 0.20 by author.Also please guide me that how can I fix this
> error
> > in either case. Here is the error I am getting.
> >
> > in thread "main" java.io.EOFException
> > at java.io.DataInputStream.readFully(DataInputStream.java:180)
> > at java.io.DataInputStream.readFully(DataInputStream.java:152)
> > at
> > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
> > at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486)
> > at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> > at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470)
> > at TestMatrixMultiply.fillMatrix(TestMatrixMultiply.java:60)
> > at TestMatrixMultiply.readMatrix(TestMatrixMultiply.java:87)
> > at TestMatrixMultiply.checkAnswer(TestMatrixMultiply.java:112)
> > at TestMatrixMultiply.runOneTest(TestMatrixMultiply.java:150)
> > at TestMatrixMultiply.testRandom(TestMatrixMultiply.java:278)
> > at TestMatrixMultiply.main(TestMatrixMultiply.java:308)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >
> > Thanks in advance
> >
> > Regards,
> > waqas
> >
>

Re: EOFException at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)......

waqas latif 2012-05-27, 16:15

But the thing is, it works with hadoop 0.20. even with 100 x100(and even
bigger matrices) but when it comes to hadoop 1.0.3 then even there is a
problem with 3x3 matrix.

On Sun, May 27, 2012 at 12:00 PM, Prashant Kommireddi
<[EMAIL PROTECTED]>wrote:

> I have seen this issue with large file writes using SequenceFile writer.
> Not found the same issue when testing with writing fairly small files ( <
> 1GB).
>
> On Fri, May 25, 2012 at 10:33 PM, Kasi Subrahmanyam
> <[EMAIL PROTECTED]>wrote:
>
> > Hi,
> > If you are using a custom writable object while passing data from the
> > mapper to the reducer make sure that the read fields and the write has
> the
> > same number of variables. It might be possible that you wrote datavtova
> > file using custom writable but later modified the custom writable (like
> > adding new attribute to the writable) which the old data doesn't have.
> >
> > It might be a possibility is please check once
> >
> > On Friday, May 25, 2012, waqas latif wrote:
> >
> > > Hi Experts,
> > >
> > > I am fairly new to hadoop MapR and I was trying to run a matrix
> > > multiplication example presented by Mr. Norstadt under following link
> > > http://www.norstad.org/matrix-multiply/index.html. I can run it
> > > successfully with hadoop 0.20.2 but I tried to run it with hadoop 1.0.3
> > but
> > > I am getting following error. Is it the problem with my hadoop
> > > configuration or it is compatibility problem in the code which was
> > written
> > > in hadoop 0.20 by author.Also please guide me that how can I fix this
> > error
> > > in either case. Here is the error I am getting.
> > >
> > > in thread "main" java.io.EOFException
> > > at java.io.DataInputStream.readFully(DataInputStream.java:180)
> > > at java.io.DataInputStream.readFully(DataInputStream.java:152)
> > > at
> > > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
> > > at
> > > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486)
> > > at
> > > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> > > at
> > > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470)
> > > at TestMatrixMultiply.fillMatrix(TestMatrixMultiply.java:60)
> > > at TestMatrixMultiply.readMatrix(TestMatrixMultiply.java:87)
> > > at TestMatrixMultiply.checkAnswer(TestMatrixMultiply.java:112)
> > > at TestMatrixMultiply.runOneTest(TestMatrixMultiply.java:150)
> > > at TestMatrixMultiply.testRandom(TestMatrixMultiply.java:278)
> > > at TestMatrixMultiply.main(TestMatrixMultiply.java:308)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > >
> > > Thanks in advance
> > >
> > > Regards,
> > > waqas
> > >
> >
>

Re: EOFException at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)......

waqas latif 2012-05-29, 11:42

So my question is that do hadoop 0.20 and 1.0.3 differ in their support of
writing or reading sequencefiles? same code works fine with hadoop 0.20 but
problem occurs when run it under hadoop 1.0.3.

On Sun, May 27, 2012 at 6:15 PM, waqas latif <[EMAIL PROTECTED]> wrote:

> But the thing is, it works with hadoop 0.20. even with 100 x100(and even
> bigger matrices) but when it comes to hadoop 1.0.3 then even there is a
> problem with 3x3 matrix.
>
>
> On Sun, May 27, 2012 at 12:00 PM, Prashant Kommireddi <[EMAIL PROTECTED]
> > wrote:
>
>> I have seen this issue with large file writes using SequenceFile writer.
>> Not found the same issue when testing with writing fairly small files ( <
>> 1GB).
>>
>> On Fri, May 25, 2012 at 10:33 PM, Kasi Subrahmanyam
>> <[EMAIL PROTECTED]>wrote:
>>
>> > Hi,
>> > If you are using a custom writable object while passing data from the
>> > mapper to the reducer make sure that the read fields and the write has
>> the
>> > same number of variables. It might be possible that you wrote datavtova
>> > file using custom writable but later modified the custom writable (like
>> > adding new attribute to the writable) which the old data doesn't have.
>> >
>> > It might be a possibility is please check once
>> >
>> > On Friday, May 25, 2012, waqas latif wrote:
>> >
>> > > Hi Experts,
>> > >
>> > > I am fairly new to hadoop MapR and I was trying to run a matrix
>> > > multiplication example presented by Mr. Norstadt under following link
>> > > http://www.norstad.org/matrix-multiply/index.html. I can run it
>> > > successfully with hadoop 0.20.2 but I tried to run it with hadoop
>> 1.0.3
>> > but
>> > > I am getting following error. Is it the problem with my hadoop
>> > > configuration or it is compatibility problem in the code which was
>> > written
>> > > in hadoop 0.20 by author.Also please guide me that how can I fix this
>> > error
>> > > in either case. Here is the error I am getting.
>> > >
>> > > in thread "main" java.io.EOFException
>> > > at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> > > at java.io.DataInputStream.readFully(DataInputStream.java:152)
>> > > at
>> > > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
>> > > at
>> > >
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486)
>> > > at
>> > >
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
>> > > at
>> > >
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470)
>> > > at TestMatrixMultiply.fillMatrix(TestMatrixMultiply.java:60)
>> > > at TestMatrixMultiply.readMatrix(TestMatrixMultiply.java:87)
>> > > at TestMatrixMultiply.checkAnswer(TestMatrixMultiply.java:112)
>> > > at TestMatrixMultiply.runOneTest(TestMatrixMultiply.java:150)
>> > > at TestMatrixMultiply.testRandom(TestMatrixMultiply.java:278)
>> > > at TestMatrixMultiply.main(TestMatrixMultiply.java:308)
>> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > > at
>> > >
>> > >
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > > at
>> > >
>> > >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > > at java.lang.reflect.Method.invoke(Method.java:597)
>> > > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> > >
>> > > Thanks in advance
>> > >
>> > > Regards,
>> > > waqas
>> > >
>> >
>>
>
>

Re: EOFException at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)......

waqas latif 2012-05-30, 14:16

I got the problem with I am unable to solve it. I need to apply a filter
for _SUCCESS file while using FileSystem.listStatus method. Can someone
please guide me how to filter _SUCCESS files. Thanks

On Tue, May 29, 2012 at 1:42 PM, waqas latif <[EMAIL PROTECTED]> wrote:

> So my question is that do hadoop 0.20 and 1.0.3 differ in their support of
> writing or reading sequencefiles? same code works fine with hadoop 0.20 but
> problem occurs when run it under hadoop 1.0.3.
>
>
> On Sun, May 27, 2012 at 6:15 PM, waqas latif <[EMAIL PROTECTED]> wrote:
>
>> But the thing is, it works with hadoop 0.20. even with 100 x100(and even
>> bigger matrices) but when it comes to hadoop 1.0.3 then even there is a
>> problem with 3x3 matrix.
>>
>>
>> On Sun, May 27, 2012 at 12:00 PM, Prashant Kommireddi <
>> [EMAIL PROTECTED]> wrote:
>>
>>> I have seen this issue with large file writes using SequenceFile writer.
>>> Not found the same issue when testing with writing fairly small files ( <
>>> 1GB).
>>>
>>> On Fri, May 25, 2012 at 10:33 PM, Kasi Subrahmanyam
>>> <[EMAIL PROTECTED]>wrote:
>>>
>>> > Hi,
>>> > If you are using a custom writable object while passing data from the
>>> > mapper to the reducer make sure that the read fields and the write has
>>> the
>>> > same number of variables. It might be possible that you wrote datavtova
>>> > file using custom writable but later modified the custom writable (like
>>> > adding new attribute to the writable) which the old data doesn't have.
>>> >
>>> > It might be a possibility is please check once
>>> >
>>> > On Friday, May 25, 2012, waqas latif wrote:
>>> >
>>> > > Hi Experts,
>>> > >
>>> > > I am fairly new to hadoop MapR and I was trying to run a matrix
>>> > > multiplication example presented by Mr. Norstadt under following link
>>> > > http://www.norstad.org/matrix-multiply/index.html. I can run it
>>> > > successfully with hadoop 0.20.2 but I tried to run it with hadoop
>>> 1.0.3
>>> > but
>>> > > I am getting following error. Is it the problem with my hadoop
>>> > > configuration or it is compatibility problem in the code which was
>>> > written
>>> > > in hadoop 0.20 by author.Also please guide me that how can I fix this
>>> > error
>>> > > in either case. Here is the error I am getting.
>>> > >
>>> > > in thread "main" java.io.EOFException
>>> > > at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>> > > at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>> > > at
>>> > > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
>>> > > at
>>> > >
>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486)
>>> > > at
>>> > >
>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
>>> > > at
>>> > >
>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470)
>>> > > at TestMatrixMultiply.fillMatrix(TestMatrixMultiply.java:60)
>>> > > at TestMatrixMultiply.readMatrix(TestMatrixMultiply.java:87)
>>> > > at TestMatrixMultiply.checkAnswer(TestMatrixMultiply.java:112)
>>> > > at TestMatrixMultiply.runOneTest(TestMatrixMultiply.java:150)
>>> > > at TestMatrixMultiply.testRandom(TestMatrixMultiply.java:278)
>>> > > at TestMatrixMultiply.main(TestMatrixMultiply.java:308)
>>> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > > at
>>> > >
>>> > >
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > > at
>>> > >
>>> > >
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>> > >
>>> > > Thanks in advance
>>> > >
>>> > > Regards,
>>> > > waqas
>>> > >
>>> >
>>>
>>
>>
>

Re: EOFException at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)......

Harsh J 2012-05-30, 15:58

When your code does a listStatus, you can pass a PathFilter object
along that can do this filtering for you. See
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listStatus(org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.PathFilter)
for the API javadocs on that.

On Wed, May 30, 2012 at 7:46 PM, waqas latif <[EMAIL PROTECTED]> wrote:
> I got the problem with I am unable to solve it. I need to apply a filter
> for _SUCCESS file while using FileSystem.listStatus method. Can someone
> please guide me how to filter _SUCCESS files. Thanks
>
> On Tue, May 29, 2012 at 1:42 PM, waqas latif <[EMAIL PROTECTED]> wrote:
>
>> So my question is that do hadoop 0.20 and 1.0.3 differ in their support of
>> writing or reading sequencefiles? same code works fine with hadoop 0.20 but
>> problem occurs when run it under hadoop 1.0.3.
>>
>>
>> On Sun, May 27, 2012 at 6:15 PM, waqas latif <[EMAIL PROTECTED]> wrote:
>>
>>> But the thing is, it works with hadoop 0.20. even with 100 x100(and even
>>> bigger matrices) but when it comes to hadoop 1.0.3 then even there is a
>>> problem with 3x3 matrix.
>>>
>>>
>>> On Sun, May 27, 2012 at 12:00 PM, Prashant Kommireddi <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> I have seen this issue with large file writes using SequenceFile writer.
>>>> Not found the same issue when testing with writing fairly small files ( <
>>>> 1GB).
>>>>
>>>> On Fri, May 25, 2012 at 10:33 PM, Kasi Subrahmanyam
>>>> <[EMAIL PROTECTED]>wrote:
>>>>
>>>> > Hi,
>>>> > If you are using a custom writable object while passing data from the
>>>> > mapper to the reducer make sure that the read fields and the write has
>>>> the
>>>> > same number of variables. It might be possible that you wrote datavtova
>>>> > file using custom writable but later modified the custom writable (like
>>>> > adding new attribute to the writable) which the old data doesn't have.
>>>> >
>>>> > It might be a possibility is please check once
>>>> >
>>>> > On Friday, May 25, 2012, waqas latif wrote:
>>>> >
>>>> > > Hi Experts,
>>>> > >
>>>> > > I am fairly new to hadoop MapR and I was trying to run a matrix
>>>> > > multiplication example presented by Mr. Norstadt under following link
>>>> > > http://www.norstad.org/matrix-multiply/index.html. I can run it
>>>> > > successfully with hadoop 0.20.2 but I tried to run it with hadoop
>>>> 1.0.3
>>>> > but
>>>> > > I am getting following error. Is it the problem with my hadoop
>>>> > > configuration or it is compatibility problem in the code which was
>>>> > written
>>>> > > in hadoop 0.20 by author.Also please guide me that how can I fix this
>>>> > error
>>>> > > in either case. Here is the error I am getting.
>>>> > >
>>>> > > in thread "main" java.io.EOFException
>>>> > > at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>>> > > at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>>> > > at
>>>> > > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
>>>> > > at
>>>> > >
>>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486)
>>>> > > at
>>>> > >
>>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
>>>> > > at
>>>> > >
>>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470)
>>>> > > at TestMatrixMultiply.fillMatrix(TestMatrixMultiply.java:60)
>>>> > > at TestMatrixMultiply.readMatrix(TestMatrixMultiply.java:87)
>>>> > > at TestMatrixMultiply.checkAnswer(TestMatrixMultiply.java:112)
>>>> > > at TestMatrixMultiply.runOneTest(TestMatrixMultiply.java:150)
>>>> > > at TestMatrixMultiply.testRandom(TestMatrixMultiply.java:278)
>>>> > > at TestMatrixMultiply.main(TestMatrixMultiply.java:308)
>>>> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> > > at
>>>> > >
>>>> > >
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

Harsh J

Re: EOFException at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)......

waqas latif 2012-05-30, 17:30

Thanks Harsh. I got it running.

On Wed, May 30, 2012 at 5:58 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> When your code does a listStatus, you can pass a PathFilter object
> along that can do this filtering for you. See
>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listStatus(org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.PathFilter)
> for the API javadocs on that.
>
> On Wed, May 30, 2012 at 7:46 PM, waqas latif <[EMAIL PROTECTED]> wrote:
> > I got the problem with I am unable to solve it. I need to apply a filter
> > for _SUCCESS file while using FileSystem.listStatus method. Can someone
> > please guide me how to filter _SUCCESS files. Thanks
> >
> > On Tue, May 29, 2012 at 1:42 PM, waqas latif <[EMAIL PROTECTED]> wrote:
> >
> >> So my question is that do hadoop 0.20 and 1.0.3 differ in their support
> of
> >> writing or reading sequencefiles? same code works fine with hadoop 0.20
> but
> >> problem occurs when run it under hadoop 1.0.3.
> >>
> >>
> >> On Sun, May 27, 2012 at 6:15 PM, waqas latif <[EMAIL PROTECTED]>
> wrote:
> >>
> >>> But the thing is, it works with hadoop 0.20. even with 100 x100(and
> even
> >>> bigger matrices) but when it comes to hadoop 1.0.3 then even there is
> a
> >>> problem with 3x3 matrix.
> >>>
> >>>
> >>> On Sun, May 27, 2012 at 12:00 PM, Prashant Kommireddi <
> >>> [EMAIL PROTECTED]> wrote:
> >>>
> >>>> I have seen this issue with large file writes using SequenceFile
> writer.
> >>>> Not found the same issue when testing with writing fairly small files
> ( <
> >>>> 1GB).
> >>>>
> >>>> On Fri, May 25, 2012 at 10:33 PM, Kasi Subrahmanyam
> >>>> <[EMAIL PROTECTED]>wrote:
> >>>>
> >>>> > Hi,
> >>>> > If you are using a custom writable object while passing data from
> the
> >>>> > mapper to the reducer make sure that the read fields and the write
> has
> >>>> the
> >>>> > same number of variables. It might be possible that you wrote
> datavtova
> >>>> > file using custom writable but later modified the custom writable
> (like
> >>>> > adding new attribute to the writable) which the old data doesn't
> have.
> >>>> >
> >>>> > It might be a possibility is please check once
> >>>> >
> >>>> > On Friday, May 25, 2012, waqas latif wrote:
> >>>> >
> >>>> > > Hi Experts,
> >>>> > >
> >>>> > > I am fairly new to hadoop MapR and I was trying to run a matrix
> >>>> > > multiplication example presented by Mr. Norstadt under following
> link
> >>>> > > http://www.norstad.org/matrix-multiply/index.html. I can run it
> >>>> > > successfully with hadoop 0.20.2 but I tried to run it with hadoop
> >>>> 1.0.3
> >>>> > but
> >>>> > > I am getting following error. Is it the problem with my hadoop
> >>>> > > configuration or it is compatibility problem in the code which was
> >>>> > written
> >>>> > > in hadoop 0.20 by author.Also please guide me that how can I fix
> this
> >>>> > error
> >>>> > > in either case. Here is the error I am getting.
> >>>> > >
> >>>> > > in thread "main" java.io.EOFException
> >>>> > > at
> java.io.DataInputStream.readFully(DataInputStream.java:180)
> >>>> > > at
> java.io.DataInputStream.readFully(DataInputStream.java:152)
> >>>> > > at
> >>>> > >
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
> >>>> > > at
> >>>> > >
> >>>>
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486)
> >>>> > > at
> >>>> > >
> >>>>
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
> >>>> > > at
> >>>> > >
> >>>>
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470)
> >>>> > > at
> TestMatrixMultiply.fillMatrix(TestMatrixMultiply.java:60)
> >>>> > > at
> TestMatrixMultiply.readMatrix(TestMatrixMultiply.java:87)
> >>>> > > at
> TestMatrixMultiply.checkAnswer(TestMatrixMultiply.java:112)
> >>>> > > at
> TestMatrixMultiply.runOneTest(TestMatrixMultiply.java:150)
> >>>> > > at
> TestMatrixMultiply.testRandom(TestMatrixMultiply.java:278)