Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re:S3N copy creating recursive folders


+
卖报的小行家 2013-03-05, 14:32
Copy link to this message
-
Re: S3N copy creating recursive folders
Hi,

Its not because there are too many recursive folders in S3 bucket; in-fact there is no recursive folder in the source.
If I list the S3 bucket with Native S3 tools I can find a file srcData with size 0 in the folder srcData.
The copy command keeps on creating folder  /test/srcData/srcData/srcData (keep on appending srcData).

Cheers,
Subroto Sanyal

On Mar 5, 2013, at 3:32 PM, 卖报的小行家 wrote:

> Hi Subroto,
>
> I didn't use the s3n filesystem.But  from the output "cp: java.io.IOException: mkdirs: Pathname too long.  Limit 8000 characters, 1000 levels.", I think this is because the problem of the path. Is the path longer than 8000 characters or the level is more than 1000?
> You only have 998 folders.Maybe the last one is more than 8000 characters.Why not count the last one's length?
>
> BRs//Julian
>
>
>
>
>
> ------------------ Original ------------------
> From:  "Subroto"<[EMAIL PROTECTED]>;
> Date:  Tue, Mar 5, 2013 10:22 PM
> To:  "user"<[EMAIL PROTECTED]>;
> Subject:  S3N copy creating recursive folders
>
> Hi,
>
> I am using Hadoop 1.0.3 and trying to execute:
> hadoop fs -cp s3n://acessKey:[EMAIL PROTECTED]/srcData" /test/srcData
>
> This ends up with:
> cp: java.io.IOException: mkdirs: Pathname too long.  Limit 8000 characters, 1000 levels.
>
> When I try to list the folder recursively /test/srcData: it lists 998 folders like:
> drwxr-xr-x   - root supergroup          0 2013-03-05 08:49 /test/srcData/srcData
> drwxr-xr-x   - root supergroup          0 2013-03-05 08:49 /test/srcData/srcData/srcData
> drwxr-xr-x   - root supergroup          0 2013-03-05 08:49 /test/srcData/srcData/srcData/srcData
> drwxr-xr-x   - root supergroup          0 2013-03-05 08:49 /test/srcData/srcData/srcData/srcData/srcData
> drwxr-xr-x   - root supergroup          0 2013-03-05 08:49 /test/srcData/srcData/srcData/srcData/srcData/srcData
>
> Is there a problem with s3n filesystem ??
>
> Cheers,
> Subroto Sanyal

+
George Datskos 2013-03-07, 07:51
+
Subroto 2013-03-07, 08:33
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB