Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> problem configuring hadoop with s3 bucket


Copy link to this message
-
Re: problem configuring hadoop with s3 bucket
>From the log, I guess that you have made wrong configurations or operations.
The S3 storage layer is at the same level with HDFS.
So if you configure to use S3, it's irrational to occur the HDFS related
logs such as  "org.apache.hadoop.hdfs.server.namenode.FSNamesystem".
The HDFS related logs only occur at the condition that you use HDFS as the
underlying storage layer.

I infer that you should not format the file system, because S3 is already
service delivery and the Hadoop framework just use it as the underlying
file system.
Don't execute " bin/hadoop namenode -format ", it is used only when your
underlying file system is HDFS.

Yanbo

2012/7/25 Alok Kumar <[EMAIL PROTECTED]>

> Hi Yanbo,
>
> I'm using hadoop-1.0.1 on local machine (not EC2)
>
> conf/master : localhost
> conf/slave : localhost
> conf/core-site.xml :
> <property>
>     <name>fs.default.name</name>
>     <value>s3://bucketname</value>
>   </property>
>
> <property>
> <name>fs.s3.awsAccessKeyId</name>
> <value>***********</value>
> </property>
>
> <property>
> <name>fs.s3.awsSecretAccessKey</name>
> <value>**************</value>
> </property>
>
> /etc/hosts :
> 192.168.2.24    DRPM4    # Added by NetworkManager
> 127.0.0.1    localhost.localdomain    localhost
>
>
> $ bin/hadoop namenode -format : successful
>
> $ bin/start-dfs.sh
>
> Namenode log :
>
> 2012-07-25 13:19:32,896 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG:   host = DRPM4/192.168.2.24
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 1.0.1
> STARTUP_MSG:   build > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1243785; compiled by 'hortonfo' on Tue Feb 14 08:15:38 UTC 2012
> ************************************************************/
> 2012-07-25 13:19:33,037 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2012-07-25 13:19:33,049 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2012-07-25 13:19:33,050 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2012-07-25 13:19:33,050 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
> 2012-07-25 13:19:33,889 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2012-07-25 13:19:33,896 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2012-07-25 13:19:33,904 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
> registered.
> 2012-07-25 13:19:33,906 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> NameNode registered.
> 2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet: VM
> type       = 32-bit
> 2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet: 2% max
> memory = 17.77875 MB
> 2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet:
> capacity      = 2^22 = 4194304 entries
> 2012-07-25 13:19:33,946 INFO org.apache.hadoop.hdfs.util.GSet:
> recommended=4194304, actual=4194304
> 2012-07-25 13:19:33,986 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=alok
> 2012-07-25 13:19:33,986 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
> 2012-07-25 13:19:33,986 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isPermissionEnabled=true
> 2012-07-25 13:19:33,995 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.block.invalidate.limit=100
> 2012-07-25 13:19:33,995 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s),
> accessTokenLifetime=0 min(s)
> 2012-07-25 13:19:34,396 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemStateMBean and NameNodeMXBean
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB