Hi,

  Are there any logs created when running the metastore using "bin/hive
-service metastore" ? From what I can tell this command generates output
on the screen and runs in the foreground. Would the right way to run
this as a server be "bin/hive -service metstatore > metastore.log 2>&1
&" ? Is there any other logfile which is written to, while the metastore
server is running?

 

Thanks,

Pradeep

 

 

Also a related question is how do I stop the server if have started the
server with "nohup bin/hive -service metstatore > metastore.log 2>&1 &"
command ? When I run "ps -aef |grep hive", I get :

 

pradeepk  5463 14236  0 16:00 pts/6    00:00:00 grep hive

pradeepk 27803 14226  0 13:47 pts/5    00:00:04
/usr/java/jdk1.6.0_01/bin/java -Xmx256m
-Dhadoop.log.dir=/homes/pradeepk/hadoopcluster/hadoop/bin/../logs
-Dhadoop.log.file=hadoop.log
-Dhadoop.home.dir=/homes/pradeepk/hadoopcluster/hadoop/bin/..
-Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
-Djava.library.path=/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/nati
ve/Linux-i386-32 -Dhadoop.policy.file=hadoop-policy.xml -classpath
/homes/pradeepk/hadoopcluster/hadoop/conf:/usr/java/jdk1.6.0_01/lib/tool
s.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/..:/homes/pradeepk/hadoop
cluster/hadoop/bin/../hadoop-0.20.1-core.jar:/homes/pradeepk/hadoopclust
er/hadoop/bin/../lib/commons-cli-1.2.jar:/homes/pradeepk/hadoopcluster/h
adoop/bin/../lib/commons-codec-1.3.jar:/homes/pradeepk/hadoopcluster/had
oop/bin/../lib/commons-el-1.0.jar:/homes/pradeepk/hadoopcluster/hadoop/b
in/../lib/commons-httpclient-3.0.1.jar:/homes/pradeepk/hadoopcluster/had
oop/bin/../lib/commons-logging-1.0.4.jar:/homes/pradeepk/hadoopcluster/h
adoop/bin/../lib/commons-logging-api-1.0.4.jar:/homes/pradeepk/hadoopclu
ster/hadoop/bin/../lib/commons-net-1.4.1.jar:/homes/pradeepk/hadoopclust
er/hadoop/bin/../lib/core-3.1.1.jar:/homes/pradeepk/hadoopcluster/hadoop
/bin/../lib/hsqldb-1.8.0.10.jar:/homes/pradeepk/hadoopcluster/hadoop/bin
/../lib/jasper-compiler-5.5.12.jar:/homes/pradeepk/hadoopcluster/hadoop/
bin/../lib/jasper-runtime-5.5.12.jar:/homes/pradeepk/hadoopcluster/hadoo
p/bin/../lib/jets3t-0.6.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/.
./lib/jetty-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/j
etty-util-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jun
it-3.8.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/kfs-0.2.2.j
ar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/log4j-1.2.15.jar:/hom
es/pradeepk/hadoopcluster/hadoop/bin/../lib/oro-2.0.8.jar:/homes/pradeep
k/hadoopcluster/hadoop/bin/../lib/servlet-api-2.5-6.1.14.jar:/homes/prad
eepk/hadoopcluster/hadoop/bin/../lib/slf4j-api-1.4.3.jar:/homes/pradeepk
/hadoopcluster/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/homes/pradeepk
/hadoopcluster/hadoop/bin/../lib/xmlenc-0.52.jar:/homes/pradeepk/hadoopc
luster/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/homes/pradeepk/hadoopclust
er/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:/homes/pradeepk/dev/hive/bu
ild/dist/conf:/homes/pradeepk/dev/hive/build/dist/lib/antlr-runtime-3.0.
1.jar:/homes/pradeepk/dev/hive/build/dist/lib/asm-3.1.jar:/homes/pradeep
k/dev/hive/build/dist/lib/commons-cli-2.0-SNAPSHOT.jar:/homes/pradeepk/d
ev/hive/build/dist/lib/commons-codec-1.3.jar:/homes/pradeepk/dev/hive/bu
ild/dist/lib/commons-collections-3.2.1.jar:/homes/pradeepk/dev/hive/buil
d/dist/lib/commons-dbcp-1.2.2.jar:/homes/pradeepk/dev/hive/build/dist/li
b/commons-lang-2.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-l
ogging-1.0.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-logging
-api-1.0.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-pool-1.2.
jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-connectionpool-1
.0.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-core-1.1.2.
jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-enhancer-1.1.2.j
ar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-rdbms-1.1.2.jar:/
homes/pradeepk/dev/hive/build/dist/lib/derby.jar:/homes/pradeepk/dev/hiv
e/build/dist/lib/hbase-0.20.3.jar:/homes/pradeepk/dev/hive/build/dist/li
b/hbase-0.20.3-test.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-ant
tasks-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-cli-0.6.0.j
ar:/homes/pradeepk/dev/hive/build/dist/lib/hive-common-0.6.0.jar:/homes/
pradeepk/dev/hive/build/dist/lib/hive_contrib.jar:/homes/pradeepk/dev/hi
ve/build/dist/lib/hive-exec-0.6.0.jar:/homes/pradeepk/dev/hive/build/dis
t/lib/hive_hbase-handler.jar:/homes/pradeepk/dev/hive/build/dist/lib/hiv
e-hwi-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-jdbc-0.6.0.
jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-metastore-0.6.0.jar:/ho
mes/pradeepk/dev/hive/build/dist/lib/hive-serde-0.6.0.jar:/homes/pradeep
k/dev/hive/build/dist/lib/hive-service-0.6.0.jar:/homes/pradeepk/dev/hiv
e/build/dist/lib/hive-shims-0.6.0.jar

 

Is there a recommended way to identify the metastore server process?

 

Thanks,

Pradeep

________________________________

From: Pradeep Kamath [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, June 09, 2010 3:10 PM
To: [EMAIL PROTECTED]
Subject: Logs when running metastore

 

Hi,

  Are there any logs created when running the metastore using "bin/hive
-service metastore" ? From what I can tell this command generates output
on the screen and runs in the foreground. Would the right way to run
this as a server be "bin/hive -service metstatore > metastore.log 2>&1
&" ? Is there any other logfile which is written to, while the metastore
server is running?

 

Thanks,

Pradeep

 

 

On Wed, Jun 9, 2010 at 7:10 PM, Pradeep Kamath <[EMAIL PROTECTED]>wrote:

>  Also a related question is how do I stop the server if have started the
> server with “nohup bin/hive –service metstatore > metastore.log 2>&1 &”
> command ? When I run “ps -aef |grep hive”, I get :
>
>
>
> pradeepk  5463 14236  0 16:00 pts/6    00:00:00 grep hive
>
> pradeepk 27803 14226  0 13:47 pts/5    00:00:04
> /usr/java/jdk1.6.0_01/bin/java -Xmx256m
> -Dhadoop.log.dir=/homes/pradeepk/hadoopcluster/hadoop/bin/../logs
> -Dhadoop.log.file=hadoop.log
> -Dhadoop.home.dir=/homes/pradeepk/hadoopcluster/hadoop/bin/..
> -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
> -Djava.library.path=/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/native/Linux-i386-32
> -Dhadoop.policy.file=hadoop-policy.xml -classpath
> /homes/pradeepk/hadoopcluster/hadoop/conf:/usr/java/jdk1.6.0_01/lib/tools.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/..:/homes/pradeepk/hadoopcluster/hadoop/bin/../hadoop-0.20.1-core.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-cli-1.2.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-codec-1.3.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-el-1.0.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-logging-1.0.4.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-net-1.4.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/core-3.1.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jets3t-0.6.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jetty-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jetty-util-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/junit-3.8.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/kfs-0.2.2.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/log4j-1.2.15.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/oro-2.0.8.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/servlet-api-2.5-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/slf4j-api-1.4.3.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/xmlenc-0.52.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:/homes/pradeepk/dev/hive/build/dist/conf:/homes/pradeepk/dev/hive/build/dist/lib/antlr-runtime-3.0.1.jar:/homes/pradeepk/dev/hive/build/dist/lib/asm-3.1.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-cli-2.0-SNAPSHOT.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-codec-1.3.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-collections-3.2.1.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-dbcp-1.2.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-lang-2.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-logging-1.0.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-logging-api-1.0.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-pool-1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-connectionpool-1.0.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-core-1.1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-enhancer-1.1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-rdbms-1.1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/derby.jar:/homes/pradeepk/dev/hive/build/dist/lib/hbase-0.20.3.jar:/homes/pradeepk/dev/hive/build/dist/lib/hbase-0.20.3-test.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-anttasks-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-cli-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-common-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive_contrib.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-exec-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive_hbase-handler.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-hwi-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-jdbc-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-metastore-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-serde-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-service-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-shims-0.6.0.jar
haha, There is no official way. A few suggestions I have seen
1) do not use nohup use runit svc (daemon tools)
2) use screen
3) start it as its own user (easer to identify)
4) throw some bogus option in the startup like -Dthisishiveserver=true

When calling "bin/hive -service metastore", all the error messages are printed to stdout/err. The command you proposed is a workable approach (but maybe replace > with >> so that it appends, instead of overwriting). Ideally, you would use log4j to configure logging to a file e.g. with an argument like  "-Dlog4j.configuration=configuration_file" when starting the server. But the way that the startup scripts work now, it looks like  you might need to make modifications to bin/ext/metastore.sh to make that happen. The advantage to using log4j is that it can be configured to do daily roll-over logs, so they never get too big.

There's not really a good way to identify the metastore process once it's started - the command you ran is similar to what I use too. One way to identify the process might be to record to a file the PID of the metastore when you start it up -

nohup bin/hive -service metstatore > metastore.log 2>&1 &
echo $! > metastore_pid

From: Pradeep Kamath [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, June 09, 2010 4:11 PM
To: [EMAIL PROTECTED]
Subject: RE: Logs when running metastore

Also a related question is how do I stop the server if have started the server with "nohup bin/hive -service metstatore > metastore.log 2>&1 &" command ? When I run "ps -aef |grep hive", I get :

pradeepk  5463 14236  0 16:00 pts/6    00:00:00 grep hive
pradeepk 27803 14226  0 13:47 pts/5    00:00:04 /usr/java/jdk1.6.0_01/bin/java -Xmx256m -Dhadoop.log.dir=/homes/pradeepk/hadoopcluster/hadoop/bin/../logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/homes/pradeepk/hadoopcluster/hadoop/bin/.. -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/native/Linux-i386-32 -Dhadoop.policy.file=hadoop-policy.xml -classpath /homes/pradeepk/hadoopcluster/hadoop/conf:/usr/java/jdk1.6.0_01/lib/tools.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/..:/homes/pradeepk/hadoopcluster/hadoop/bin/../hadoop-0.20.1-core.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-cli-1.2.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-codec-1.3.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-el-1.0.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-logging-1.0.4.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-net-1.4.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/core-3.1.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jets3t-0.6.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jetty-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jetty-util-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/junit-3.8.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/kfs-0.2.2.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/log4j-1.2.15.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/oro-2.0.8.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/servlet-api-2.5-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/slf4j-api-1.4.3.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/xmlenc-0.52.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:/homes/pradeepk/dev/hive/build/dist/conf:/homes/pradeepk/dev/hive/build/dist/lib/antlr-runtime-3.0.1.jar:/homes/pradeepk/dev/hive/build/dist/lib/asm-3.1.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-cli-2.0-SNAPSHOT.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-codec-1.3.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-collections-3.2.1.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-dbcp-1.2.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-lang-2.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-logging-1.0.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-logging-api-1.0.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-pool-1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-connectionpool-1.0.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-core-1.1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-enhancer-1.1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-rdbms-1.1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/derby.jar:/homes/pradeepk/dev/hive/build/dist/lib/hbase-0.20.3.jar:/homes/pradeepk/dev/hive/build/dist/lib/hbase-0.20.3-test.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-anttasks-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-cli-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-common-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive_contrib.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-exec-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive_hbase-handler.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-hwi-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-jdbc-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-metastore-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-serde-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-service-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-shims-0.6.0.jar

Is there a recommended way to identify the metastore server process?

Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, June 09, 2010 3:10 PM
To: [EMAIL PROTECTED]
Subject: Logs when running metastore

Hi,
  Are there any logs created when running the metastore using "bin/hive -service metastore" ? From what I can tell this command generates output on the screen and runs in the foreground. Woul