Discussion:
[jira] Created: (HBASE-573) HBase does not read hadoop-*.xml for dfs configuration after moving out hadoop/contrib
(too old to reply)
Rong-En Fan (JIRA)
2008-04-11 04:34:04 UTC
Permalink
HBase does not read hadoop-*.xml for dfs configuration after moving out hadoop/contrib
--------------------------------------------------------------------------------------

Key: HBASE-573
URL: https://issues.apache.org/jira/browse/HBASE-573
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.1.0
Reporter: Rong-En Fan
Priority: Blocker


When HBase was in hadoop/contrib, the hbase script set both HADOOP_CONF_DIR
and HBASE_CONF_DIR to CLASSPATH, so that dfs's configuration can be loaded
correctly. However, when moved out hadoop/contrib, it only sets HBASE_CONF_DIR.

I can think of several possible solutions:

1) set HADOOP_CONF_DIR in hbase-env.sh, then add HADOOP_CONF_DIR to CLASSPATH as before
2) Instruct user to create links for hadoop-*.xml if they want to customize some dfs settings.
3) If only a small set of dfs confs are related to dfs's client, maybe they can be set via hbase-site.xml, then hbase sets these for us when create a FileSystem obj.

Please see the thread "# of dfs replications when using hbase" on hbase-***@.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Rong-En Fan (JIRA)
2008-04-11 04:36:05 UTC
Permalink
[ https://issues.apache.org/jira/browse/HBASE-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rong-En Fan updated HBASE-573:
------------------------------

Affects Version/s: 0.1.1
0.1.2
0.2.0
Post by Rong-En Fan (JIRA)
HBase does not read hadoop-*.xml for dfs configuration after moving out hadoop/contrib
--------------------------------------------------------------------------------------
Key: HBASE-573
URL: https://issues.apache.org/jira/browse/HBASE-573
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.2.0, 0.1.2, 0.1.1, 0.1.0
Reporter: Rong-En Fan
Priority: Blocker
When HBase was in hadoop/contrib, the hbase script set both HADOOP_CONF_DIR
and HBASE_CONF_DIR to CLASSPATH, so that dfs's configuration can be loaded
correctly. However, when moved out hadoop/contrib, it only sets HBASE_CONF_DIR.
1) set HADOOP_CONF_DIR in hbase-env.sh, then add HADOOP_CONF_DIR to CLASSPATH as before
2) Instruct user to create links for hadoop-*.xml if they want to customize some dfs settings.
3) If only a small set of dfs confs are related to dfs's client, maybe they can be set via hbase-site.xml, then hbase sets these for us when create a FileSystem obj.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
stack (JIRA)
2008-04-11 05:10:11 UTC
Permalink
[ https://issues.apache.org/jira/browse/HBASE-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-573:
------------------------

Fix Version/s: 0.1.2
0.2.0

Lets address in 0.1.2 (as well as TRUNK)
Post by Rong-En Fan (JIRA)
HBase does not read hadoop-*.xml for dfs configuration after moving out hadoop/contrib
--------------------------------------------------------------------------------------
Key: HBASE-573
URL: https://issues.apache.org/jira/browse/HBASE-573
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.2.0, 0.1.2, 0.1.1, 0.1.0
Reporter: Rong-En Fan
Priority: Blocker
Fix For: 0.2.0, 0.1.2
When HBase was in hadoop/contrib, the hbase script set both HADOOP_CONF_DIR
and HBASE_CONF_DIR to CLASSPATH, so that dfs's configuration can be loaded
correctly. However, when moved out hadoop/contrib, it only sets HBASE_CONF_DIR.
1) set HADOOP_CONF_DIR in hbase-env.sh, then add HADOOP_CONF_DIR to CLASSPATH as before
2) Instruct user to create links for hadoop-*.xml if they want to customize some dfs settings.
3) If only a small set of dfs confs are related to dfs's client, maybe they can be set via hbase-site.xml, then hbase sets these for us when create a FileSystem obj.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
R. James Firby
2008-04-11 20:06:26 UTC
Permalink
Let's be careful with this one.

We run hadoop with non-default configs that live in a different directory from where hbase and its configs are installed. For us, a really nice thing would be to have a value we can set in the hbase config that would point to the proper hadoop config to use.

Slurping up whatever hadoop config happens to be on the classpath may not be correct and will definitely add confusion to the config process.

Jim
stack
2008-04-11 20:43:23 UTC
Permalink
Post by R. James Firby
Let's be careful with this one.
We run hadoop with non-default configs that live in a different directory from where hbase and its configs are installed. For us, a really nice thing would be to have a value we can set in the hbase config that would point to the proper hadoop config to use.
Does adding wherever HADOOP_CONF_DIR points to the HBASE_CLASSPATH in
hbase-env.sh work for you Jim? Or would you like something else?
Thanks,
St.Ack
R. James Firby
2008-04-14 16:48:00 UTC
Permalink
Yes, I guess that would work. We actually split our hadoop configs into two directories, one for the dfs and one for map/reduce so we can run multiple map/reduce clusters on the same set of machines that are all running the dfs.

So, we could just set HADOOP_CONF_DIR in hbase-env.sh to point to the dfs config directory and all would be well. Right?

Jim
Post by R. James Firby
Let's be careful with this one.
We run hadoop with non-default configs that live in a different directory from where hbase and its configs are installed. For us, a really nice thing would be to have a value we can set in the hbase config that would point to the proper hadoop config to use.
Does adding wherever HADOOP_CONF_DIR points to the HBASE_CLASSPATH in
hbase-env.sh work for you Jim? Or would you like something else?
Thanks,
St.Ack
stack
2008-04-14 16:59:27 UTC
Permalink
Yes.
St.Ack
Post by R. James Firby
Yes, I guess that would work. We actually split our hadoop configs into two directories, one for the dfs and one for map/reduce so we can run multiple map/reduce clusters on the same set of machines that are all running the dfs.
So, we could just set HADOOP_CONF_DIR in hbase-env.sh to point to the dfs config directory and all would be well. Right?
Jim
Post by R. James Firby
Let's be careful with this one.
We run hadoop with non-default configs that live in a different directory from where hbase and its configs are installed. For us, a really nice thing would be to have a value we can set in the hbase config that would point to the proper hadoop config to use.
Does adding wherever HADOOP_CONF_DIR points to the HBASE_CLASSPATH in
hbase-env.sh work for you Jim? Or would you like something else?
Thanks,
St.Ack
stack (JIRA)
2008-04-14 16:23:05 UTC
Permalink
[ https://issues.apache.org/jira/browse/HBASE-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack reassigned HBASE-573:
---------------------------

Assignee: stack
Post by Rong-En Fan (JIRA)
HBase does not read hadoop-*.xml for dfs configuration after moving out hadoop/contrib
--------------------------------------------------------------------------------------
Key: HBASE-573
URL: https://issues.apache.org/jira/browse/HBASE-573
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.2.0, 0.1.2, 0.1.1, 0.1.0
Reporter: Rong-En Fan
Assignee: stack
Priority: Blocker
Fix For: 0.2.0, 0.1.2
When HBase was in hadoop/contrib, the hbase script set both HADOOP_CONF_DIR
and HBASE_CONF_DIR to CLASSPATH, so that dfs's configuration can be loaded
correctly. However, when moved out hadoop/contrib, it only sets HBASE_CONF_DIR.
1) set HADOOP_CONF_DIR in hbase-env.sh, then add HADOOP_CONF_DIR to CLASSPATH as before
2) Instruct user to create links for hadoop-*.xml if they want to customize some dfs settings.
3) If only a small set of dfs confs are related to dfs's client, maybe they can be set via hbase-site.xml, then hbase sets these for us when create a FileSystem obj.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
stack (JIRA)
2008-04-14 20:39:04 UTC
Permalink
[ https://issues.apache.org/jira/browse/HBASE-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack resolved HBASE-573.
-------------------------

Resolution: Fixed

Committed doc. to branch, trunk and to FAQ.
Post by Rong-En Fan (JIRA)
HBase does not read hadoop-*.xml for dfs configuration after moving out hadoop/contrib
--------------------------------------------------------------------------------------
Key: HBASE-573
URL: https://issues.apache.org/jira/browse/HBASE-573
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.2.0, 0.1.2, 0.1.0, 0.1.1
Reporter: Rong-En Fan
Assignee: stack
Priority: Blocker
Fix For: 0.2.0, 0.1.2
When HBase was in hadoop/contrib, the hbase script set both HADOOP_CONF_DIR
and HBASE_CONF_DIR to CLASSPATH, so that dfs's configuration can be loaded
correctly. However, when moved out hadoop/contrib, it only sets HBASE_CONF_DIR.
1) set HADOOP_CONF_DIR in hbase-env.sh, then add HADOOP_CONF_DIR to CLASSPATH as before
2) Instruct user to create links for hadoop-*.xml if they want to customize some dfs settings.
3) If only a small set of dfs confs are related to dfs's client, maybe they can be set via hbase-site.xml, then hbase sets these for us when create a FileSystem obj.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Loading...