博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
Hadoop部署配置文件
阅读量:4365 次
发布时间:2019-06-07

本文共 9459 字,大约阅读时间需要 31 分钟。

为了方便大家修,我把要修改的地方标红了,找到对应文件,复制粘贴过去就可以了

注:这个是我安装的Hadoop的配置,要根据我之前发的文章配置才行,

里面有一些东西比如文件夹名称,路径不一样,稍加修改也可以用。

配置文件在/hadoop-2.5.2/etc/hadoop/下

core-site.xml

fs.defaultFS
hdfs://master:9000
hadoop.tmp.dir
/home/zkpk/hadoopdata

hdfs-site.xml

dfs.replication
1
mapred-site.xml.template

mapreduce.framework.name
yarn
yarn-xite.xml

yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.resourcemanager.address
master:18040
yarn.resourcemanager.scheduler.address
master:18030
yarn.resourcemanager.resource-tracker.address
master:18025
yarn.resourcemanager.admin.address
master:18041
yarn.resourcemanager.webapp.address
master:18088
hadoop-env.sh

这里就把Java的路径改一下就行

# Licensed to the Apache Software Foundation (ASF) under one# or more contributor license agreements.  See the NOTICE file# distributed with this work for additional information# regarding copyright ownership.  The ASF licenses this file# to you under the Apache License, Version 2.0 (the# "License"); you may not use this file except in compliance# with the License.  You may obtain a copy of the License at##     http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.# Set Hadoop-specific environment variables here.# The only required environment variable is JAVA_HOME.  All others are# optional.  When running a distributed configuration it is best to# set JAVA_HOME in this file, so that it is correctly defined on# remote nodes.# The java implementation to use.export JAVA_HOME=/usr/java/jdk1.7.0_71/    //java的安装路径# The jsvc implementation to use. Jsvc is required to run secure datanodes.#export JSVC_HOME=${JSVC_HOME}export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}# Extra Java CLASSPATH elements.  Automatically insert capacity-scheduler.for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do  if [ "$HADOOP_CLASSPATH" ]; then    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f  else    export HADOOP_CLASSPATH=$f  fidone# The maximum amount of heap to use, in MB. Default is 1000.#export HADOOP_HEAPSIZE=#export HADOOP_NAMENODE_INIT_HEAPSIZE=""# Extra Java runtime options.  Empty by default.export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"# Command specific options appended to HADOOP_OPTS when specifiedexport HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS"export HADOOP_PORTMAP_OPTS="-Xmx512m $HADOOP_PORTMAP_OPTS"# The following applies to multiple commands (fs, dfs, fsck, distcp etc)export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"#HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData $HADOOP_JAVA_PLATFORM_OPTS"# On secure datanodes, user to run the datanode as after dropping privilegesexport HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER}# Where log files are stored.  $HADOOP_HOME/logs by default.#export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER# Where log files are stored in the secure data environment.export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}# The directory where pid files are stored. /tmp by default.# NOTE: this should be set to a directory that can only be written to by #       the user that will run the hadoop daemons.  Otherwise there is the#       potential for a symlink attack.export HADOOP_PID_DIR=${HADOOP_PID_DIR}export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}# A string representing this instance of hadoop. $USER by default.export HADOOP_IDENT_STRING=$USER
yarn-env.sh
# Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements.  See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License.  You may obtain a copy of the License at##     http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.# User for YARN daemonsexport HADOOP_YARN_USER=${HADOOP_YARN_USER:-yarn}# resolve links - $0 may be a softlinkexport YARN_CONF_DIR="${YARN_CONF_DIR:-$HADOOP_YARN_HOME/conf}"# some Java parametersexport JAVA_HOME=/usr/java/jdk1.7.0_71/   //java安装路径if [ "$JAVA_HOME" != "" ]; then  #echo "run java in $JAVA_HOME"  JAVA_HOME=$JAVA_HOMEfi  if [ "$JAVA_HOME" = "" ]; then  echo "Error: JAVA_HOME is not set."  exit 1fiJAVA=$JAVA_HOME/bin/javaJAVA_HEAP_MAX=-Xmx1000m # For setting YARN specific HEAP sizes please use this# Parameter and set appropriately# YARN_HEAPSIZE=1000# check envvars which might override default argsif [ "$YARN_HEAPSIZE" != "" ]; then  JAVA_HEAP_MAX="-Xmx""$YARN_HEAPSIZE""m"fi# Resource Manager specific parameters# Specify the max Heapsize for the ResourceManager using a numerical value# in the scale of MB. For example, to specify an jvm option of -Xmx1000m, set# the value to 1000.# This value will be overridden by an Xmx setting specified in either YARN_OPTS# and/or YARN_RESOURCEMANAGER_OPTS.# If not specified, the default value will be picked from either YARN_HEAPMAX# or JAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.#export YARN_RESOURCEMANAGER_HEAPSIZE=1000# Specify the max Heapsize for the timeline server using a numerical value# in the scale of MB. For example, to specify an jvm option of -Xmx1000m, set# the value to 1000.# This value will be overridden by an Xmx setting specified in either YARN_OPTS# and/or YARN_TIMELINESERVER_OPTS.# If not specified, the default value will be picked from either YARN_HEAPMAX# or JAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.#export YARN_TIMELINESERVER_HEAPSIZE=1000# Specify the JVM options to be used when starting the ResourceManager.# These options will be appended to the options specified as YARN_OPTS# and therefore may override any similar flags set in YARN_OPTS#export YARN_RESOURCEMANAGER_OPTS=# Node Manager specific parameters# Specify the max Heapsize for the NodeManager using a numerical value# in the scale of MB. For example, to specify an jvm option of -Xmx1000m, set# the value to 1000.# This value will be overridden by an Xmx setting specified in either YARN_OPTS# and/or YARN_NODEMANAGER_OPTS.# If not specified, the default value will be picked from either YARN_HEAPMAX# or JAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.#export YARN_NODEMANAGER_HEAPSIZE=1000# Specify the JVM options to be used when starting the NodeManager.# These options will be appended to the options specified as YARN_OPTS# and therefore may override any similar flags set in YARN_OPTS#export YARN_NODEMANAGER_OPTS=# so that filenames w/ spaces are handled correctly in loops belowIFS=# default log directory & fileif [ "$YARN_LOG_DIR" = "" ]; then  YARN_LOG_DIR="$HADOOP_YARN_HOME/logs"fiif [ "$YARN_LOGFILE" = "" ]; then  YARN_LOGFILE='yarn.log'fi# default policy file for service-level authorizationif [ "$YARN_POLICYFILE" = "" ]; then  YARN_POLICYFILE="hadoop-policy.xml"fi# restore ordinary behaviourunset IFSYARN_OPTS="$YARN_OPTS -Dhadoop.log.dir=$YARN_LOG_DIR"YARN_OPTS="$YARN_OPTS -Dyarn.log.dir=$YARN_LOG_DIR"YARN_OPTS="$YARN_OPTS -Dhadoop.log.file=$YARN_LOGFILE"YARN_OPTS="$YARN_OPTS -Dyarn.log.file=$YARN_LOGFILE"YARN_OPTS="$YARN_OPTS -Dyarn.home.dir=$YARN_COMMON_HOME"YARN_OPTS="$YARN_OPTS -Dyarn.id.str=$YARN_IDENT_STRING"YARN_OPTS="$YARN_OPTS -Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"YARN_OPTS="$YARN_OPTS -Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then  YARN_OPTS="$YARN_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"fi  YARN_OPTS="$YARN_OPTS -Dyarn.policy.file=$YARN_POLICYFILE"

转载于:https://www.cnblogs.com/beeman/p/7776491.html

你可能感兴趣的文章
如何去除交叉表的子行(列)的小计?
查看>>
Web字体(链接)嵌入
查看>>
switch… case 语句的用法
查看>>
day07补充-数据类型总结及拷贝
查看>>
语言、数据和运算符
查看>>
正则表达式30分钟入门教程
查看>>
sqlserver try catch·
查看>>
1028: 可乐(2018年中南大学研究生复试机试题 )
查看>>
珍藏的最全的windows操作系统快捷键
查看>>
【DBAplus】SQL优化:一篇文章说清楚Oracle Hint的正确使用姿势
查看>>
二叉树结点删除操作
查看>>
图论-单源最短路-SPFA算法
查看>>
转换文件的字符集
查看>>
软件质量理解
查看>>
jquery 在 table 中修改某行值
查看>>
pyc文件是什么【转载】
查看>>
POM.xml 标签详解
查看>>
hdu 3635 Dragon Balls (并查集)
查看>>
文件操作
查看>>
7.java集合,泛型简单总结,IO流
查看>>