2019-12-19 (Thursday)
HBASE 异常 java.lang.ClassCastException: org.apache.hadoop.hbase.client.Put cannot be cast to org.apache.hadoop.hbase.KeyValue

java.lang.ClassCastException: org.apache.hadoop.hbase.client.Put cannot be cast to org.apache.hadoop.hbase.KeyValue

下面我们来看报错信息

WARN mapred.LocalJobRunner: job_local244116549_0001
java.lang.ClassCastException: org.apache.hadoop.hbase.client.Put cannot be cast to org.apache.hadoop.hbase.KeyValue
    at org.apache.hadoop.hbase.mapreduce.KeyValueSortReducer.reduce(KeyValueSortReducer.java:43)
    at org.apache.hadoop.hbase.mapreduce.KeyValueSortReducer.reduce(KeyValueSortReducer.java:36)
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:164)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:610)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:444)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:449)

异常解决:
这个问题的原因是

HFileOutputFormat2.configureIncrementalLoad(job, myuser, myuserRegion);

在Dirver的这段代码之前要将job的输入路径,输出路径,输出数据类型要先设置好,如果没有设置好输出类型,就会爆出警告,且没有结果数据输出

大萝卜 250
评论 0

2019-12-17 (Tuesday)
HBASE的Java API操作手册---大萝卜博客网整理

[Meting]
[Music server="netease" id="1338388700" type="song"/]
[/Meting]

我们使用Maven来搭建项目,首先导入pom文件

<repositories>
        <repository>
            <id>cloudera</id>
            <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
        </repository>
    </repositories>

    <dependencies>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.6.0-mr1-cdh5.14.0</version>
        </dependency>


        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>1.2.0-cdh5.14.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-server</artifactId>
            <version>1.2.0-cdh5.14.0</version>
        </dependency>


        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>6.14.3</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                    <!--    <verbal>true</verbal>-->
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.2</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*/RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

表的操作实例

  • 创建一个表 名字是myuser
       Configuration configuration = HBaseConfiguration.create();
        configuration.set("hbase.zookeeper.property.clientPort", "2181");
        configuration.set("hbase.zookeeper.quorum", "hadoop01,hadoop02,hadoop03");

        configuration.set("hbase.master", "hadoop01:60000");

        Connection connection = ConnectionFactory.createConnection(configuration);
        Admin admin = connection.getAdmin();

        //通过HTableDescriptor来实现我们表的参数设置,包括表名,列族等等
        HTableDescriptor user = new HTableDescriptor(TableName.valueOf("myuser"));
        //添加列族
        user.addFamily(new HColumnDescriptor("info"));
        user.addFamily(new HColumnDescriptor("data"));
        //创建表
        if (!admin.tableExists(TableName.valueOf("myuser"))) {
            admin.createTable(user);
        }
        admin.close();
  • 创建一个表 名字是myuser
        //获取config对象
        Configuration configuration = HBaseConfiguration.create();
        configuration.set("hbase.zookeeper.quorum", "hadoop01:2181,hadoop02:2181,hadoop03:2181");
        //获取连接对象
        Connection connection = ConnectionFactory.createConnection(configuration);
        //获取表
        Table myuser = connection.getTable(TableName.valueOf("myuser"));
        //创建put对象,并指定rowkey
        Put put = new Put("0001".getBytes());
        put.addColumn("f1".getBytes(),"id".getBytes(), Bytes.toBytes(1));
        put.addColumn("f1".getBytes(),"name".getBytes(), Bytes.toBytes("张三"));
        put.addColumn("f1".getBytes(),"age".getBytes(), Bytes.toBytes(18));

        put.addColumn("f2".getBytes(),"address".getBytes(), Bytes.toBytes("地球人"));
        put.addColumn("f2".getBytes(),"phone".getBytes(), Bytes.toBytes("15874102589"));
        //插入数据
        myuser.put(put);
        //关闭表
        myuser.close();
  • 查询数据
 Configuration configuration = HBaseConfiguration.create();
        configuration.set("hbase.zookeeper.quorum","hadoop01,hadoop02,hadoop03");
        Connection connection = ConnectionFactory.createConnection(configuration);
        Table myuser = connection.getTable(TableName.valueOf("myuser"));
        //创建put对象,并指定rowkey
        Put put = new Put("0002".getBytes());
        put.addColumn("info".getBytes(),"id".getBytes(),Bytes.toBytes(1));
        put.addColumn("info".getBytes(),"name".getBytes(),Bytes.toBytes("曹操"));
        put.addColumn("info".getBytes(),"age".getBytes(),Bytes.toBytes(30));
        put.addColumn("data".getBytes(),"sex".getBytes(),Bytes.toBytes("1"));
        put.addColumn("data".getBytes(),"address".getBytes(),Bytes.toBytes("沛国谯县"));
        put.addColumn("data".getBytes(),"phone".getBytes(),Bytes.toBytes("16888888888"));
        put.addColumn("data".getBytes(),"say".getBytes(),Bytes.toBytes("helloworld"));

        Put put2 = new Put("0003".getBytes());
        put2.addColumn("info".getBytes(),"id".getBytes(),Bytes.toBytes(2));
        put2.addColumn("info".getBytes(),"name".getBytes(),Bytes.toBytes("刘备"));
        put2.addColumn("info".getBytes(),"age".getBytes(),Bytes.toBytes(32));
        put2.addColumn("data".getBytes(),"sex".getBytes(),Bytes.toBytes("1"));
        put2.addColumn("data".getBytes(),"address".getBytes(),Bytes.toBytes("幽州涿郡涿县"));
        put2.addColumn("data".getBytes(),"phone".getBytes(),Bytes.toBytes("17888888888"));
        put2.addColumn("data".getBytes(),"say".getBytes(),Bytes.toBytes("talk is cheap , show me the code"));


        Put put3 = new Put("0004".getBytes());
        put3.addColumn("info".getBytes(),"id".getBytes(),Bytes.toBytes(3));
        put3.addColumn("info".getBytes(),"name".getBytes(),Bytes.toBytes("孙权"));
        put3.addColumn("info".getBytes(),"age".getBytes(),Bytes.toBytes(35));
        put3.addColumn("data".getBytes(),"sex".getBytes(),Bytes.toBytes("1"));
        put3.addColumn("data".getBytes(),"address".getBytes(),Bytes.toBytes("下邳"));
        put3.addColumn("data".getBytes(),"phone".getBytes(),Bytes.toBytes("12888888888"));
        put3.addColumn("data".getBytes(),"say".getBytes(),Bytes.toBytes("what are you 弄啥嘞!"));

        Put put4 = new Put("0005".getBytes());
        put4.addColumn("info".getBytes(),"id".getBytes(),Bytes.toBytes(4));
        put4.addColumn("info".getBytes(),"name".getBytes(),Bytes.toBytes("诸葛亮"));
        put4.addColumn("info".getBytes(),"age".getBytes(),Bytes.toBytes(28));
        put4.addColumn("data".getBytes(),"sex".getBytes(),Bytes.toBytes("1"));
        put4.addColumn("data".getBytes(),"address".getBytes(),Bytes.toBytes("四川隆中"));
        put4.addColumn("data".getBytes(),"phone".getBytes(),Bytes.toBytes("14888888888"));
        put4.addColumn("data".getBytes(),"say".getBytes(),Bytes.toBytes("出师表你背了嘛"));

        Put put5 = new Put("0005".getBytes());
        put5.addColumn("info".getBytes(),"id".getBytes(),Bytes.toBytes(5));
        put5.addColumn("info".getBytes(),"name".getBytes(),Bytes.toBytes("司马懿"));
        put5.addColumn("info".getBytes(),"age".getBytes(),Bytes.toBytes(27));
        put5.addColumn("data".getBytes(),"sex".getBytes(),Bytes.toBytes("1"));
        put5.addColumn("data".getBytes(),"address".getBytes(),Bytes.toBytes("哪里人有待考究"));
        put5.addColumn("data".getBytes(),"phone".getBytes(),Bytes.toBytes("15888888888"));
        put5.addColumn("data".getBytes(),"say".getBytes(),Bytes.toBytes("跟诸葛亮死掐"));


        Put put6 = new Put("0006".getBytes());
        put6.addColumn("info".getBytes(),"id".getBytes(),Bytes.toBytes(5));
        put6.addColumn("info".getBytes(),"name".getBytes(),Bytes.toBytes("xiaobubu—吕布"));
        put6.addColumn("info".getBytes(),"age".getBytes(),Bytes.toBytes(28));
        put6.addColumn("data".getBytes(),"sex".getBytes(),Bytes.toBytes("1"));
        put6.addColumn("data".getBytes(),"address".getBytes(),Bytes.toBytes("内蒙人"));
        put6.addColumn("data".getBytes(),"phone".getBytes(),Bytes.toBytes("15788888888"));
        put6.addColumn("data".getBytes(),"say".getBytes(),Bytes.toBytes("貂蝉去哪了"));

        List<Put> listPut = new ArrayList<Put>();
        listPut.add(put);
        listPut.add(put2);
        listPut.add(put3);
        listPut.add(put4);
        listPut.add(put5);
        listPut.add(put6);

        myuser.put(listPut);
        myuser.close();
  • 按照rowkey进行查询获取所有列的所有值
  Configuration configuration = HBaseConfiguration.create();
        configuration.set("hbase.zookeeper.quorum", "hadoop01,hadoop02,hadoop03");
        Connection connection = ConnectionFactory.createConnection(configuration);
        Table myuser = connection.getTable(TableName.valueOf("myuser"));

        Get get = new Get("0002".getBytes());

        Result result = myuser.get(get);
        Cell[] cells = result.rawCells();
        for (Cell cell : cells) {
            if (new String(CellUtil.cloneQualifier(cell)).equals("id")||new String(CellUtil.cloneQualifier(cell)).equals("age")){
                System.out.println(new String(CellUtil.cloneFamily(cell))+":"+new String(CellUtil.cloneQualifier(cell))+"\t"+Bytes.toInt(CellUtil.cloneValue(cell)));
            }else {
                System.out.println(new String(CellUtil.cloneFamily(cell))+":"+new String(CellUtil.cloneQualifier(cell))+"\t"+new String(CellUtil.cloneValue(cell)));
            }
        }
        connection.close();
  • 按照rowkey进行查询获取所有列的所有值
  Configuration configuration = HBaseConfiguration.create();
        configuration.set("hbase.zookeeper.quorum", "hadoop01,hadoop02,hadoop03");
        Connection connection = ConnectionFactory.createConnection(configuration);
        Table myuser = connection.getTable(TableName.valueOf("myuser"));

        Get get = new Get("0002".getBytes());

        Result result = myuser.get(get);
        Cell[] cells = result.rawCells();
        for (Cell cell : cells) {
            if (new String(CellUtil.cloneQualifier(cell)).equals("id")||new String(CellUtil.cloneQualifier(cell)).equals("age")){
                System.out.println(new String(CellUtil.cloneFamily(cell))+":"+new String(CellUtil.cloneQualifier(cell))+"\t"+Bytes.toInt(CellUtil.cloneValue(cell)));
            }else {
                System.out.println(new String(CellUtil.cloneFamily(cell))+":"+new String(CellUtil.cloneQualifier(cell))+"\t"+new String(CellUtil.cloneValue(cell)));
            }
        }
        connection.close();

大萝卜 219
评论 0

2019-12-16 (Monday)
Hbase各个组件以及底层架构详细介绍-大萝卜博客网

简介

hbase是bigtable的开源java版本。是建立在hdfs之上,提供高可靠性、高性能、列存储、可伸缩、实时读写nosql的数据库系统。
它介于nosql和RDBMS之间,仅能通过主键(row key)和主键的range来检索数据,仅支持单行事务(可通过hive支持来实现多表join等复杂操作)。

对比传统数据表

  • 传统数据表:
  • hbase表

HBASE底层原理

继续阅读
大萝卜 195
评论 0

2019-12-12 (Thursday)
HBASE 常用的Shell命令汇总-大萝卜博客网

进入HBase客户端命令操作界面

bin/hbase shell

查看帮助命令

hbase(main):001:0> help

表的增删改

  • 查看当前数据库中有哪些表
list
  • 创建一张表
create 'user', 'info', 'data'
说明: user是表名,info是第一个列族的名字,data是第二个列族的名字
或者
create 'user', {NAME => 'info', VERSIONS => '3'},{NAME => 'data'}
  • 清空表数据
truncate 'user'
  • 删除表
首先需要先让该表为disable状态,使用命令:
disable 'user'
然后才能drop这个表,使用命令:
drop 'user'

(注意:如果直接drop表,会报错:Drop the named table. Table must first be disabled)

继续阅读
大萝卜 263
评论 0

2019-12-11 (Wednesday)
CentOS HBASE的安装部署

  • 1.软件包的上传解压
    解压命令:
tar -zxvf hbase-1.2.0-cdh5.14.0.tar.gz -C /export/install/
  • 2.修改配置文件
    第一个配置文件:vim conf/hbase-env.sh

将第27行的java_home改成自己的

将第128行的配置的true改为false 这步是为了让HBASE使用自己的Zookeeper,而不是用自带的

第二个配置文件:vim conf/hbase-site.xml

<configuration>
        <property>
                <name>hbase.rootdir</name>
                <value>hdfs://node01:8020/hbase</value>  
        </property>

        <property>
                <name>hbase.cluster.distributed</name>
                <value>true</value>
        </property>

   <!-- 0.98后的新变动,之前版本没有.port,默认端口为60000 -->
        <property>
                <name>hbase.master.port</name>
                <value>16000</value>
        </property>

        <property>
                <name>hbase.zookeeper.quorum</name>
                <value>node01:2181,node02:2181,node03:2181</value>
        </property>

        <property>
                <name>hbase.zookeeper.property.dataDir</name>
         <value>/export/servers/zookeeper-3.4.5-cdh5.14.0/zkdatas</value>
        </property>
</configuration>

第三个配置:vim conf/regionservers
在里面填入工作者节点的ip

第四个配置:vim conf/backup-masters
在里面填入管理者节点的ip(可以有多个,实现了高可用,某一台宕机的时候,其他的节点通过Zookeeper的选举机制,选出一个新的管理节点,真正工作的只有一个)

第五个配置: 将hadoop的配置文件下的core-site.xmlhdfs-site.xml拷贝到HBASE下的conf目录下
下面是最终的HBASE安装目录下conf目录的结构

第六步:将HBASE分发给其他的节点
第七步:配置环境变量

vim /etc/profile.d/hbase.sh
export HBASE_HOME=/export/install/hbase-1.2.0-cdh5.14.0
export PATH=$PATH:$HBASE_HOME/bin


之后不要忘了source /etc/profile 哦~

  • 启动HBASE

首先要确保hadoop集群和Zookeeper已经启动

继续阅读
大萝卜 116
评论 2
奇趣音乐盒 技术源于 Kico Player
Emmm,这里是歌词君

编写新日记