2019-12-20 (Friday)
抖音上很火的表白源码分享----大萝卜博客网


下载地址:https://www.lanzous.com/i1g26za

大萝卜 132
评论 0

2019-12-19 (Thursday)
HBASE 异常 java.lang.ClassCastException: org.apache.hadoop.hbase.client.Put cannot be cast to org.apache.hadoop.hbase.KeyValue

java.lang.ClassCastException: org.apache.hadoop.hbase.client.Put cannot be cast to org.apache.hadoop.hbase.KeyValue

下面我们来看报错信息

WARN mapred.LocalJobRunner: job_local244116549_0001
java.lang.ClassCastException: org.apache.hadoop.hbase.client.Put cannot be cast to org.apache.hadoop.hbase.KeyValue
    at org.apache.hadoop.hbase.mapreduce.KeyValueSortReducer.reduce(KeyValueSortReducer.java:43)
    at org.apache.hadoop.hbase.mapreduce.KeyValueSortReducer.reduce(KeyValueSortReducer.java:36)
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:164)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:610)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:444)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:449)

异常解决:
这个问题的原因是

HFileOutputFormat2.configureIncrementalLoad(job, myuser, myuserRegion);

在Dirver的这段代码之前要将job的输入路径,输出路径,输出数据类型要先设置好,如果没有设置好输出类型,就会爆出警告,且没有结果数据输出

大萝卜 155
评论 0

2019-12-17 (Tuesday)
HBASE的Java API操作手册---大萝卜博客网整理

[Meting]
[Music server="netease" id="1338388700" type="song"/]
[/Meting]

我们使用Maven来搭建项目,首先导入pom文件

<repositories>
        <repository>
            <id>cloudera</id>
            <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
        </repository>
    </repositories>

    <dependencies>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.6.0-mr1-cdh5.14.0</version>
        </dependency>


        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>1.2.0-cdh5.14.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-server</artifactId>
            <version>1.2.0-cdh5.14.0</version>
        </dependency>


        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>6.14.3</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                    <!--    <verbal>true</verbal>-->
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.2</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*/RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

表的操作实例

  • 创建一个表 名字是myuser
       Configuration configuration = HBaseConfiguration.create();
        configuration.set("hbase.zookeeper.property.clientPort", "2181");
        configuration.set("hbase.zookeeper.quorum", "hadoop01,hadoop02,hadoop03");

        configuration.set("hbase.master", "hadoop01:60000");

        Connection connection = ConnectionFactory.createConnection(configuration);
        Admin admin = connection.getAdmin();

        //通过HTableDescriptor来实现我们表的参数设置,包括表名,列族等等
        HTableDescriptor user = new HTableDescriptor(TableName.valueOf("myuser"));
        //添加列族
        user.addFamily(new HColumnDescriptor("info"));
        user.addFamily(new HColumnDescriptor("data"));
        //创建表
        if (!admin.tableExists(TableName.valueOf("myuser"))) {
            admin.createTable(user);
        }
        admin.close();
  • 创建一个表 名字是myuser
        //获取config对象
        Configuration configuration = HBaseConfiguration.create();
        configuration.set("hbase.zookeeper.quorum", "hadoop01:2181,hadoop02:2181,hadoop03:2181");
        //获取连接对象
        Connection connection = ConnectionFactory.createConnection(configuration);
        //获取表
        Table myuser = connection.getTable(TableName.valueOf("myuser"));
        //创建put对象,并指定rowkey
        Put put = new Put("0001".getBytes());
        put.addColumn("f1".getBytes(),"id".getBytes(), Bytes.toBytes(1));
        put.addColumn("f1".getBytes(),"name".getBytes(), Bytes.toBytes("张三"));
        put.addColumn("f1".getBytes(),"age".getBytes(), Bytes.toBytes(18));

        put.addColumn("f2".getBytes(),"address".getBytes(), Bytes.toBytes("地球人"));
        put.addColumn("f2".getBytes(),"phone".getBytes(), Bytes.toBytes("15874102589"));
        //插入数据
        myuser.put(put);
        //关闭表
        myuser.close();
  • 查询数据
 Configuration configuration = HBaseConfiguration.create();
        configuration.set("hbase.zookeeper.quorum","hadoop01,hadoop02,hadoop03");
        Connection connection = ConnectionFactory.createConnection(configuration);
        Table myuser = connection.getTable(TableName.valueOf("myuser"));
        //创建put对象,并指定rowkey
        Put put = new Put("0002".getBytes());
        put.addColumn("info".getBytes(),"id".getBytes(),Bytes.toBytes(1));
        put.addColumn("info".getBytes(),"name".getBytes(),Bytes.toBytes("曹操"));
        put.addColumn("info".getBytes(),"age".getBytes(),Bytes.toBytes(30));
        put.addColumn("data".getBytes(),"sex".getBytes(),Bytes.toBytes("1"));
        put.addColumn("data".getBytes(),"address".getBytes(),Bytes.toBytes("沛国谯县"));
        put.addColumn("data".getBytes(),"phone".getBytes(),Bytes.toBytes("16888888888"));
        put.addColumn("data".getBytes(),"say".getBytes(),Bytes.toBytes("helloworld"));

        Put put2 = new Put("0003".getBytes());
        put2.addColumn("info".getBytes(),"id".getBytes(),Bytes.toBytes(2));
        put2.addColumn("info".getBytes(),"name".getBytes(),Bytes.toBytes("刘备"));
        put2.addColumn("info".getBytes(),"age".getBytes(),Bytes.toBytes(32));
        put2.addColumn("data".getBytes(),"sex".getBytes(),Bytes.toBytes("1"));
        put2.addColumn("data".getBytes(),"address".getBytes(),Bytes.toBytes("幽州涿郡涿县"));
        put2.addColumn("data".getBytes(),"phone".getBytes(),Bytes.toBytes("17888888888"));
        put2.addColumn("data".getBytes(),"say".getBytes(),Bytes.toBytes("talk is cheap , show me the code"));


        Put put3 = new Put("0004".getBytes());
        put3.addColumn("info".getBytes(),"id".getBytes(),Bytes.toBytes(3));
        put3.addColumn("info".getBytes(),"name".getBytes(),Bytes.toBytes("孙权"));
        put3.addColumn("info".getBytes(),"age".getBytes(),Bytes.toBytes(35));
        put3.addColumn("data".getBytes(),"sex".getBytes(),Bytes.toBytes("1"));
        put3.addColumn("data".getBytes(),"address".getBytes(),Bytes.toBytes("下邳"));
        put3.addColumn("data".getBytes(),"phone".getBytes(),Bytes.toBytes("12888888888"));
        put3.addColumn("data".getBytes(),"say".getBytes(),Bytes.toBytes("what are you 弄啥嘞!"));

        Put put4 = new Put("0005".getBytes());
        put4.addColumn("info".getBytes(),"id".getBytes(),Bytes.toBytes(4));
        put4.addColumn("info".getBytes(),"name".getBytes(),Bytes.toBytes("诸葛亮"));
        put4.addColumn("info".getBytes(),"age".getBytes(),Bytes.toBytes(28));
        put4.addColumn("data".getBytes(),"sex".getBytes(),Bytes.toBytes("1"));
        put4.addColumn("data".getBytes(),"address".getBytes(),Bytes.toBytes("四川隆中"));
        put4.addColumn("data".getBytes(),"phone".getBytes(),Bytes.toBytes("14888888888"));
        put4.addColumn("data".getBytes(),"say".getBytes(),Bytes.toBytes("出师表你背了嘛"));

        Put put5 = new Put("0005".getBytes());
        put5.addColumn("info".getBytes(),"id".getBytes(),Bytes.toBytes(5));
        put5.addColumn("info".getBytes(),"name".getBytes(),Bytes.toBytes("司马懿"));
        put5.addColumn("info".getBytes(),"age".getBytes(),Bytes.toBytes(27));
        put5.addColumn("data".getBytes(),"sex".getBytes(),Bytes.toBytes("1"));
        put5.addColumn("data".getBytes(),"address".getBytes(),Bytes.toBytes("哪里人有待考究"));
        put5.addColumn("data".getBytes(),"phone".getBytes(),Bytes.toBytes("15888888888"));
        put5.addColumn("data".getBytes(),"say".getBytes(),Bytes.toBytes("跟诸葛亮死掐"));


        Put put6 = new Put("0006".getBytes());
        put6.addColumn("info".getBytes(),"id".getBytes(),Bytes.toBytes(5));
        put6.addColumn("info".getBytes(),"name".getBytes(),Bytes.toBytes("xiaobubu—吕布"));
        put6.addColumn("info".getBytes(),"age".getBytes(),Bytes.toBytes(28));
        put6.addColumn("data".getBytes(),"sex".getBytes(),Bytes.toBytes("1"));
        put6.addColumn("data".getBytes(),"address".getBytes(),Bytes.toBytes("内蒙人"));
        put6.addColumn("data".getBytes(),"phone".getBytes(),Bytes.toBytes("15788888888"));
        put6.addColumn("data".getBytes(),"say".getBytes(),Bytes.toBytes("貂蝉去哪了"));

        List<Put> listPut = new ArrayList<Put>();
        listPut.add(put);
        listPut.add(put2);
        listPut.add(put3);
        listPut.add(put4);
        listPut.add(put5);
        listPut.add(put6);

        myuser.put(listPut);
        myuser.close();
  • 按照rowkey进行查询获取所有列的所有值
  Configuration configuration = HBaseConfiguration.create();
        configuration.set("hbase.zookeeper.quorum", "hadoop01,hadoop02,hadoop03");
        Connection connection = ConnectionFactory.createConnection(configuration);
        Table myuser = connection.getTable(TableName.valueOf("myuser"));

        Get get = new Get("0002".getBytes());

        Result result = myuser.get(get);
        Cell[] cells = result.rawCells();
        for (Cell cell : cells) {
            if (new String(CellUtil.cloneQualifier(cell)).equals("id")||new String(CellUtil.cloneQualifier(cell)).equals("age")){
                System.out.println(new String(CellUtil.cloneFamily(cell))+":"+new String(CellUtil.cloneQualifier(cell))+"\t"+Bytes.toInt(CellUtil.cloneValue(cell)));
            }else {
                System.out.println(new String(CellUtil.cloneFamily(cell))+":"+new String(CellUtil.cloneQualifier(cell))+"\t"+new String(CellUtil.cloneValue(cell)));
            }
        }
        connection.close();
  • 按照rowkey进行查询获取所有列的所有值
  Configuration configuration = HBaseConfiguration.create();
        configuration.set("hbase.zookeeper.quorum", "hadoop01,hadoop02,hadoop03");
        Connection connection = ConnectionFactory.createConnection(configuration);
        Table myuser = connection.getTable(TableName.valueOf("myuser"));

        Get get = new Get("0002".getBytes());

        Result result = myuser.get(get);
        Cell[] cells = result.rawCells();
        for (Cell cell : cells) {
            if (new String(CellUtil.cloneQualifier(cell)).equals("id")||new String(CellUtil.cloneQualifier(cell)).equals("age")){
                System.out.println(new String(CellUtil.cloneFamily(cell))+":"+new String(CellUtil.cloneQualifier(cell))+"\t"+Bytes.toInt(CellUtil.cloneValue(cell)));
            }else {
                System.out.println(new String(CellUtil.cloneFamily(cell))+":"+new String(CellUtil.cloneQualifier(cell))+"\t"+new String(CellUtil.cloneValue(cell)));
            }
        }
        connection.close();

大萝卜 143
评论 0

2019-12-16 (Monday)
Hbase各个组件以及底层架构详细介绍-大萝卜博客网

简介

hbase是bigtable的开源java版本。是建立在hdfs之上,提供高可靠性、高性能、列存储、可伸缩、实时读写nosql的数据库系统。
它介于nosql和RDBMS之间,仅能通过主键(row key)和主键的range来检索数据,仅支持单行事务(可通过hive支持来实现多表join等复杂操作)。

对比传统数据表

  • 传统数据表:
  • hbase表

HBASE底层原理

继续阅读
大萝卜 114
评论 0

2019-12-12 (Thursday)
HBASE 常用的Shell命令汇总-大萝卜博客网

进入HBase客户端命令操作界面

bin/hbase shell

查看帮助命令

hbase(main):001:0> help

表的增删改

  • 查看当前数据库中有哪些表
list
  • 创建一张表
create 'user', 'info', 'data'
说明: user是表名,info是第一个列族的名字,data是第二个列族的名字
或者
create 'user', {NAME => 'info', VERSIONS => '3'},{NAME => 'data'}
  • 清空表数据
truncate 'user'
  • 删除表
首先需要先让该表为disable状态,使用命令:
disable 'user'
然后才能drop这个表,使用命令:
drop 'user'

(注意:如果直接drop表,会报错:Drop the named table. Table must first be disabled)

继续阅读
大萝卜 164
评论 0
奇趣音乐盒 技术源于 Kico Player
Emmm,这里是歌词君

编写新日记