Bigdata ideas

We have a task with description. Thare is json file (events.json) with events: ... ... {"_t":1507087562,"_p":"sburke@test.com","_n":"app_loaded","device_type":"desktop"} {"_t":1505361824,"_p":"pfitza@test.com","_n":"added_to_team","account":"1234"} {"_t":1505696264,"_p":"keiji@test.com","_n":"registered","channel":"Google_Apps"} ... ... There are about 500000 lines in file (json objects). And only this 3 types of objects possible. _t - is a timestamp _p - email and we can use it as a unique identifier of user. _n - event type (app_loaded - application was loaded by user, registered - user has registered) and last one is additional attribute, device_type, account, channel Task: 1) Load data from json file by events, only app_loaded  and registered in 2 parquet ...

In this article I begin big preparation to finish research work - Using Big Data by police to search serial criminal. Article 1. Dictionaries preparation. We need prepare and load into hive database some new dictionaries- like Districts, road web cams, cars government registration numbers and cars owners information. First of all I download wikipedia table -  List of districts of Bangkok  and save it with Excel as csv file with name bkk_dist_csv.csv Data looks like Download file into Hive table $ hadoop fs -mkdir /user/data/dics  Load with Hadoop WebUI and check  $ hadoop fs -ls /user/data/dics/  Found 1 items -rw-r--r-- 3 dr.who supergroup 1202 2018-04-03 12:01 /user/data/dics/bkk_dist_csv.csv And load it into Hive table drop table IF EXISTS d_src; CREATE EXTERNAL TABLE d_src( code string, eng_name string, thai_Name string ) COMMENT 'source external table for loading scv' ROW FORM...

First of all there is creation of database for studing external tables. Before, look on directory structure $ hadoop fs -ls /user/data Found 6 items -rw-r--r-- 2 hadoop supergroup 161 2018-03-02 14:31 /user/data/cc2.avsc drwxr-xr-x - hadoop supergroup 0 2018-04-03 12:27 /user/data/dics drwxr-xr-x - hadoop supergroup 0 2018-03-15 15:42 /user/data/js1.json drwxr-xr-x - hadoop supergroup 0 2018-03-16 10:52 /user/data/js_db drwxr-xr-x - hadoop supergroup 0 2018-03-02 13:49 /user/data/order.parquet -rw-r--r-- 2 dr.who supergroup 389 2018-02-28 16:10 /user/data/test_data.csv Create new one database with set location folder and setting some properties, like comment and creator. CREATE DATABASE IF NOT EXISTS ext_tabs COMMENT 'Database for studing external tables' LOCATION '/user/data/exttabs' WITH DBPROPERTIES ('creator' = 'Yakushev Aleksey', 'date' = '2018-04-05'); Now che...

previous posts:  hadoop cluster  and  hive installation In this topic we will go through some steps: create local json, load it into hdfs, creation external hive table, queries to this table and etc. First of all create local json file with notepad or vi, name it js1.json and populate with this data: {"ts":1520318907,"device":1,"metric":"p","value":100} {"ts":1520318908,"device":2,"metric":"p","value":110} {"ts":1520318909,"device":1,"metric":"v","value":8} {"ts":1520318910,"device":2,"metric":"v","value":9} {"ts":1520318911,"device":1,"metric":"p","value":120} {"ts":1520318912,"device":2,"metric":"p","value":140} {"ts":1520318913,"device":1,"metric":...

Previous posts about  hadoop 3.0 cluster  and  hive install  used in this article. Used hive version: [hadoop@hadoop-master myhql]$ hive --version Hive 2.3.2 You can see  here about supported file formats (Storage Formats) I gonna just try next 4: SEQUENCEFILE - Stored as compressed Sequence File. ORC - Stored as ORC file format. Supports ACID Transactions & Cost-based Optimizer (CBO). Stores column-level metadata.(ORC - Optimized Row Columnar) PARQUET - Stored as Parquet format for the Parquet columnar storage format. AVRO - Stored as Avro format. Simple script to create and populate table with test data: [hadoop@hadoop-master myhql]$ hostname hadoop-master [hadoop@hadoop-master myhql]$ pwd /opt/hive/myhql [hadoop@hadoop-master myhql]$ ls pop_tables.hql [hadoop@hadoop-master myhql]$ vi pop_tables.hql drop table tmp_seq_part_date; drop table tmp_orc_part_date; drop table tmp_prq_part_date; dr...

previous post ( Install and configure Hadoop 3 cluster ) This cluster NameNode is using in next articles. 1) Download and extract hive binary  cd /opt wget http://apache-mirror.rbc.ru/pub/apache/hive/hive-2.3.2/apache-hive-2.3.2-bin.tar.gz tar -xvf apache-hive-2.3.2-bin.tar.gz mv apache-hive-2.3.2-bin hive chown -R hadoop /opt/hive Configure environment in (hadoop user) home directory edit file .bashrc hadoop# cd ~ vi .bashrc now it looks like: # .bashrc export HADOOP_HOME=/opt/hadoop export HADOOP_INSTALL=$HADOOP_HOME export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin export HIVE_HOME=/opt/hive export PATH=$PATH:$HIVE_HOME/bin ... ... We have added HIVE_HOME and modify PATH. Don't forget execute source .bashrc after exit from edit .bashrc ...

In previous post I show how to setup and configure  hadoop 3.0 simple cluster Now we can read and write into HDFS with Java: Create dir into HDFS (from master node) # hadoop fs -mkdir -p /user/data # hadoop fs -ls /user/data 2018-02-19 18:17:58,784 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform ... using builtin-java classes where applicable HadoopSimple.java package gdev; import java.io.BufferedWriter; import java.io.IOException; import java.io.OutputStream; import java.io.OutputStreamWriter; import org.apache.commons.io.IOUtils; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.fs.FSDataOutputStream; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.util.Progressable; import org.apache.log4j.Logger; import java.net.URI; public class HadoopSimple { final static Logger logger = Logger.getLogger(Hadoop...