Extended Topic In Hive

 

TABLES

  • Internal Table/Managed Table
      • When table is dropped it deletes both schema and data from HDFS.
      • It is managed by hive
      • The queries will be done through Hive.
      • CREATE TABLE TBL_NAME(CLM DTYPE,....);
  • External Table
    • When table is dropped the schema(structure) will be deleted but not data in HDFS.
    • The metadata will be managed by hive.
    • It is used when files are in remote machine.
    • CREATE EXTERNAL TABLE TBL_NAME(CLM DTYPE,..);

Hive Partition

  • It is used to split the large tables into several tables based on one or more columns.
  • Static partition needs destination
  • Dynamic partition doesn't needs destination because hive internally takes care.
  • It is used when less number of columns are present.

Hive Bucket

  • It is similar to partitioning but with an extra function of hash function added to it.
  • It is generally used for large datasets.
  • If there is no proper selection of bucket count the performance will be decreased
  • CREATE TABLE TBL_NAME(CLM DTYPE,..) CLUSTERED BY (CLM) INTO 10 BUCKETS ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE;
  • 2^(n)>=(Size of file/block size(128)) where n=no of buckets
  • In general bucket count changes when the volume is changed. It is performed in less number of times as it's time consuming.
  • If bucket change is not performed then it may affect the performance.
  • As we can change the bucket size after creating the table we will overwrite with new bucket size into new table.
  • If sub-partition is not possible go for bucketing.
  • It is used when more number of columns are present.

Hive ORC(Optimized Row Column) File Format

  • The file will get decreased it size up to 75% in this format.
  • Performance increases.
  • Data stores in column format.
  • Hive Acid must be internal and orc type.
  • Parquet format has highest optimization(62%).

Hive ACID(Atomicity Consistency Isolation Durability)

  • Entered in 0.14 Version.
  • A table enables insert ,update and delete.
  • It should be internal table.
  • It should be used only orc file type.
  • It must be bucketed table.
  • Transactional is true.

Comments

Post a Comment

Popular posts from this blog

MR(Map Reduce)

Image
Introduction to Map Reduce It's a massive parallel processing framework In general without distributed data it's hard to do parallel processing. Map Reduce uses Java by default The aim of Map Reduce is to achieve data locality and Parallelism. Spark is an alternative for Map Reduce(functionality has 80% similarity) Hive, Sqoop, Pig, Oozie are abstract of Map Reduce. Daemons in Map Reduce are JT and TT. In map reduce we also have two functionalities called Mapper and Reducer which are responsible for achieving the processing. The MR has an disadvantage of Resource Pressure. Advantages of Map Reduce Cluster Monitoring Cluster Management Resource Allocating Scheduling Execution Speculative Execution(L) Working Principle of Map Reduce The task for the MR is given in the form of ZAR(java) file to the Job Tracker. The Job Tracker will be sending the request to the Name Node and in return it gives response. Then the Job Tracker will be sending the task information to Task Tracker. The
Read more

Spark Yarn Cluster

Image
  Spark Submit Cluster(Yarn)->(Hadoop+Spark) Spark Lens Integration is a profiling tool to check the memory allocation(executor memory, etc.) Spark has Hash Partition which is the same algorithm used by MapReduce also. Map(Parallelism which acts as input), Reduce(Aggregation which acts as output) No of blocks=No of tasks when it is HDFS. No of O/P tasks=No of I/P tasks Spark uses the algorithm called hash partition to decide which particular output partition should be set to particular output task. Hash Partition(output partition)=(Hash of the key)%(No of output task count) Few Commands data.getNumPartitions    (It is used to get no of partitions) reducedata.repartition(3).saveAsTextFile("out.txt"); Removing Null Values df.dropna(how="any").show() #removes row if any one column is null df.dropna(how="all").show() #removes row if all columns are null df.dropna(how="any",subset=["salary"]).show() #removes column if any null value is fo
Read more

Hadoop Installation

Image
If you are working with Windows Then Follow These Steps Else If(Linux) you can skip to hadoop installation. Download VMWare: VMWare Properties To Be Modified As Following: RAM-4GB ROM(MEMORY)-30GB CORES-4 ISO FILE LINK: https://ubuntu.com/download/desktop/thank-you?version=22.04.2&architecture=amd64 Accept all permissions and start using Linux. Hadoop Installation: Hadoop Installation Steps
Read more