Hadoop



About Hadoop Architecture

  • Hadoop Consists:
    • HDFS(Hadoop Distributed File System) [Parallel Processing]
    • MR(Map Reduce) [Processing]
    • Hive(Query Engine Introduced by Facebook) [Uses SQL]
    • Pig(Introduced by Yahoo) [Uses PigLatin]
    • Sqoop(Introduced by a group of people) [Uses Java]
    • Oozie(Scheduler Introduced by Yahoo) [XML,Java]
    • Flume(Messaging Queue) [Only Incoming Request]
    • Mahoth(Data Science,AI,ML Component)
    • HBASE(Hadoop Database Introduced by Facebook)

Key Points

  • All the databases in Big Data are NOSQL(Modern).
  • Any framework in Big Data is Loosely Coupled.
    • Loosely Coupled: Removal of one component doesn't affect the technology(Hadoop) function. Ex: C Sharp
  • Hadoop can be integrated with any other Big Data Technology.

Question & Answers

  • Q. What is file System ?
    • Ans. Used to read/write to and from Hard Disk
      • Ex: NTFS(Windows), EXT(Linux), MACFS(MAC)
    • A program in execution is called a process.
  • Q. What is Block ?
    • Ans. A large file is divided into small units called chunks
      • Ex: NTFS 16K
  • Q. Client and Server
    • Ans. Client->Requests and Server->Responds
  • Q. Types of File System
    • Standalone file system---NTFS,EXT,MACFS
    • Distributed file system---HDFS,S3
  • Q. Types of Distributed File System
    • Master and Slave(Hadoop, Spark) [One Master and N-Slaves]
    • Peer to Peer(NOSQL-Cassandra) [Each and every node connected to each other]
  • The background processes are called as Daemon Processes
    • In Hadoop we have 5 Daemon Processes(JP1, JP2, JP3, JP4, JP5 (Java Program))
  • Node is an Individual System or Virtual Machine and Cluster is a group of nodes together.
  • API-Application Program Interface
  • BLOCK SIZE in Hadoop
    • 1-Version(1B=64MB)(Default)
    • Latest Version(1B=128MB)(Default)
  • Replication: Replicate or duplicate's of data
    • In general the replication factor if Hadoop is 3
    • If we load 1GB of data we total of 3GB for replication
    • No duplicates in same node
    • Failure of one node can prevent data corrupt

HDFS+MR------------> HADOOP
HADOOP+OTHER--> HADOOP FRAMEWORK

Comments

Post a Comment

Popular posts from this blog

MR(Map Reduce)

Image
Introduction to Map Reduce It's a massive parallel processing framework In general without distributed data it's hard to do parallel processing. Map Reduce uses Java by default The aim of Map Reduce is to achieve data locality and Parallelism. Spark is an alternative for Map Reduce(functionality has 80% similarity) Hive, Sqoop, Pig, Oozie are abstract of Map Reduce. Daemons in Map Reduce are JT and TT. In map reduce we also have two functionalities called Mapper and Reducer which are responsible for achieving the processing. The MR has an disadvantage of Resource Pressure. Advantages of Map Reduce Cluster Monitoring Cluster Management Resource Allocating Scheduling Execution Speculative Execution(L) Working Principle of Map Reduce The task for the MR is given in the form of ZAR(java) file to the Job Tracker. The Job Tracker will be sending the request to the Name Node and in return it gives response. Then the Job Tracker will be sending the task information to Task Tracker. The
Read more

Spark Yarn Cluster

Image
  Spark Submit Cluster(Yarn)->(Hadoop+Spark) Spark Lens Integration is a profiling tool to check the memory allocation(executor memory, etc.) Spark has Hash Partition which is the same algorithm used by MapReduce also. Map(Parallelism which acts as input), Reduce(Aggregation which acts as output) No of blocks=No of tasks when it is HDFS. No of O/P tasks=No of I/P tasks Spark uses the algorithm called hash partition to decide which particular output partition should be set to particular output task. Hash Partition(output partition)=(Hash of the key)%(No of output task count) Few Commands data.getNumPartitions    (It is used to get no of partitions) reducedata.repartition(3).saveAsTextFile("out.txt"); Removing Null Values df.dropna(how="any").show() #removes row if any one column is null df.dropna(how="all").show() #removes row if all columns are null df.dropna(how="any",subset=["salary"]).show() #removes column if any null value is fo
Read more

Hadoop Installation

Image
If you are working with Windows Then Follow These Steps Else If(Linux) you can skip to hadoop installation. Download VMWare: VMWare Properties To Be Modified As Following: RAM-4GB ROM(MEMORY)-30GB CORES-4 ISO FILE LINK: https://ubuntu.com/download/desktop/thank-you?version=22.04.2&architecture=amd64 Accept all permissions and start using Linux. Hadoop Installation: Hadoop Installation Steps
Read more