Hadoop



About Hadoop Architecture

  • Hadoop Consists:
    • HDFS(Hadoop Distributed File System) [Parallel Processing]
    • MR(Map Reduce) [Processing]
    • Hive(Query Engine Introduced by Facebook) [Uses SQL]
    • Pig(Introduced by Yahoo) [Uses PigLatin]
    • Sqoop(Introduced by a group of people) [Uses Java]
    • Oozie(Scheduler Introduced by Yahoo) [XML,Java]
    • Flume(Messaging Queue) [Only Incoming Request]
    • Mahoth(Data Science,AI,ML Component)
    • HBASE(Hadoop Database Introduced by Facebook)

Key Points

  • All the databases in Big Data are NOSQL(Modern).
  • Any framework in Big Data is Loosely Coupled.
    • Loosely Coupled: Removal of one component doesn't affect the technology(Hadoop) function. Ex: C Sharp
  • Hadoop can be integrated with any other Big Data Technology.

Question & Answers

  • Q. What is file System ?
    • Ans. Used to read/write to and from Hard Disk
      • Ex: NTFS(Windows), EXT(Linux), MACFS(MAC)
    • A program in execution is called a process.
  • Q. What is Block ?
    • Ans. A large file is divided into small units called chunks
      • Ex: NTFS 16K
  • Q. Client and Server
    • Ans. Client->Requests and Server->Responds
  • Q. Types of File System
    • Standalone file system---NTFS,EXT,MACFS
    • Distributed file system---HDFS,S3
  • Q. Types of Distributed File System
    • Master and Slave(Hadoop, Spark) [One Master and N-Slaves]
    • Peer to Peer(NOSQL-Cassandra) [Each and every node connected to each other]
  • The background processes are called as Daemon Processes
    • In Hadoop we have 5 Daemon Processes(JP1, JP2, JP3, JP4, JP5 (Java Program))
  • Node is an Individual System or Virtual Machine and Cluster is a group of nodes together.
  • API-Application Program Interface
  • BLOCK SIZE in Hadoop
    • 1-Version(1B=64MB)(Default)
    • Latest Version(1B=128MB)(Default)
  • Replication: Replicate or duplicate's of data
    • In general the replication factor if Hadoop is 3
    • If we load 1GB of data we total of 3GB for replication
    • No duplicates in same node
    • Failure of one node can prevent data corrupt

HDFS+MR------------> HADOOP
HADOOP+OTHER--> HADOOP FRAMEWORK

Comments

Post a Comment

Popular posts from this blog

NOSQL Database and Pipeline tool

Image
  Kafka is a data pipeline tool. Zookeeper is needed before installing Kafka as its a cluster coordinator. Kafka stores all metadata information in zookeeper. NOSQL CAP is not possible at a time in NOSQL databases. Products like Facebook, Instagram achieves CAP by using multiple databases. CA is not possible at a time because of partition. In RDBMS no replication is present so CA is achieved. Ex: Hbase, MongoDB, Cassandra Datastores Keyvalue Datastore(Reddis) ColumnOriented Datastore(Hbase, Cassandra) DocumentOriented Datastore(MongoDB) Graph Datastore(Neo4j) Hbase is used for internal operations Lookup(veryfast) upset(insert+update)
Read more

MR(Map Reduce)

Image
Introduction to Map Reduce It's a massive parallel processing framework In general without distributed data it's hard to do parallel processing. Map Reduce uses Java by default The aim of Map Reduce is to achieve data locality and Parallelism. Spark is an alternative for Map Reduce(functionality has 80% similarity) Hive, Sqoop, Pig, Oozie are abstract of Map Reduce. Daemons in Map Reduce are JT and TT. In map reduce we also have two functionalities called Mapper and Reducer which are responsible for achieving the processing. The MR has an disadvantage of Resource Pressure. Advantages of Map Reduce Cluster Monitoring Cluster Management Resource Allocating Scheduling Execution Speculative Execution(L) Working Principle of Map Reduce The task for the MR is given in the form of ZAR(java) file to the Job Tracker. The Job Tracker will be sending the request to the Name Node and in return it gives response. Then the Job Tracker will be sending the task information to Task Tracker. The...
Read more

Spark Standalone Architecture and Working

Image
  About Spark Standalone Architecture The Daemons in spark are(Master Slave Architecture): Master Node Worker Node The Driver Program is starting point of spark which is started by Master and Responsible for Job. Lets understand the architecture of spark with an Example Client submitted job in the form of jar Each jar file contains driver program. The Spark master executes the driver code. The driver is requesting for resource from cluster manager. The cluster manager is responsible for allocation of resources. The cluster manager is allocating resources in the form of executors. The cluster manager finds data locality(particular data location) before assigning executors. Driver program is responsible for assigning the job to the executors (JAR File). Once the executor has been created it starts sending heartbeat to driver program and cluster manager also informs the driver program that it has been allocated the resources. The driver program is in Master and Executors are Workers. ...
Read more