NOSQL Database and Pipeline tool

 


  • Kafka is a data pipeline tool.
  • Zookeeper is needed before installing Kafka as its a cluster coordinator.
  • Kafka stores all metadata information in zookeeper.


NOSQL

  • CAP is not possible at a time in NOSQL databases.
  • Products like Facebook, Instagram achieves CAP by using multiple databases.
  • CA is not possible at a time because of partition.
  • In RDBMS no replication is present so CA is achieved.
  • Ex: Hbase, MongoDB, Cassandra
  • Datastores
    • Keyvalue Datastore(Reddis)
    • ColumnOriented Datastore(Hbase, Cassandra)
    • DocumentOriented Datastore(MongoDB)
    • Graph Datastore(Neo4j)
  • Hbase is used for internal operations
    • Lookup(veryfast)
    • upset(insert+update)

Comments

Post a Comment

Popular posts from this blog

Spark Standalone Architecture and Working

Image
  About Spark Standalone Architecture The Daemons in spark are(Master Slave Architecture): Master Node Worker Node The Driver Program is starting point of spark which is started by Master and Responsible for Job. Lets understand the architecture of spark with an Example Client submitted job in the form of jar Each jar file contains driver program. The Spark master executes the driver code. The driver is requesting for resource from cluster manager. The cluster manager is responsible for allocation of resources. The cluster manager is allocating resources in the form of executors. The cluster manager finds data locality(particular data location) before assigning executors. Driver program is responsible for assigning the job to the executors (JAR File). Once the executor has been created it starts sending heartbeat to driver program and cluster manager also informs the driver program that it has been allocated the resources. The driver program is in Master and Executors are Workers. ...
Read more

YARN(Yet Another Resource Negotation)

Image
  About Yarn It's a cluster manager that supports common execution. Resource Manager(JT) and Node Manager(TT) are responsible for it's functionality. It is introduced in Hadoop-2 The components like Spark can run individually without Yarn Also. Functionality of Yarn Application Manager is a combination of Scheduler and Resource Manager. Application Master acts between Resource Manager and Data Node. For every jar the Resource Manager starts new Application Master There would be more than one Resource Manager(Active and Passive) When the job is submitted to the Application Manager then it sends the information to application master and the data node starts the mapper task and when it's done reducer task starts. When reducer job is done the Application master sends the final heartbeat to the Application Manager. If there is more than one jar is submitted then new Resource Manager is started. If Application Master is failed then new Application Master Would be started and job ...
Read more