Memory Calculations In Spark

 



  • Dynamic Resource Allocator(DRA)-Spark itself take care of memory calculations which highly not prefer.
  •  Cores: No of concurrent tasks that can be executed in each node.
    • 5 cores which means 5 parallel tasks can execute for a executor.
  • No of executors=(Total no of cores)/(No of cores in a node)
  • Executor Memory=((RAM)/(No of executors))-(Yarn Memory(2gb))=x
    • Executor memory range is (no of cores-x)
  • Driver Memory varies based on condition.

Comments

Post a Comment

Popular posts from this blog

MR(Map Reduce)

Image
Introduction to Map Reduce It's a massive parallel processing framework In general without distributed data it's hard to do parallel processing. Map Reduce uses Java by default The aim of Map Reduce is to achieve data locality and Parallelism. Spark is an alternative for Map Reduce(functionality has 80% similarity) Hive, Sqoop, Pig, Oozie are abstract of Map Reduce. Daemons in Map Reduce are JT and TT. In map reduce we also have two functionalities called Mapper and Reducer which are responsible for achieving the processing. The MR has an disadvantage of Resource Pressure. Advantages of Map Reduce Cluster Monitoring Cluster Management Resource Allocating Scheduling Execution Speculative Execution(L) Working Principle of Map Reduce The task for the MR is given in the form of ZAR(java) file to the Job Tracker. The Job Tracker will be sending the request to the Name Node and in return it gives response. Then the Job Tracker will be sending the task information to Task Tracker. The...
Read more

Hadoop Installation

Image
If you are working with Windows Then Follow These Steps Else If(Linux) you can skip to hadoop installation. Download VMWare: VMWare Properties To Be Modified As Following: RAM-4GB ROM(MEMORY)-30GB CORES-4 ISO FILE LINK: https://ubuntu.com/download/desktop/thank-you?version=22.04.2&architecture=amd64 Accept all permissions and start using Linux. Hadoop Installation: Hadoop Installation Steps
Read more

NOSQL Database and Pipeline tool

Image
  Kafka is a data pipeline tool. Zookeeper is needed before installing Kafka as its a cluster coordinator. Kafka stores all metadata information in zookeeper. NOSQL CAP is not possible at a time in NOSQL databases. Products like Facebook, Instagram achieves CAP by using multiple databases. CA is not possible at a time because of partition. In RDBMS no replication is present so CA is achieved. Ex: Hbase, MongoDB, Cassandra Datastores Keyvalue Datastore(Reddis) ColumnOriented Datastore(Hbase, Cassandra) DocumentOriented Datastore(MongoDB) Graph Datastore(Neo4j) Hbase is used for internal operations Lookup(veryfast) upset(insert+update)
Read more