Spark

 

About Spark

  • Spark is a data processing framework which is a solution for bigdata.
  • Spark is subset of Hadoop
  • Spark SQL is used to connect with hive for retrieving data.
  • Spark has two data process methods
    • Batch Processing
      • It's about dealing with files, databases.
    • Stream Processing
      • It's about dealing with live data.
  • It is has less security.
  • Spark can run on standalone file system then distributes the data in memory is called Inmemory Processing.
  • Spark can connect to any data storage.
  • Spark replaces MR in Hadoop.
  • Spark is a data processing framework which do's batch processing (or) stream processing.
  • Spark works on Java, Python, Scala and R.

Architecture of Spark

  • Deployment Modes
    • Standalone(Installation of only spark in single/multi node)
    • Yarn(Spark+Hadoop)
    • Mesos
  • Daemon Processes
    • Master(Similar to Master in HDFS)
    • Worker(Similar to Slave in HDFS)
  • The memory will be distributed its data in it and then processes which is called as Memory Computation
  • Spark REPL(Read Evaluate Print Loop)
    • Any technology based on cmd is called as repl
    • Sparkshell(scala)
    • Pyshell(python)

API's in Spark(Processing Data)

  • RDD(Coding)
  • DataFrames(Rows and Columns)
  • Dataset(Extension of RDD)

Lazy Evaluation

  • In spark we have two technologies called transformations and actions
    • Transformations: map, group, sort, filter
    • Actions: count, savefile, show, output
  • For any job after performing transformations there should be an action at the end for the compilation as it is bottom up approach which is called as Lazy Evaluation
  • If there is no action after performing transformation then there is no usage of the job so this was introduced.
  • Ex
    • Map->Filter->Group->Sort->...........->group (No output as there is no action at the end)
    • Map->Filter->Group->Sort->...........->count (Output as there is action at the end)

Tooling

  • To start spark
    • sbin/start-all.sh
  • Spark UI is at localhost:8080
  • To start spark shell(scala)
    • localhost:4040
    • bin/spark-shell
  • To start pyspark(python)
    • bin/pyspark
  • WordCount is the basic program for spark similar to Hello World.
  • There are two types of variables in spark
    • var (similar to static variable)
    • val (similar to final variable)
  • sc is the spark context where the compilation will be started by the compiler


Wordcount Program

var a=sc.textFile("file:/home/user/data.txt").flatMap(line => line.split(" ")).map(word => (word,1));
var b=a.reduceByKey(_ + _);
b.collect;


Comments

Post a Comment

Popular posts from this blog

MR(Map Reduce)

Image
Introduction to Map Reduce It's a massive parallel processing framework In general without distributed data it's hard to do parallel processing. Map Reduce uses Java by default The aim of Map Reduce is to achieve data locality and Parallelism. Spark is an alternative for Map Reduce(functionality has 80% similarity) Hive, Sqoop, Pig, Oozie are abstract of Map Reduce. Daemons in Map Reduce are JT and TT. In map reduce we also have two functionalities called Mapper and Reducer which are responsible for achieving the processing. The MR has an disadvantage of Resource Pressure. Advantages of Map Reduce Cluster Monitoring Cluster Management Resource Allocating Scheduling Execution Speculative Execution(L) Working Principle of Map Reduce The task for the MR is given in the form of ZAR(java) file to the Job Tracker. The Job Tracker will be sending the request to the Name Node and in return it gives response. Then the Job Tracker will be sending the task information to Task Tracker. The...
Read more

Spark Yarn Cluster

Image
  Spark Submit Cluster(Yarn)->(Hadoop+Spark) Spark Lens Integration is a profiling tool to check the memory allocation(executor memory, etc.) Spark has Hash Partition which is the same algorithm used by MapReduce also. Map(Parallelism which acts as input), Reduce(Aggregation which acts as output) No of blocks=No of tasks when it is HDFS. No of O/P tasks=No of I/P tasks Spark uses the algorithm called hash partition to decide which particular output partition should be set to particular output task. Hash Partition(output partition)=(Hash of the key)%(No of output task count) Few Commands data.getNumPartitions    (It is used to get no of partitions) reducedata.repartition(3).saveAsTextFile("out.txt"); Removing Null Values df.dropna(how="any").show() #removes row if any one column is null df.dropna(how="all").show() #removes row if all columns are null df.dropna(how="any",subset=["salary"]).show() #removes column if any null value is fo...
Read more

Hadoop Installation

Image
If you are working with Windows Then Follow These Steps Else If(Linux) you can skip to hadoop installation. Download VMWare: VMWare Properties To Be Modified As Following: RAM-4GB ROM(MEMORY)-30GB CORES-4 ISO FILE LINK: https://ubuntu.com/download/desktop/thank-you?version=22.04.2&architecture=amd64 Accept all permissions and start using Linux. Hadoop Installation: Hadoop Installation Steps
Read more