Apache Spark Test 2 — Questions and Answers
Question 1: Spark supports which cluster managers?
- MESOS
- YARN
- Standalone Cluster Manager
- All of the above (Correct answer)
Correct answer: All of the above
Apache Spark is designed to be flexible regarding cluster management. It supports running on various cluster managers, including its own Standalone cluster manager, Apache Mesos, and Hadoop YARN, allowing users to choose the best environment for their needs. This broad compatibility makes Spark adaptable to different infrastructure setups.
Question 2: Which of the following statements about Spark MLlib is correct?
- Enables powerful interactive and data analytics application across live streaming data
- Provides an execution platform for all the Spark applications
- It is the scalable machine learning library which delivers efficiencies (Correct answer)
- All of the above
Correct answer: It is the scalable machine learning library which delivers efficiencies
Spark MLlib is Apache Spark's scalable machine learning library, offering a wide range of machine learning algorithms and utilities. It is designed for efficiency and can process large datasets in a distributed manner, making it suitable for big data analytics. Its focus on scalability and efficiency is a core characteristic.
Question 3: RDDs are immutable and fault-tolerant.
- False
- True (Correct answer)
Correct answer: True
RDDs (Resilient Distributed Datasets) are indeed immutable, meaning once created, their contents cannot be changed. They are also fault-tolerant because they can be rebuilt from their lineage of transformations if a partition is lost, ensuring data integrity. These properties are fundamental to Spark's reliability and performance.
Question 4: Which algorithm is not a solution for the regression problem?
- Gradient-Boosted Trees
- Decision Trees
- Logistic Regression (Correct answer)
- Ridge Regression
Correct answer: Logistic Regression
Logistic Regression is primarily used for classification problems, where the goal is to predict a categorical outcome (e.g., yes/no, true/false). While it has 'regression' in its name, it models the probability of a binary outcome rather than predicting a continuous value, unlike the other options which are suitable for regression tasks. This distinction is crucial in machine learning.
Question 5: Which of the following statements about Spark R is correct?
- It enables users to run SQL / HQL queries on the top of Spark.
- It allows data scientists to analyze large datasets and interactively run jobs (Correct answer)
- It is the kernel of Spark
- It is the scalable machine learning library which delivers efficiencies
Correct answer: It allows data scientists to analyze large datasets and interactively run jobs
SparkR is an R package that provides a lightweight frontend to use Apache Spark from R. It enables R users to interact with Spark, perform data analysis on large datasets, and leverage Spark's distributed computing capabilities within their familiar R environment. This integration empowers data scientists to scale their R workflows.
Question 6: Which of the following statements regarding DataFrame is correct?
- DataFrame API have provision for compile-time type safety
- DataFrames provide a more user-friendly API than RDDs. (Correct answer)
- Both the above
- None of the above
Correct answer: DataFrames provide a more user-friendly API than RDDs.
DataFrames offer a higher-level, more structured, and user-friendly API compared to RDDs, resembling tables in a relational database. They provide schema information and allow for optimized execution plans, making data manipulation and querying more intuitive and efficient for many use cases. This improved usability is a key advantage over raw RDDs.
Question 7: Which of the following statements about Spark Shell is correct?
- It allows reading from many types of data sources
- It helps Spark applications to easily run on the command line of the system
- It runs/tests application code interactively
- All of the above (Correct answer)
Correct answer: All of the above
The Spark Shell is an interactive environment (REPL) that allows users to quickly experiment with Spark code, test applications, and perform interactive data analysis. It supports reading from various data sources and facilitates running Spark applications directly from the command line. Its versatility makes it an essential tool for development and exploration.
Question 8: Is MLlib a deprecated library?
- No (Correct answer)
- Yes
Correct answer: No
MLlib is not deprecated; it is actively maintained and continues to be Apache Spark's primary machine learning library. While there's also ML (Spark ML), which is a newer, DataFrame-based API, MLlib (the RDD-based API) still exists and is used, especially for certain algorithms or custom implementations. Both libraries coexist and serve different use cases.
Question 9: On RDD, the read operation is
- Coarse-grained
- Fine-grained
- Neither fine-grained nor coarse-grained
- Either fine-grained or coarse-grained (Correct answer)
Correct answer: Either fine-grained or coarse-grained
RDDs support both coarse-grained and fine-grained read operations, though they are primarily optimized for coarse-grained transformations. Coarse-grained operations apply to the entire dataset (e.g., map, filter), while fine-grained operations allow access to specific elements or partitions. This flexibility allows for various data access patterns.
Apache Spark Test 2
Spark supports which cluster managers?