Free CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Databricks CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice Test Question - Page 2

Question 4 (Topic 1)

The code block shown below contains an error. The code block is intended to create a Python UDF assessPerformanceUDF() using the integer-returning Python function assessPerformance() and apply it to column customerSatisfaction in DataFrame storesDF. Identify the error. Code block: assessPerformanceUDF – udf(assessPerformance) storesDF.withColumn("result", assessPerformanceUDF(col("customerSatisfaction")))

Select an option, then click Submit answer.

○ The assessPerformance() operation is not properly registered as a UDF.
○ The withColumn() operation is not appropriate here – UDFs should be applied by iterating over rows instead.
○ UDFs can only be applied vie SQL and not through the DataFrame API.
○ The return type of the assessPerformanceUDF() is not specified in the udf() operation.
○ The assessPerformance() operation should be used on column customerSatisfaction rather than the assessPerformanceUDF() operation.

Reference / correct answer:

The return type of the assessPerformanceUDF() is not specified in the udf() operation.

Most accepted answer: D. The return type of the assessPerformanceUDF() is not specified in the udf() operation.

Community votes: A=1, D=4

The right answer is D. pyspark.sql.functions.udf(f=None, returnType=StringType) The default return type is string, but this question requires integer returning. so it should be D. "The return type of the assessPerformanceUDF() is not specified in the udf() operation." upvoted 12 times jds0 2 years ago Good explanation for Answer being D. Thank you! upvoted 2 times ...

Good explanation for Answer being D. Thank you! upvoted 2 times

Selected Answer: D D is the right answer as otherwise the return type is the default StringType(). Test code below: from pyspark.sql import SparkSession from pyspark.sql.functions import col, udf from pyspark.sql.types import IntegerType spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 3, "A"), (1, 1, "A"), (2, 2, "A"), ] storesDF = spark.createDataFrame(data, ["storeID", "customerSatisfaction", "division"]) def assessPerformance(x): return 1 if x > 3 else 0 print("IntegerType()") assessPerformanceUDF = udf(assessPerformance, IntegerType()) df = storesDF.withColumn("result", assessPerformanceUDF(col("customerSatisfaction"))) df.printSchema() print("Default") assessPerformanceUDF = udf(assessPerformance) df = storesDF.withColumn("result", assessPerformanceUDF(col("customerSatisfaction"))) df.printSchema() upvoted 3 times

correct answer is D upvoted 1 times

Selected Answer: D It is necessary to inform the return type as IntegerType(). from pyspark.sql.functions import udf, col from pyspark.sql.types import IntegerType storesDF = spark.createDataFrame([('1', '123'), ('2', '234')], ['id', 'customerSatisfaction']) assessPerformance = lambda x: int(x) assessPerformanceUDF = udf(assessPerformance, IntegerType()) storesDF.withColumn('result', assessPerformanceUDF(col('customerSatisfaction'))).printSchema() upvoted 1 times

Question 5 (Topic 1)

The code block shown below contains an error. The code block is intended to use SQL to return a new DataFrame containing column storeId and column managerName from a table created from DataFrame storesDF. Identify the error. Code block: storesDF.createOrReplaceTempView("stores") storesDF.sql("SELECT storeId, managerName FROM stores")

Select an option, then click Submit answer.

○ The createOrReplaceTempView() operation does not make a Dataframe accessible via SQL.
○ The sql() operation should be accessed via the spark variable rather than DataFrame storesDF.
○ There is the sql() operation in DataFrame storesDF. The operation query() should be used instead.
○ This cannot be accomplished using SQL – the DataFrame API should be used instead.
○ The createOrReplaceTempView() operation should be accessed via the spark variable rather than DataFrame storesDF.

Reference / correct answer:

The sql() operation should be accessed via the spark variable rather than DataFrame storesDF.

Most accepted answer: B. The sql() operation should be accessed via the spark variable rather than DataFrame storesDF.

Community votes: B=3

Selected Answer: B B is correct. 'storeDF' has not attribute or method `sql` Test code below: from pyspark.sql import SparkSession spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 3, "John"), (1, 1, "Jane"), (2, 2, "Jack"), ] storesDF = spark.createDataFrame(data, ["storeID", "customerSatisfaction", "managerName"]) storesDF.createOrReplaceTempView("stores") try: storesDF.sql("SELECT storeId, managerName FROM stores") except AttributeError as e: print(e) finally: spark.sql("SELECT storeId, managerName FROM stores").show() upvoted 2 times

Selected Answer: B B is correct: storesDF = spark.createDataFrame([('1', 'juan'), ('2', 'perez')], ['storeId', 'managerName']) storesDF.createOrReplaceTempView("stores") spark.sql("SELECT storeId, managerName FROM stores").show() upvoted 2 times

Selected Answer: B Option B is correct because the sql() function is not a method of a DataFrame object. It is actually a method of the SparkSession object spark. Therefore, the correct way to execute a SQL statement using Spark SQL is to call sql() on the SparkSession object as follows: spark.sql("SELECT storeId, managerName FROM stores") In the code block provided in the question, sql() is called on a DataFrame object, which will result in a DataFrame object without executing the SQL statement. Therefore, option B correctly identifies the error in the code block. upvoted 2 times

Question 6 (Topic 1)

The code block shown below contains an error. The code block is intended to return a DataFrame containing a column openDateString, a string representation of Java’s SimpleDateFormat. Identify the error. Note that column openDate is of type integer and represents a date in the UNIX epoch format – the number of seconds since midnight on January 1st, 1970. An example of Java’s SimpleDateFormat is "Sunday, Dec 4, 2008 1:05 PM". A sample of storesDF is displayed below: Code block: storesDF.withColumn("openDateString", from_unixtime(col("openDate"), "EEE, MMM d, yyyy h:mm a", TimestampType()))

Select an option, then click Submit answer.

○ The from_unixtime() operation only accepts two parameters – the TimestampTime() arguments not necessary.
○ The from_unixtime() operation only works if column openDate is of type long rather than integer – column openDate must first be converted.
○ The second argument to from_unixtime() is not correct – it should be a variant of TimestampType() rather than a string.
○ The from_unixtime() operation automatically places the input column in java’s SimpleDateFormat – there is no need for a second or third argument.
○ The column openDate must first be converted to a timestamp, and then the Date() function can be used to reformat to java’s SimpleDateFormat.

Reference / correct answer:

The from_unixtime() operation only accepts two parameters – the TimestampTime() arguments not necessary.

Most accepted answer: A. The from_unixtime() operation only accepts two parameters – the TimestampTime() arguments not necessary.

Community votes: A=2, B=1

Selected Answer: A A is correct: from pyspark.sql.functions import from_unixtime, col storesDF = spark.createDataFrame([(0, 1100746394), (1, 1474410343)], ['storeId', 'openDate']) storesDF = storesDF.withColumn("openDateString", from_unixtime(col("openDate"), "EEE, MMM d, yyyy h:mm a")) display(storesDF) upvoted 2 times

Selected Answer: A A is the right answer. Function signature from the docs: pyspark.sql.functions.from_unixtime(timestamp, format='uuuu-MM-dd HH:mm:ss') upvoted 1 times

A is also right. upvoted 2 times

Selected Answer: B B. The from_unixtime() operation only works if column openDate is of type long rather than integer - column openDate must first be converted. This option is correct. The code block has an error because the from_unixtime() function expects the column openDate to be of type long, not integer. The column should be cast to long before applying the function. upvoted 1 times juliom6 1 year, 8 months ago That not make sense, the code below works perfectly: from pyspark.sql.functions import from_unixtime, col storesDF = spark.createDataFrame([(0, 1100746394), (1, 1474410343)], ['storeId', 'openDate']) storesDF = storesDF.withColumn('openDate', col('openDate').cast('integer')) storesDF = storesDF.withColumn("openDateString", from_unixtime(col("openDate"), "EEE, MMM d, yyyy h:mm a")) display(storesDF) upvoted 1 times ... ZSun 2 years, 1 month ago This is completely nonsense about long and integer. long (or bigint): It is a 64-bit signed integer data type anging from -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807. integer (or int): It is a 32-bit signed integer data ranging from -2,147,483,648 to 2,147,483,647 upvoted 3 times ...

That not make sense, the code below works perfectly: from pyspark.sql.functions import from_unixtime, col storesDF = spark.createDataFrame([(0, 1100746394), (1, 1474410343)], ['storeId', 'openDate']) storesDF = storesDF.withColumn('openDate', col('openDate').cast('integer')) storesDF = storesDF.withColumn("openDateString", from_unixtime(col("openDate"), "EEE, MMM d, yyyy h:mm a")) display(storesDF) upvoted 1 times

Free CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK – Databricks CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK Practice Test Question