Question 22
(Topic 1)
A data scientist is working on a large dataset in Apache Spark using PySpark. The data scientist has a DataFrame df with columns ‘user_id’, 'product_id’, and ‘purchase_amount’ and needs to perform some operations on this data efficiently. Which sequence of operations results in transformations that require shuffle followed by transformations that do not?
Select an option, then click Submit answer.
- ○ df.filter(df.purchase_amount > 100).groupBy("user_id").sum("purchase_amount")
- ○ df.withcolumn("discount", df.purchase_amount * 0.1).select ("discount")
- ○ df.withcolumn("purchase_date", current_date()).where("total_purchase > 50")
- ○ df.groupBy("user_id").agg (sum ("purchase_amount").alias("total_purchase")).repartition(10)
