3. pyspark interview questions and answers for experienced | databricks interview question & answer

#Databricks #PysparkInterviewQuestions #deltalake Azure Databricks #spark #pyspark #azuredatabricks #azure In this video, I discussed PySpark scenario based interview questions and answers. PySpark advanced interview questions answers? databricks interview questions and answers? Create dataframe: ====================================================== data1=[(1,"Steve"),(2,"David"),(3,"John"),(4,"Shree"),(5,"Helen")] data2=[(1,"SQL",90),(1,"PySpark",100),(2,"SQL",70),(2,"PySpark",60),(3,"SQL",30),(3,"PySpark",20),(4,"SQL",50),(4,"PySpark",50),(5,"SQL",45),(5,"PySpark",45)] schema1=["Id","Name"] schema2=["Id","Subject","Mark"] df1=spark.createDataFrame(data1,schema1) df2=spark.createDataFrame(data2,schema2) display(df1) display(df2) ----------------------------------------------------------------------------------------------------------------------- df_join=df1.join(df2,df1.Id==df2.Id).drop(df2.Id) display(df_join) --------------------------------------------------------------------------------------------------------------------- from pyspark.sql.functions import * df_per=df_join.groupBy('Id','Name').agg( (sum('Mark')/count('*')).alias('Percentage') ) display(df_per) ------------------------------------------------------------------------------------------------------------------- ============================================================ 37. schema comparison in pyspark | How to Compare Two DataFrames in PySpark | pyspark interview:    • 37. schema comparison in pyspark | How to ...   Learn PySpark, an interface for Apache Spark in Python. PySpark is often used for large-scale data processing and machine learning. Azure Databricks Tutorial Platlist:    • Azure Databricks Tutorial   Azure data factory tutorial playlist:    • Azure Data factory (adf)   ADF interview question & answer:    • adf interview questions and answers for ex...   1. pyspark introduction | pyspark tutorial for beginners | pyspark tutorial for data engineers:    • 1. pyspark introduction | pyspark tutorial...   2. what is dataframe in pyspark | dataframe in azure databricks | pyspark tutorial for data engineer:    • 2. what is dataframe in pyspark | datafram...   3. How to read write csv file in PySpark | Databricks Tutorial | pyspark tutorial for data engineer:    • 3. How to read write csv file in PySpark |...   4. Different types of write modes in Dataframe using PySpark | pyspark tutorial for data engineers:    • 4. Different types of write modes in Dataf...   5. read data from parquet file in pyspark | write data to parquet file in pyspark:    • 5. read data from parquet file in pyspark ...   6. datatypes in PySpark | pyspark data types | pyspark tutorial for beginners:    • 6. datatypes in PySpark | pyspark data typ...   7. how to define the schema in pyspark | structtype & structfield in pyspark | Pyspark tutorial:    • 7. how to define the schema in pyspark | s...   8. how to read CSV file using PySpark | How to read csv file with schema option in pyspark:    • 8. how to read CSV file using PySpark | Ho...   9. read json file in pyspark | read nested json file in pyspark | read multiline json file:    • 9. read json file in pyspark | read nested...   10. add, modify, rename and drop columns in dataframe | withcolumn and withcolumnrename in pyspark:    • 10. add, modify, rename and drop columns i...   11. filter in pyspark | how to filter dataframe using like operator | like in pyspark:    • 11. filter in pyspark | how to filter data...   12. startswith in pyspark | endswith in pyspark | contains in pyspark | pyspark tutorial:    • 12. startswith in pyspark | endswith in py...   13. isin in pyspark and not isin in pyspark | in and not in in pyspark | pyspark tutorial:    • 13. isin in pyspark and not isin in pyspar...   14. select in PySpark | alias in pyspark | azure Databricks #spark #pyspark #azuredatabricks #azure    • 14. select in PySpark | alias in pyspark |...   15. when in pyspark | otherwise in pyspark | alias in pyspark | case statement in pyspark:    • 15. when in pyspark | otherwise in pyspark...   16. Null handling in pySpark DataFrame | isNull function in pyspark | isNotNull function in pyspark:    • 16. Null handling in pySpark DataFrame | i...   17. fill() & fillna() functions in PySpark | how to replace null values in pyspark | Azure Databrick:    • 17. fill() & fillna() functions in PySpark...   18. GroupBy function in PySpark | agg function in pyspark | aggregate function in pyspark:    • 18. GroupBy function in PySpark | agg func...   19. count function in pyspark | countDistinct function in pyspark | pyspark tutorial for beginners:    • 19. count function in pyspark | countDisti...   20. orderBy in pyspark | sort in pyspark | difference between orderby and sort in pyspark:    • 20. orderBy in pyspark | sort in pyspark |...   21. distinct and dropduplicates in pyspark | how to remove duplicate in pyspark | pyspark tutorial:    • 21. distinct and dropduplicates in pyspark...