what is Apache Parquet file | Lec-7
In this video I have talked about parquet file reading in spark. If you want to optimize your file and process in Spark then you should have a solid understanding of Parquet file format. Please do ask your doubts in comment section. Directly connect with me on:- https://topmate.io/manish_kumar25 Download Parquet Data:- https://github.com/databricks/Spark-T... Download parquet tools in your local to run all the below commands. Parquet tools can be downloaded using pip command. Run the below command in cmd or terminal pip install parquet-tools Run the blow command inside python import pyarrow as pa import pyarrow.parquet as pq parquet_file = pq.ParquetFile(r'C:\Users\nikita\Downloads\Spark-The-Definitive-Guide-master\data\flight-data\parquet\2010-summary.parquet\part-r-00000-1a9822ba-b8fb-4d8e-844a-ea30d0801b9e.gz.parquet') parquet_file.metadata parquet_file.metadata.row_group(0) parquet_file.metadata.row_group(0).column(0) parquet_file.metadata.row_group(0).column(0).statistics Run the below command in cmd/terminal parquet-tools show C:\Users\manish\Downloads\Spark-The-Definitive-Guide-master\data\flight-data\parquet\2010-summary.parquet\part-r-00000-1a9822ba-b8fb-4d8e-844a-ea30d0801b9e.gz.parquet parquet-tools inspect (path of your file location as above) https://parquet.apache.org/docs/file-... For more queries reach out to me on my below social media handle. Follow me on LinkedIn:- / manish-kumar-373b86176 Follow Me On Instagram:- / competitive_gyan1 Follow me on Facebook:- / manish12340 My Second Channel -- / @competitivegyan1 Interview series Playlist:- • Interview Questions and answers My Gear:- Rode Mic:-- https://amzn.to/3RekC7a Boya M1 Mic-- https://amzn.to/3uW0nnn Wireless Mic:-- https://amzn.to/3TqLRhE Tripod1 -- https://amzn.to/4avjyF4 Tripod2:-- https://amzn.to/46Y3QPu camera1:-- https://amzn.to/3GIQlsE camera2:-- https://amzn.to/46X190P Pentab (Medium size):-- https://amzn.to/3RgMszQ (Recommended) Pentab (Small size):-- https://amzn.to/3RpmIS0 Mobile:-- https://amzn.to/47Y8oa4 ( Aapko ye bilkul nahi lena hai) Laptop -- https://amzn.to/3Ns5Okj Mouse+keyboard combo -- https://amzn.to/3Ro6GYl 21 inch Monitor-- https://amzn.to/3TvCE7E 27 inch Monitor-- https://amzn.to/47QzXlA iPad Pencil:-- https://amzn.to/4aiJxiG iPad 9th Generation:-- https://amzn.to/470I11X Boom Arm/Swing Arm:-- https://amzn.to/48eH2we My PC Components:- intel i7 Processor:-- https://amzn.to/47Svdfe G.Skill RAM:-- https://amzn.to/47VFffI Samsung SSD:-- https://amzn.to/3uVSE8W WD blue HDD:-- https://amzn.to/47Y91QY RTX 3060Ti Graphic card:- https://amzn.to/3tdLDjn Gigabyte Motherboard:-- https://amzn.to/3RFUTGl O11 Dynamic Cabinet:-- https://amzn.to/4avkgSK Liquid cooler:-- https://amzn.to/472S8mS Antec Prizm FAN:-- https://amzn.to/48ey4Pj

How to write dataframe to disk in spark | Lec-8

Parquet File Format - Explained to a 5 Year Old!

rdd in spark | Lec-9

Apache Parquet: Parquet file internals and inspecting Parquet file structure

Antoine Pitrou - Apache Parquet : the standard, efficient file format for tabular data

Apache Iceberg: What It Is and Why Everyone’s Talking About It.

Data Lake Fundamentals, Apache Iceberg and Parquet in 60 minutes on DataExpert.io

The Parquet Format and Performance Optimization Opportunities Boudewijn Braams (Databricks)

Partitioning and bucketing in Spark | Lec-9 | Practical video

how to read json file in pyspark

Apache Iceberg Deep Dive | Part 1 | Crash Course

Apache Spark Was Hard Until I Learned These 30 Concepts!

AQE in spark | Lec-19

An introduction to Apache Parquet

Broadcast Join in spark | Spark Interview Question | Lec-14

Spark Job, Stages, Tasks | Lec-11

How SpaceX Humiliated Wall Street

Looking under the hood of the parquet format

cache and persist in spark | Lec-20

