Pub/Sub - Dataflow - BigQuery. How to get your data ready to tell you a story?

This is the first of a series of episodes I will create on how to use the services available in the Google Cloud Platform for Data Engineering and more. In this Pub/Sub - Dataflow episode, you are going to learn how to send some messages ( sensor data from highway speed sensors ) to Pub/Sub and start a Dataflow pipeline template to read them and sink it to BigQuery for further analytics using Google Data Studio (average link speed, max speed and min speed on each highway ). In this video, I will show you how to download and install GCloud SDK on your local pc and how to use the UI. https://cloud.google.com/sdk/docs/dow... The data we are going to use can be downloaded at this link. https://github.com/GoogleCloudPlatfor... The JSON format of the data is also available at this link. https://storage.googleapis.com/gcprai... Please let me know if you need any other video to explain some concepts you are curious about in more detail. Enjoy, Like, Subscribe and Share If you liked the video, want more, and are part of deciding what's next, please support us at https://www.patreon.com/cslearning?fa....   / cslearning86     / cslearning3   https://www.facebook.com/profile.php?... Follow my blog on https://cslearning.blog/ Thank you 00:00 Introduction 02:24 How to install GCP SDK 04:19 Create Pub/Sub Topic 09:07 Publish data on the topic 11:30 Set up BigQuery tables 16:56 Create Dataflow pipeline sending CSV 24:52 Transform CSV To JSON 27:26 Correct BigQuery Schema 32:53 Creating a new Dataflow pipeline 37:33 Query the data 38:19 Show data in DataStudio 40:21 Final words