Storage

6. Lab: Life cycle Management On Buckets

Let's say I have a bunch of photographs in my cloud storage bucket, and I want every one of my friends to be able to access them. In fact, I want to make the link public. How would I do this in this video? Let's see how we can assign permissions to specific objects within buckets as well as to buckets themselves. I may not always want to perform bucket-related tasks using the Web console. Google Cloud also gives me a command-line option to work with cloud storage using GS Util. GS Util is a Python object that allows you to perform cloud storage tasks using the command line, such as copying stuff into buckets,retrieving stuff from buckets, and so on. Here I use GS Utilities to create a new bucket, or MB, is the command. I want a regional bucket located in Asia's east one.And I want to call it Loony Asia Bucket. Once I've created the bucket using the GS utility on the command line, I can simply hit refresh on my Webconsole, and you can see that the Loony Asia bucket is now listed amongst my buckets on screen. If you want to run a process on some VM instance that's located in a particular region, it's ideal for your cloud storage bucket to be located in the same region. G Cloud compute zones The list will list all the regions which Google Cloud offers. You can use the GS Utility command onBuckets just like you would use the LS command on your local directory structure. GS Utils will list all the buckets that you have available. You can also copy objects from one bucket to another using CP and GS Utilities. GS Utilities CP allows you to specify a source bucket and a destination bucket, and we perform the copy. The R parameter indicates that we want to recursively copy over all files and directories in our bucket,and the P parameter indicates we want to copyover individual object permissions as well. Once the copy is complete, we can verify that the Lunar Asia Bucket now has all the objects that were originally present in the Lunar US Bucket, that is, the photos of Oba and Moji. Notice the URL here storage googleapis.com Lunarsha Bucket MoJ Jpg If you switch to viewing the objects in the Webconsole, you'll notice that Moji JPEG is public. It has a public link, and the permissions on Oba jpg are the same as the permissions that existed in the source bucket. There you see it. usercontact@lunacon.com can access it, and the domain Lunacon.com can access it as well. For now, you might want to use cloud storage for temporary storage, such as for your log files. And you want these log files to stick around for, say, one week, three months, or six months. In that case, you can set it up to have your log files automatically deleted by

7. Lab: Running A Program On a VM Instance And Storing Results on Cloud Storage

We've been manually uploading data into our buckets so far, but it's more likely that you'll fill these buckets programmatically. You'll typically run a bunch of computations on one of your VM instances and store the result in your bucket. What will happen to this data once your VM instance is deleted? In this lecture, we will see how we caningest and transform data using a programme running on a Compute Engine instance where all the data is stored in a cloud storage bucket. I have a couple of VM instances setup on my Computers engine page, the Kubernetescluster that we created for a demo earlier. And here is my brand new VM instance, which I'm going to use to perform some processing. In this lecture, I'll first SSH into this VM instance called Instance One in order to run some programs. Here I first run the Sudo app Get updates to ensure that all software packages are up to date. They are, and now I can go ahead and install the software packages that I need. The first thing I need is git. We've already seen how we can installgit before, so I'm going to simplyrun through this installation really fast. Check that we have the latest git version. That looks good. We'll now use Git in order to clone the repository that we are interested in onto this instance machine using the Git clone command. The Google Cloud Platform Training Data Analyst repositorycontains a whole bunch of sample data sets and code files for data-related code labs on the Google Cloud platform. Many of the labs in this particular course will use files from this training data analyst repository. There is a treasure trove of information here. You should definitely explore this on your own as well. Once we have all this data cloned onto our local instancemachine, we are now ready to move on with the labs. The lab that we'll do right now is the lab to be in CPB 100. CPB 100 is a beginner's training course for the Data Engineer certification program. All its labs are available for free for you to view and to run to view the files that are in this directory. We will be working with ingest sh, installing missing sh, and transforming PY. Let's first look at the Ingest.sh file. This simply downloads earthquake data from the US Geological Survey website onto our local machine. This is a CSV file called "Earthquakes. CSV." The first line of the script removes the existing CSV file on your local machine. That is, if one exists, it will remove it. And the second line simply makes a W getcall and downloads this data to our instance bash. Ingestsh will run all the commands in this file. The Earthquake CSV file will be downloaded to your instance. Use the head command to explore this file. The very first line is the list of column headers for the CSV. It will tell you the time of the earthquake, latitude and longitude, depth, and a whole bunch of other information. The Install Missing Assets script simply downloads all the packages that you need to run Transform PY. In this instance, The three Python libraries that are needed are basemap, Python NumPy, and Python Matplotlib. You might be familiar with NumPy for numerical computation and Matplotlib, which allows you to plot graphs. Base Map is a great Python library that allows you to visualise and generate maps. Transform PY will plot earthquake information on a map using this Base Map library. Run bashinstallmissing sh in order to download and install all these packages, if they don't already exist on your instance. These packages are not installed by default. Now let's see what operations performs in order to generate a map of our earthquake data, which is currently in CSV form. The Earthquake Class passes all the information about an earthquake from one row in the CSV file. It contains the latitude, longitude, timestamp,and magnitude for each quake. The Get Earthquake Data method reads the CSV file from the URL provided and generates an array of Earthquake Class instances. The marker colour for each earthquake on the resultant map is determined by the magnitude of that earthquake. Get marker is the function that decides the marker color. The Create PNG method contains the main logic of this file. It reads the CSV data and uses BaseMap to draw the world map and plot where the earthquakes have occurred. And finally, this map is saved as a PNG file. The file is saved to the local machine where we run this code, that is to our VM instance. Let's now run this Python code through Python transform PY,and then, once this code has run through tocompletion, you can check your current working directory. Notice there is an earthquake dot PNG, which is the PNGfile which contains all the quakes plotted on a map. In order to share this file with the world, we'll simply copy it over to our Luni US bucket using the GS utility. Go over to the Google Cloud Platform web console to the Loony US bucket and you'll see our earthquake files have been copied over there. Share all of these files publicly and then click on the public link for Earthquakes dot PNG. Let's see how the earthquakes look plotted on a map. Now that we've finished running Transform PY, let's delete the VM instance because we don't need it anymore. Go to Compute Engine VM instances and go ahead and delete instance one. Confirm the deletion and wait for the instance to disappear. Now let's check whether the Earthquakes PNG that we transferred from the instance to cloud storage still exists now that the VM instance has disappeared. Go to Storage Browser, click on Loony US Bucket, and you'll notice that Earthquakes PNG and all other files are still present in the bucket. The bucket's contents are not tied to the VM instance that created them. Which brings us back to our original question. Once we store something on cloud storage, even if the instance that created the data has disappeared and no longer exists, cloud storage contents are preserved. They are not tied to a particular VM instance.

8. Transfer Service

Here is a question that I'd like you to think about. We are not actually going to discuss this particular question in this video, but even so,this is an interesting thought experiment. We've covered a whole bunch of storage technologies. Which of these support groups? Cloud, SQL, Big Build, Cloud Spanner, BigQuery, and Data Store are the alternatives. The question is which of these technologies supports all that applies? Let's round off a conversation about storage options in the GCP by talking about the Transfer Service. The Transfer Service is basically an intelligent way of getting data into cloud storage. This is important. Remember that the Transfer Service will only help with getting data into cloud storage and not out of cloud storage. Okay, you see? From where can I import the data? And here, Transfer Service has a whole bunch of options. You can get data into cloud storage from AWS, i.e., from one of the three buckets. You could get the data from an http or https location on the web. You might also just import the data from local files. Although here you'd probably use another option like gsutil. And lastly, you can use it to transfer data from one cloud storage bucket into another. An obvious question, if you're familiar with GS Util, is when would you use GS Util in preference to Transfer Service? We've seen during the course of the variousdemos how gsutil can be used to get data into and out of cloud storage. And we should use the Transfer Service when transferring from AWS or from some other cloud provider into cloud storage. If we are loading files from an on-premise location into the cloud for the first time, we should prefer GSU to notify us. The Transfer Service has a whole bunch of bells and whistles related to transfers, which you might findhandy depending on how you set up your ops. It's possible to set up recurring transfers. It's possible to specify that you would like to delete files from the destination if they do not exist in the source. This way, you can keep a source and a destination in perfect sync with each other. You can also delete them from the source after you've copied them over. This way, you make sure that you have only one copy of your data. And lastly, it's possible to set up periodic synchronisation between a source and a destination, right? The AWS and GCP Data Transfer Service can really help keep data in sync. That gets us to the end of this look at the storage options in GCP. We will move on next to talk about big data technologies and then to machine learning. But first, let's recap the vast majority of the ground we've covered. We started with a laundry list of usecases, the simplest of which was block storage. If what you need is block storage for your computer VMs, just go with assistant discs or SSDs. One level up from block storage is File Storage.And if that's what you're interested in, If you need to store immutable blog like videos or images, go with cloud storage buckets. We also discussed how the Hadoop equivalent in GCP, Dataproc, will not use HDFS and will instead rely on cloud storage. If what you need is heavy transaction processing support,then make use of Cloud SQL or Cloud Spanner. Cloud SQL is open source. Cloud Spanner is Google's proprietary but also has a bunch of interesting optimizations. If OLAP rather than OLTP better describes your use case, then you should use BigQuery. We get to talk about BigQuery in detail. We'll get to it in the big data section of this course. And there are also a whole bunch of no SQL options in GCP. For document-oriented storage, you can use Data Store for key value storage. For fast sequential scans, you can make use of the Big Table, which closely resembles each bay. And lastly, for getting data into cloud storage,GCP offers something known as a transfer service. This makes sense. For instance, if you're getting data into GCP fromAWS, if you are getting data into cloud storagefrom your on-premise files, just use Jsutil instead. Let's turn back to the question that we posed at the beginning of this video. The question was which of these technologies, which are listed below, support joints? Well, let's answer the negative. Let's first talk about those technologies that do not support joins. BigTable, which is the GCP equivalent of HBase, and Data Store, which is an adocument-oriented database, do not support joins. All operations in Big Table are at the row level, so they cannot be across tables. Multi-table operations are not allowed. Datastore uses a lot of indices but does not support operations across tables,so no joins are allowed there either. The remaining alternatives, Cloud SQL and Cloud Spanner, clearly support joins because they are relational database technologies. And BigQuery, which is a relational wrapper on top of distributed storage, also supports joins, albeit with some conditions. The answer to this question is that Big Table and Data Store do not support joins. The other three do support joining.

9. Lab: Migrating Data Using The Transfer Service

It's quite possible that you have a whole bunch of data stored somewhere that you want to move into cloud storage because you've decided to adopt the Google Cloud platform. How do you move existing data from AWS rackspace to a local machine? In this lecture, we will study the transfer service for data migration. Let's say that you have some data that you want backed up. You're not going to access this data very often. Let's say you only want to accessit a couple of times a year. You would create a nearline bucket on the cloud in order to store this data. You can do so using the GS utility. From your command line, simply specify that you want a mirrorline bucket when you use theMB command to view details on the bucket, its associated permissions, and its corresponding metadata. You can use the GS Util list command on the bucket. This is shown right here on the screen. The L parameter gives us detailed information about the bucket, and the dash B parameter allows us to specify the bucket name instead of just listing all the buckets that we have in our cloud storage. Notice that the storage class for this bucket is near line and it's located in the US. This bucket is currently empty. The LS command on this bucket will return nothing in order to transfer and back up the data in this nearline bucket. You can do it via the Webconsole by setting up a transfer. Click on Transfer from the navigation sidebar on the left, and then click on Create Transfer to start a new transfer. The Web console will walk you through how you want your transfer set up. You can choose to transfer from another Google Cloud Storage bucket. You can choose an Amazon S3 bucket that you already own as a source, or give a list of object URLs from where you want the transfer to be performed. Whatever the source of your transfer, you must be able to read data from it. You must have read access to the source. If you were transferring data from your local machine to cloud storage, we would prefer to use GS Util. That's the fastest and easiest way to perform this transfer. But if you're transferring from an entity such as S Three or Rackspace, you would use the transfer service. In the case of an Amazon S Threebucket, you either need an access key or the bucket should be readable by everyone. For simplicity's sake, in this example, we simply transfer from one Google Cloud Storage bucket to another. Let's type out the source bucket from where we want to perform the transfer. This will be a looney bucket. As I'm typing it out, the Web console will check to see whether the bucket exists for a valid bucket name. You'll see a green check mark on the left. This was the source bucket. If you hit Continue, you'll now specify the destination bucket where you want your data to go. That console will check to ensure that the source and destination buckets are not exactly the same. Once you get the reassuring green checkmark indicating that it's a valid bucket, you can move on and look at the options that you have when you transfer data. You can choose either to overwrite your destination with the source objects, even when the objects are exactly identical, or you can delete objects from the source once they are transferred. When you want to save storage space, you'll do this, or you'll delete objects from the destination if there is no corresponding version in your source. This is when you want both buckets to be exactly identical. We will choose to delete objects from the source once they are transferred. Hit Continue after you've made this choice. Now you have a few options as to how you want to set this transfer up. You can choose for it to be a one-time transfer. You can only run it at this current moment, or you can choose for it to be a recurring transfer occurring at the same time every day. Give the transfer a name in order to identify it. Now we are ready to go ahead and create it. Click on Create and watch the transfer service. Pick up your source files and put them into your destination bucket. The source object will be deleted once the transfer is complete. As the transfer progresses, you can choose to pause it or even cancel it completely. Using these buttons on screen, the Files transferred column will show you the current status of your transfer. Once all our files had been transferred, they were just two. We can use GS Utility to verify that our destination bucket now has all the files that we expected to. Our mirrorline bucket now has both Oba and MoJS images. Visit the bucket on your web console and verify that the files have been transferred successfully. Because we chose to delete the files from our source bucket, you can check whether the Loony US bucket is now empty. It should be. Hopefully, you now know the answer to this question. If I were to perform a transfer from AWS or Rackspace, I would prefer to use the transfer service. If I want to perform a transfer from my local machine or somewhere on site to Google cloud storage, I would use GS Utility.

ExamSnap's Google Professional Data Engineer Practice Test Questions and Exam Dumps, study guide, and video training course are complicated in premium bundle. The Exam Updated are monitored by Industry Leading IT Trainers with over 15 years of experience, Google Professional Data Engineer Exam Dumps and Practice Test Questions cover all the Exam Objectives to make sure you pass your exam easily.