Storage

10. Lab: Cloud Storage ACLs and API access with Service Account

In this demo for cloud storage, we will be creating some cloud storage buckets and then playing around with some of its available features, including access control lists, encryption, lifecycle management, versioning directory synchronization, and cross-project sharing of buckets. To start with, though, let us go ahead and create an a service account which we will use to access the bucket.

So we navigate to I Am and Service Accounts and click to create a new service account name. This One Store Core we assigned the role of an editor, which means that it will also have the ability to edit buckets and furnish a new private key in the form of a.json file. And once we hit Create, it will download our JSON file. So now that we have our service account along with this private key in a JSON file, go ahead and create a cloud storage bucket. So we navigate in the menu to storage and browser and here we create a new bucket. Since the name of each bucket needs to be unique globally, choose something that is available. So for us, Lunacon bucket four is available. We then hit Create, leaving all the default settings in place. And so we now have a bucket ready. The next step for us is to provision a VM instance, which we will use to access this bucket. So, for that, we navigate to compute engine instances and create a new VM instance. It's also called this One Store Core.

We created it in the US central region and left all the other settings as default and hit Create. And now with our virtual machine provision, SSH into it. Once the terminal is up, copy over the contents of the JSON file containing our service account private key to this VM instance. So for that, let us first create a credentials JSON file over here. You can use any text editor for it and then navigate to our own machines. We copy over the contents of the JSON file and paste it into our VM instance. And now that we have this file in place, let's use it to interact with our GCP project from the command line. We first run this gcc or activate serviceaccount command to allow our VM to interact with the Google Cloud API using the service account credentials. So once we have executed this command, the next step for us is to reset the local profile on this instance and initialise the API for which we run G Cloud on it. And when we run that command, we first choose option one to re-initialize the current configuration rather than create a new one. In a second set of options, we choose to use the new service account we created in order to interact with the API.

So, choose option two. The project to use should just be the one where we have provisioned our service account bucket and VM instance. So in my case, it is Loonycon project Zero six, and the next prompt just hit yes to confirm. And when prompted for the zone, I'm just going to select US Central Zone C, which is the location of this VM instance. Now that we have configured this instance to interact with our project from the command line using our service account, let us go ahead and get some files which we will use to upload into a cloud storage bucket. Or rather, let us just get one file and make copies of it. So the file I'm downloading is a hadoop setup file, and in order to have three files to upload to our bucket, let us just cheat and create two copies of the same file to create the first copy. And we now have a second copy. So, with three files in place, just copy one of the files into our cloud storage bucket. So this is the command to execute. For that, just grab our bucket name, and once we have that, we just upload the setup HTML file. Okay, so that was successful. So, with the file uploaded into a cloud storage bucket, let us take a look at its default ACL access control list.

By running this GS utility, ACL get command And once the ACL file is downloaded, taking a look at it, we see that there are a number of permissions which are assigned over here. So let us try to constrain it. For that, set this file to be private. For that, we just run GS Util ACL setprivate, and once that command has been executed, let us get the new ACL for it. And we see here that there is just one entry that a service account grants the owner role on this file. Next, let us loosen up the permissions on this file and grant read access to pretty much everyone. So for that, run this GS utility ACLch command and specify that all users should have read access to this file. So once the command has been executed, download the new ACL and take a look. And now we see that there is one additional entry in the ACL file which grants all users read access. Now navigate to the console and take a quick look at the file from over there. So we go into a bucket and we may need to refresh and we see that the setup HTML file is over here and it also has this public link. Because we have just recently set all users to have read access to it. Let us now head back to the SSHterminal of our instance and we will now delete the setup HTML file from our local storage. This is because we have uploaded a file so far and modified its permissions, but let us test the downloading file from our bucket. So we get rid of the local file and then try to download the file from the bucket using the GS utilityCP command, and we see that the download is successful.

11. Lab: Cloud Storage Customer-Supplied Encryption Keys and Life-Cycle Management

One thing we have not explicitly seen so far is that Google Cloud Storage always encrypts the file on the server side before it is written to disk. This is so that if someone malicious were to gain access to the disk, they would still be able to read the contents of the file. However, GCP also provides the option of a customer supplying an encryption key which will be used for server-side encryption. So let us see how that works. The first thing to do is to generate an encryption key, and for that, let us just use a Python library as an example, and we will generate what is called a customer-supplied encryption key, or Csek. So after running a command, we get an encryption key and, in order to use it for Google Cloud Storage, we are going to edit a config file called dot boto. So we should find this in our workspace, and once we have located it, edit this one in our text editor.

We find that there is a field called encryption key. And over here, let us just append the encryption key we just created. So with that added, save the file. We can now upload a new file which will be encrypted using our new encryption key. So, just as before, we use the GS Util CPcommand, but this time we upload the second file. Let's just go ahead and upload the three dot HTML as well. And now, with two new files in our bucket, navigate to the console and see what our bucket looks like now. And once we go to a bucket, we see that the two new files appear over here. And it also states that these files have been encrypted with a customer-supplied key. Let us now try to download all of these files into our instance and see if you are able to view them. First, just delete all our local copies of the file.

So all the three set-up HTML files would be removed. And now download the file from the bucket. We just download all the set up HTML files. And now let us see if you're able to read the contents. So the first HTML file was encrypted automatically by GCP for us. So this is not a key which we supplied and we're able to view this. Let us take a look at the other files. So, setup two HTML and we're able to read this one as well. And finally, move over to the last setup file, and we can see that all of these files are readable now, even though the first one is encrypted with a Google supplied key, and the last two are encrypted with our own customer supplied key. For our next step, let us perform a key rotation and see what a possible side effect is if certain steps are not followed when performing this rotation. So first, edit the photo file again and comment out the encryption key, which is unused. We will soon be replacing this with a new one. But let us use this encryption key and make it a decryption key. So we copy over this value and we uncomment the encryption key one and we use this key for decryption. Now we save the file and next let us go and create a new encryption key.

So use Python once again and when the key is generated, copy it over and add it to our boto file. And now for the encryption key property, just uncomment the line first and then remove the old encryption key value and add the new one. And just to note, the old encryption key is now the new decryption key. What we will do next is to rewrite the key for our setup two-dot HTML file in the cloud storage bucket. So we will use this GS utility rewrite command. And what this essentially does is decrypt a file using the decryption key specified in our boto config, then encrypt the file using the new encryption key value specified in our config file. So we have now rotated our encryption keys and we have applied that to the setup of the HTML file in our bucket. Let us now go into our botoconfigand comment out the decryption key property. And once that is complete, now try to download the setup and set up three HTML files and see how that works. So we have downloaded the setup to an HTML file and let us try to do that for setup three. And this download fails because we did not perform a rewrite of the setup HTML using our new encryption key. And we also commented out the decryption key from a property file which could have been used to decrypt this file.

As a result, whenever a key rotation is performed, it is best to rewrite the files with the new keys. Now that we have some familiarity with encryption for cloud storage buckets, let us move along to lifecycle management. So first let's take a look at the Lifecycle policy on the bucket we just created. And we see that, by default, there is no lifecycle policy set. So let us go ahead and add one. So for that, we create a JSON file and within the file we specify a rule. So what this says is that the action to be performed is to delete files in the bucket on the condition that their age has exceeded 31 days. So once we have saved the file, we apply this rule to our bucket. We do that by executing the Lifecycle set command. And once that has been executed, let us just verify that the policy has been applied correctly. So we run the Lifecycle get command again and see that our JSON file has indeed been loaded. So the lifecycle has been set for our buckets from lifecycle management. Let us move along to versioning. So let us run the GS util versioning get command and we see that, by default, versioning is disabled. So let us activate that for our bucket.

So we do a versioning set. We see thatversioning has now been enabled for the bucket. Just run versioning get again for a quick confirmation that versioning is indeed enabled. Let us now test versioning by creating a few copies of the setup.dot HTML file. So for the original copy, just take a look and we see that the size is about 53.8. Edit this file and remove a few lines. So once that is complete, we save this version and upload it into a bucket. We expect this version to be a little smaller than the previous one. Now let us create a third version of the file. So we do that by just editing the setup.html locally again and we remove a few more lines, so it's going to be even smaller, and once we have completed that, we save the file and we upload a third version of the file. Let us now take a look at the different versions of the file in our bucket. So we just do a gsutil LS and see the three different versions which are listed here. Let us now grab the oldest version of the file, which is also the original copy. So this is the one which appears right at the top of our list and we download it into recover. TXT. Now let us compare the relative sizes of the original version and the latest version, which is on our file system. So we can see that the latest version is about a shade under 53. The original version is about 53.8 KB. So, using versioning, we have been able to recover the original file.

12. Lab: Cloud Storage Versioning, Directory Sync

Moving along now from versioning, try to setup synchronisation between a local folder in our file system and our remote cloud storage bucket. So to do this, create two levels of directories. So we call them the first level and the second level. Let us copy over the setup HTML file into our first level and let us copy it over as well into the second level directory. Now set up the syncing of our local file system directory with a cloud storage bucket by using the RSNC command. And over here, we're just going to sync our entire home directory with a bucket.

So once we execute the command, we see that a whole bunch of files have been uploaded into the bucket. And once this is complete, let us navigate into the console and view the contents of our bucket from over there. So, navigating to a bucket, we see that all our files are indeed present here, including all our config files. And specifically, we see that the first level directory is also visible. So let us navigate into that and we see, as expected, that the setup HTML as well as the second level directory are present here. And inside the second level directory, we see the setup HTML just as we expected. As a quick verification, let us go back to the command line and see if we can list the contents of our remote bucket from over here, specifically the directory which we just created. So we recursively list the contents of the first level directory and we see that the files and directory structure can be viewed from here. So, moving on now to our final exercise in this lab on cloud storage. And this is going to involve the sharing of buckets across projects. To see what that means, let us go ahead and create a new bucket in a different project.

So we navigate from Project Six in my case, and I'm going to select another project within my organisation called Loonikon Project Seven. And once inside, I'm going to navigate to the storage browser and create a new bucket over here. So just to reiterate, the name of the bucket must be unique globally. So you may not always get the bucket name you want. In my case, I'm getting a Looney Bucket, leaving all the other values as default, and creating the bucket. And once it is ready, let us upload a file. And in my case, I'm just going to pick a file from the file system. And once this file has been uploaded, let me just rename this to something a little smaller. So I'm just going to call the sample PNG. And now we have a bucket with a file in it. The next step is to create a service account for this project. So we navigate to the Im and service account. And when we create this new service account,let us call it Cross Project Storage. And the role we will assign is that of a storage object viewer. So we simply navigate through the menu and provide a new private key in JSON format, after which the file is downloaded to our file system.

And now we are set up with a service account along with a new private key for it. So for our next step, let's navigate back to the original project. So we just select the project from the menu and, once inside, we will now provision a second VM instance. So, going to the VM instances page, we will call this one the Cross project. This is going to be in the European West region, and the machine type is going to be a micro. With that ready, let us just hit "create." And we now have two instances of this project. Let us SSH into the newly created instance once our terminal is up and try to list the contents of our newly created bucket from the command line. So the bucket is on a different project, you will recall. And also, we haven't really done any configuration in this instance to use the Google Cloud API. And running this LS command, we see that we get an access denied message. That is because this instance is using the default service account linked with this project and it does not have the permissions to access the bucket created in the other project. However, the new service account that we created does have the required permissions.

So let us configure this instance to use that service account. So we create a new credentials JSON file, and we copy over the contents from the private key file of our new service account. Once that is done, we authorise thisVM to use the service account credentials when accessing the Google Cloud API. So we run this G cloud auth activate serviceaccount command and then we run the G cloudinit command, which will reset the local profile and initialise the Google Cloud API from here. So when we run the command, we first select option one to reinitialize the configuration. We then select the service account which we would like to use. So we hit option two. If asked whether to enable the API, we just say yes. And finally, when we want to specify the project which we would like to access, we enter the ID of the second project. So that is where our bucket is. So in my case, it's Lunacon Project Seven.

So this instance, which is in our original project, has now been set up to access the second project using the service account credentials from there. So let us just check if you're able to list the contents of our new bucket from here. We once again run the command and see that the LS was successful this time. Now let us try to write something into that bucket. So we have one file here, which is our credentials adjacent file. So let us just use that and we see that we do not have the permission to write to this bucket. And if you remember, this is because the new service account that we created had the storage object viewer permission. So let us go ahead and modify that. So let us navigate to the IAM page for the second project. And for our service account over here, we see that we just have the storage object viewer role attached to it. So let us go ahead and add another one. This time, let us add the storage object admin role as well. Once that is done, we see the service account has multiple roles assigned to it. So, as a final test, let us head back to our instance and see if the upload to the bucket works now. So we just run the same command and this time the upload is successful. So we have been able to configure this instance to access a bucket created in a different project. In other words, we have enabled cross-project sharing.

Cloud SQL, Cloud Spanner ~ OLTP ~ RDBMS

1. Cloud SQL

Here is a question that I'd like you to try to answer, or at least keep in the back of your mind as we discuss Cloud SQL in the remainder of this video. The question is, does it make sense to use a cloud proxy to access cloud SQL? In other words, using a cloud proxy to access a cloud SQL instance is a good idea. Is the statement true or false? The next technology that we are going to discuss is going to be Cloud SQL. This is a relational database, as its name would suggest. Because relational databases are used for transaction processing, that is, OLTP applications, they need to make very strong commitments regarding transactions. So the asset properties of automatic consistency, isolation, and durability must be enforced by RDBMS.

Relational databases like Cloud SQL and Cloud Spanner need to work with extremely structured data and enforce constraints on that data. As a result, they are unsuitable for OLAP, analytics, Bi, or business intelligence applications. They tend to deal with much larger datasets, particularly in these days of big data. Such large data sets cannot be efficiently handled by relational databases like Cloud SQL. The transaction support provided by relational databases is overkill for OLAP applications. OLAP applications typically do not require very strict writing consistency. We won't spend a whole lot of time talking about relational data and the relational data model that underlies RDBMS. Just very quickly, discuss the basic idea.

These databases represent data tables or relationships. These tables tend to have rows and columns. The number and type of the columns are strictly defined and are known as the schema. If you find yourself wondering what exactly a schema is or what a primary key is, my suggestion is that you take some time off to very quickly brush up on those fundamentals of databases before picking up here. Once again, the relational data model works extremely well when we have data that is very well structured but does not exhibit particularly strong hierarchical relationships. This is data without a lot of missing values. As we shall see, when there are a lot of nulls or missing values, columnar databases tend to work a bit better. Let's quickly review the differences between Cloud SQL and Cloud Spanner. We had discussed this previously. Cloud Spanner is Google proprietary and it's also more advanced than Cloud SQL. We shall discuss Cloud Spanner in a lot of detail, but for now, do keep in mind that, unlike Cloud SQL, Cloud Spanner offers horizontal scaling, which means that one can increase the number of instances of one's database application and thereby improve its performance.

Cloud SQL supports two open source SQL implementations: MySQL, which is the fast and usual and efficient implementation, as well as a beta version of PostgreSQL. This is a beta currently as of July 2017. The very broad brush generalization is that PostgreSQL functions better when you have very complex sequel queries because the PostgreSQL version is in Beta. Let's mostly restrict our conversation to MySQL. Most of this will also give you an idea of how cloud SQL works. In general, cloud SQL is not server less. That's because we need to explicitly instantiate cloud SQL. This is typically the case with any technology which provides transaction support. Here, while creating a Cloud SQL instance,we need to specify the region. We also need to pick between first and second generation instances. Second generation instances have a slew of new features, including higher availability configuration, proxy support, and the ability to remain operational during maintenance. For all of these reasons, we should prefer second generation or first generation, which is pretty intuitive.

Let's talk a little bit about the high availability configuration because this is conceptually interesting. This is a feature on second-generation MySQL instances. Here, the instances have a failover replica, and that failover replica must be in a different zone than the original instance. The original is called the "Master." By the way, this gives us a little bit of a hint about how Cloud SQL works under the hood. Any changes made to data on the Master, whether to users or system tables, are reflected in the failure replica via a process known as semi synchronous replication.

Later on in this course, when we discuss networking in detail, we will be quite interested in proxies of all varieties. This is important if you are looking to pass the Cloud Architect certification. So let's also talk a bit about the SQL cloud proxy. In general, proxies are a security feature because they help shield the real IP addresses of your instances. So, for instance, you can use a Cloud SQL proxy in order to give secure access to your instances without having to whitelist the individual IP addresses of the instances that are going to be connected to it. And this proxy is a secure one because it will set up a secure encrypted connection and use SSL certificates to verify the identities of the client and the server. This also makes for easier connection management. By the way, security and easier connectionmanagement are general features of proxies. The proxy eliminates the need for static IP addresses on your server.

Let's take a very quick look at how a cloud proxy works. Here is a diagram taken from the GCP docs. This illustrates how a client application running on a client machine can communicate with Cloud SQL via a TCP Secure tunnel. The Cloud SQL proxy works by having a local client, which is called the proxy client, running on the client machine. The client application only communicates with the proxy. It does not directly talk with Cloud SQL, and as a result of that, there is a great deal of additional security. The client app uses something like ODBC or GDBC to connect to this proxy client, and the proxy, in turn, uses a secure tunnel. This is a secure TCP tunnel to communicate with cloud SQL. Like all proxies, this requires a proxy server running at the other end, with which the proxy client can communicate with.

And when we fire up the proxy on the client side, you need to tell the client where its corresponding proxy server is, i.e., to what cloud SQL instances it should be connecting. You also need to tell it where to listen for data coming from the cloud SQL application. This is important because this will not be sent along with the standard TCP port. This is one of the reasons why this is more secure. And lastly, the client proxy also needs to be told where to find the credentials that will be used to authenticate the application. That's the client application with cloudSQL running in the cloud. Let's now come back to the question posed at the start of the video. The question was whether using a cloud proxy to access cloud SQL is a good idea. True or false? Well, this statement is true. It is a good idea to use a cloud proxy to access cloud SQL. This has to do with security. Using a cloud proxy allows your applications to connect directly to cloud SQL via a secure tunnel. In the absence of a cloud proxy, you would either need to whitelist some IP addresses, which is typically a bad idea from a security point of view, or you would need to set up SSL, which would have its own administrative overhead. So using a cloud proxy with a cloud sequel is a good idea.

Prepared by Top Experts, the top IT Trainers ensure that when it comes to your IT exam prep and you can count on ExamSnap Professional Data Engineer on Google Cloud Platform certification video training course that goes in line with the corresponding Google Professional Data Engineer exam dumps, study guide, and practice test questions & answers.