Skip to main content

Scheduling Google Compute Instances with Cloud Scheduler

Our company needs to save some cost for the running staging instances. Since the instance and any related resources are used only when we are at work, we need the instance to be available only at working hours, or from Monday to Friday, 9 to 5. It approximately reduces our cost to roughly 45%. A huge diff.

diary on keyboard


The Ingredients

We need the following components from Google Cloud:
  • Google Cloud Scheduler: to run a function at an exact day/time.
  • Google Cloud Functions: to start and stop our compute instances.
  • Google Cloud Pub/Sub: as a bridge or messenger from the Cloud Scheduler to the  Cloud Functions.
  • Lastly, the Google Cloud Compute Engine Instances.
Each of those will cost us money, but the good news is we will work under the fair usage policy so it can be said that all of those will be free of charge. For example, now Google Cloud gives us 93x each month for Cloud Functions. That means if we only use it twice a day (to start and stop the instance), it will cost us nothing.

We also can use the Google Cloud HTTP Functions as an alternative for the Google Cloud Pub/Sub, but notice that there is no authentication so basically it is less secure than the Pub/Sub component.

Prepare the Compute Engine Instances

Create our compute engine instance, or edit the existing ones. You can work at any zone location, specs or anything. The most important things for this tutorial is the label of our instances should be "env=dev". You can use any value but make sure it is unique enough because we need the label as an identifier.

labels on google compute engine

Setup Google Cloud Functions with Cloud Pub/Sub

Go to your Google Cloud Functions page, and create a new function. This function will be used to start the instance.
  • Set the name to "startInstancePubSub"
  • Set the 'Trigger" to "Cloud Pub/Sub"
  • Set the Topic to "start-instance-event"
  • Set "Runtime" to "Node.js 8"
IMPORTANT. Set the starter code with the following code. The current code used as an example at the official page is obsolete for node.js 8.  This way, we will start all instance with "env=dev" label. We don't care about the zone, etc.

Here is the content of our index.js.


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
/**
 * Copyright 2018, Google, Inc.
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

// [START functions_start_instance_pubsub]
// [START functions_stop_instance_pubsub]
const Buffer = require('safe-buffer').Buffer;
const Compute = require('@google-cloud/compute');
const compute = new Compute();
// [END functions_stop_instance_pubsub]

/**
 * Starts a Compute Engine instance.
 *
 * Expects a PubSub message with JSON-formatted event data containing the
 * following attributes:
 *  zone - the GCP zone the instances are located in.
 *  label - the label of instances to start.
 *
 * @param {!object} event Cloud Function PubSub message event.
 * @param {!object} callback Cloud Function PubSub callback indicating
 *  completion.
 */
exports.startInstancePubSub = (event, context, callback) => {
  try {
    const payload = _validatePayload(
      JSON.parse(Buffer.from(event.data, 'base64').toString())
    );
    const options = {filter: `labels.${payload.label}`};
    compute.getVMs(options).then(vms => {
      vms[0].forEach(instance => {
          compute
            .zone(instance.zone.id)
            .vm(instance.name)
            .start()
            .then(data => {
              // Operation pending.
              const operation = data[0];
              return operation.promise();
            })
            .then(() => {
              // Operation complete. Instance successfully started.
              const message = 'Successfully started instance ' + instance.name;
              console.log(message);
              callback(null, message);
            })
            .catch(err => {
              console.log(err);
              callback(err);
            });
      });
    });
  } catch (err) {
    console.log(err);
    callback(err);
  }
};
// [END functions_start_instance_pubsub]

// [START functions_start_instance_pubsub]

/**
 * Validates that a request payload contains the expected fields.
 *
 * @param {!object} payload the request payload to validate.
 * @return {!object} the payload object.
 */
function _validatePayload(payload) {
//   if (!payload.zone) {
//     throw new Error(`Attribute 'zone' missing from payload`);
//   } else 
  if (!payload.label) {
    throw new Error(`Attribute 'label' missing from payload`);
  }
  return payload;
}
// [END functions_start_instance_pubsub]
// [END functions_stop_instance_pubsub]


At the package.json tab, copy and paste the following code:


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
{
  "name": "cloud-functions-schedule-instance",
  "version": "0.0.2",
  "private": true,
  "license": "Apache-2.0",
  "author": "Google Inc.",
  "repository": {
    "type": "git",
    "url": "https://github.com/GoogleCloudPlatform/nodejs-docs-samples.git"
  },
  "engines": {
    "node": ">=8.13.0"
  },
  "scripts": {
    "test": "mocha test/*.test.js --timeout=20000"
  },
  "devDependencies": {
    "@google-cloud/nodejs-repo-tools": "^3.3.0",
    "mocha": "^6.0.0",
    "proxyquire": "^2.0.0",
    "sinon": "^7.0.0"
  },
  "dependencies": {
    "@google-cloud/compute": "^0.12.0",
    "safe-buffer": "^5.1.2"
  }
}

Theoretically, the code will
  1. Search for any instances by label (our label is "env=dev"). It is not limited to 1 instance.
  2. Start the instances.
I think it doesn't matter if we have no experience using node.js before.

For "Function to execute", enter "startInstancePubSub"

Next, we are going to set the "stop instance function"
  • Create another function named "stopInstancePubSub"
  • Set everything else the same way we set up the start instance function above.
  • For pub-sub topic, set it to "stop-instance-event"
Here is our index.js code.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
/**
 * Copyright 2018, Google, Inc.
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

// [START functions_start_instance_pubsub]
// [START functions_stop_instance_pubsub]
const Buffer = require('safe-buffer').Buffer;
const Compute = require('@google-cloud/compute');
const compute = new Compute();
// [END functions_stop_instance_pubsub]

/**
 * Starts a Compute Engine instance.
 *
 * Expects a PubSub message with JSON-formatted event data containing the
 * following attributes:
 *  zone - the GCP zone the instances are located in.
 *  label - the label of instances to start.
 *
 * @param {!object} event Cloud Function PubSub message event.
 * @param {!object} callback Cloud Function PubSub callback indicating
 *  completion.
 */
exports.startInstancePubSub = (event, context, callback) => {
  try {
    const payload = _validatePayload(
      JSON.parse(Buffer.from(event.data, 'base64').toString())
    );
    const options = {filter: `labels.${payload.label}`};
    compute.getVMs(options).then(vms => {
      vms[0].forEach(instance => {
          compute
            .zone(instance.zone.id)
            .vm(instance.name)
            .start()
            .then(data => {
              // Operation pending.
              const operation = data[0];
              return operation.promise();
            })
            .then(() => {
              // Operation complete. Instance successfully started.
              const message = 'Successfully started instance ' + instance.name;
              console.log(message);
              callback(null, message);
            })
            .catch(err => {
              console.log(err);
              callback(err);
            });
      });
    });
  } catch (err) {
    console.log(err);
    callback(err);
  }
};
// [END functions_start_instance_pubsub]
// [START functions_stop_instance_pubsub]

/**
 * Stops a Compute Engine instance.
 *
 * Expects a PubSub message with JSON-formatted event data containing the
 * following attributes:
 *  zone - the GCP zone the instances are located in.
 *  instance - the name of a single instance.
 *  label - the label of instances to start.
 *
 * Exactly one of instance or label must be specified.
 *
 * @param {!object} event Cloud Function PubSub message event.
 * @param {!object} callback Cloud Function PubSub callback indicating completion.
 */
exports.stopInstancePubSub = (event, context, callback) => {
  try {
    const payload = _validatePayload(
      JSON.parse(Buffer.from(event.data, 'base64').toString())
    );
    const options = {filter: `labels.${payload.label}`};
    compute.getVMs(options).then(vms => {
      vms[0].forEach(instance => {
          compute
            .zone(instance.zone.id)
            .vm(instance.name)
            .stop()
            .then(data => {
              // Operation pending.
              const operation = data[0];
              return operation.promise();
            })
            .then(() => {
              // Operation complete. Instance successfully stopped.
              const message = 'Successfully stopped instance ' + instance.name;
              console.log(message);
              callback(null, message);
            })
            .catch(err => {
              console.log(err);
              callback(err);
            });
      });
    });
  } catch (err) {
    console.log(err);
    callback(err);
  }
};
// [START functions_start_instance_pubsub]

/**
 * Validates that a request payload contains the expected fields.
 *
 * @param {!object} payload the request payload to validate.
 * @return {!object} the payload object.
 */
function _validatePayload(payload) {
  // ignore the zone
  //  if (!payload.zone) {
  //    throw new Error(`Attribute 'zone' missing from payload`);
  //  } else 
  if (!payload.label) {
    throw new Error(`Attribute 'label' missing from payload`);
  }
  return payload;
}
// [END functions_start_instance_pubsub]
// [END functions_stop_instance_pubsub]


Then, replace the package.json tab to the following code:


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
{
  "name": "cloud-functions-schedule-instance",
  "version": "0.0.2",
  "private": true,
  "license": "Apache-2.0",
  "author": "Google Inc.",
  "repository": {
    "type": "git",
    "url": "https://github.com/GoogleCloudPlatform/nodejs-docs-samples.git"
  },
  "engines": {
    "node": ">=8.13.0"
  },
  "scripts": {
    "test": "mocha test/*.test.js --timeout=20000"
  },
  "devDependencies": {
    "@google-cloud/nodejs-repo-tools": "^3.3.0",
    "mocha": "^6.0.0",
    "proxyquire": "^2.0.0",
    "sinon": "^7.0.0"
  },
  "dependencies": {
    "@google-cloud/compute": "^0.12.0",
    "safe-buffer": "^5.1.2"
  }
}


Then, we need to set "Function to Execute" to "stopInstancePubSub".

We can optionally test the function using the following Triggering Event:

We can use this command on our console to make the base64-encoding.

echo '{"label":"env=dev"}' | base64
eyJsYWJlbCI6ImVudj1kZXYifQo=

Setup the Google Cloud Scheduler

Our functions will not work if we don't trigger them. That's why we need the Google Cloud Scheduler component: to trigger our start/stop functions at an exact day/time.

  • Go to the Google Scheduler page and create a new job named "startup-dev-instances".
  • Set the Frequency to your preference. 0 9 * * 1-5. That means we will start the instance from the 1st day of the week to 5th day of the week (1-5), at 9 AM (0 9).
  • Select our related PubSub topic as the Target
  • For payload, enter the following:
    {"label":"env=dev"}

Then, create the exact job as the start one, except for the name of the scheduler and Frequency.

  • Set the Frequency to 0 17 * * 1-5. That's it, we just create a scheduler to call the PubSub topic from Monday to Friday, at 17:00 or 5 PM.
  • That's it. We are almost done.

Bear in mind that the functions will run at the exact time, but with a random order if we have multiple instances to be started/stopped. In our case, we need to start the A instance first, before the B and C running. That way, I create another Google Scheduler which starts 15 minutes earlier to the B and C instances.

We can modify the flow or the starter code to be more suitable to our needs. I hope you enjoy the article. Don't hesitate to ask a question and give me your comment below. Thank you!

Comments

Popular posts from this blog

About Prabowo Murti

Assalaamu'alaykum.

Hello. My name is Prabowo Murti. This website is made to write all things about programming, business, religion (especially Islam), my family, minimalism, etc, basically anything that related to my journey in life. I will do my best to write all articles in English, not because I am good at it, but just for practice.



About Me I obtained my bachelor's degree in Computer Science from Gadjah Mada University. I love to read and write. I was born in a small town named Kubu Raya, in West Kalimantan. Now I live in Tangerang, Banten. Let's say I am proud to be a muslim and an Indonesian.

I believe that we can make this world a better place to live by our writings.

If you have any question, feel free to ask by writing an email to prabowo.murti at Gmail or submit your comment (at the end of each article). I also have some digital footprints elsewhere.

Muhajirin Dotnet It is a small company I built from 2008. It also has a subdomain named Blog Belajar Bisnis, captur…

Learn Nested Set on Yii2 Framework

The nested set behaviour is an approach to store hierarchical data in relational databases. For example, if we have many categories for our product or items. One category can be a "parent" for other categories, means that one category consists of more than one category. The model can be drawn using a "tree" model. There are other approaches available but what we will learn in this article is specifically the NestedSetsBehavior made by  Alexander Kochetov, which utilizing the Modified Preorder Tree Traversal algorithm.

Install Yii2 framework advanced template
My assumption is that we have our standard installation of Yii2 framework advanced template. You can read more about advanced template from the repository page on Github: https://github.com/yiisoft/yii2-app-advanced. Make sure that our installation is working by calling the default URL localhost:8080 after running the following command

$ ./yii serve --docroot="backend/web"

We also need to make sure tha…

Improving Our Ignoring Skills

We live in a world full of information. Unfortunately, most of them are junk. As what Daniel Pink said on his book, "A Whole New Mind: Why Right-Brainers will Rule the World", abundance is one the three forces which changes our era to what he called "Conceptual Age". All people around the globe get wealthier, have more materials, buy more stuff, but not fulfilment. Looks rich from the outside, but not happier.
We need to differ the one which useful and the ones which don’t deserve our attention. Most of us nowadays are already aware of advertisement that pops up or shown at the sidebar. We already know that ads should be blocked using Chrome extensions such as AdBlock or any other similar stuff. Our unconscious mind tells us about how to not stare at the sales person’s eyes, how not to give a $hit. The small man that lays inside our head makes us reject the phone call from a number prefixed by 021-555 or any other similar phone number because we have a strong feel…

DIY Table Legs with PVC for Under $5

I now have two pieces of IKEA Lerberg table legs. It's a great product and has a minimalist design. But the price was about $20 each. My pocket is not deep enough, so I decided to make a similar structure but with cheaper materials. Half inch PVC pipes come in rescue. The total cost for the entire project is about IDR 65,000 or still under 5 dollars.


For this DIY project, we need to make two table legs. Each leg needs:

1 x 21.5 cm: middle leg (bottom)
1 x 18.3 cm: middle segment
2 x 17.8 cm: lower legs (right and left)
4 x 10.9 cm: this depends on your top table's width
2 x 47 cm: upper legs (right and left)
1 x 43.9 cm: middle leg (upper)
5 x T-shape fittings
2 x L-shape fittings


Each leg needs about 2.6 metres so in total we need ~5.2 metres. That means 2 x 4 metres half inch pipes. Each 4 m pipe costs about IDR 18,000. Each fitting costs about IDR 2,000. We also need an iron saw. I get one for about IDR 5,000, but I am a little bit disappointed about the quality. The IDR 20…