Jonathan Author of Robopenguins

Creating a Library for Google Nest API Monitoring

After the Roomba and Reolink, I thought Nest devices would be a snap. How wrong I was. Google made this much, much then it had any right to be.

Long story short, if you want a well documented Python library for accessing Nest devices I ended up making since it seemed like none of the existing libraries worked with the new API Google rolled out.

This is a part four in the series of setting up checkmk monitoring for all the IoT devices on my network.

However, here connecting to the nest devices was 99% of the work since I mostly reused my system for adding devices to checkmk.

Choosing an Approach

Apparently, there used to be a way to get Nest developer accounts that a lot of earlier projects were based on. Google closed this down which invalidated a lot of the existing guides.

A bit of Googling brought up three options.

  1. Use the Google Device Access interface
  2. Google assistant nest integration / IFTTT - this actually didn’t seem to work for my setup. My Google Home app showed the devices, but the assistant didn’t seem to be able to link to them.
  3. Scrape the web interface
  4. lets you get any data Google collects from you. This would be more of a post processing approach.

One thing that made researching this difficult was that Google had killed off the API used by pretty much all the open source projects about a year ago. It appears that Nest had a series of public hacking incidents in 2019 that caused Google to lock down the interface in 2020.

It was especially disappointing that unlike the Reolink and Roomba, there appears to be no way to access the Nest over the LAN. You need to go through Googles servers to a device on your own network. There were some ancient articles about setting up shell access, but nothing in the last 5 years.

I figured I’d go with the new Google sanctioned approach since it should be the best, but I was about to learn why Google’s cloud products have a reputation for being a bit of a mess.

Creating a Google Smart Device Library

agent link lays out the process fairly well, though it gets off to a bad start:

Before creating your first project, you must register for Device Access. Registration consists of the acceptance of the Google API and Device Access Sandbox Terms of Service, along with a one-time, non-refundable fee (US$5) per account.

This probably would stop 99% of hobbyists from going any further, but after exploring the other options, and being a completionist, I pressed on.

The next issue is that the authentication system Google provides is strangely spread across multiple different types of accounts, even though they’re all provided by Google. Here’s my understanding:

  1. You add the Nest devices to your personal Google account (ie. the one you use for Gmail)
  2. You create a Google Cloud Platform project. This is used to set up the OAuth Credential system. You need to add your personal account as a test user.
  3. You create a Device Access Console project, this actually creates the endpoint you’ll be accessing the devices through. You configure the credentials you generated in 2. to access this interface.

I don’t really understand why this is split across multiple projects like this. I assume it’s legacy from the history of Nest being a separate system. Regardless, it made things much more complicated to understand, since I don’t have the firmest grasp of the how OAuth works without the extra complications.

This whole process seems more focussed on providing a way for a company (say a security company) to interface with Nest devices. There’s a whole bunch of warnings, that you would need to sign up for a commercial license and have your app validated if you wanted to set this up without having all to click through a bunch of warnings every time you log in.

Once I actually got this working, I needed wanted to make a fully featured library, so maybe people in the future wouldn’t have to suffer through this like I did.

I started by trying to use the Google API Client Python libraries. These were a maze of tools that were either deprecated, or only partially implemented. This was especially apparent for an API like this that clearly isn’t one Google focusses on. I was really disappointed coming from the AWS CLI and Python libraries which are pretty solid.

Next, I looked into generating a tool based on the “Discovery Document” This seemed like a viable way to go, but didn’t seem like it would actually make the process much easier for my use case.

Finally, I forked off one of the GitHub Nest Python projects and basically rewrote it from scratch for the new API:

Update: Of course after I finish my library, I find out there was an existing one that I somehow missed

I wanted to justify all the time I wasted on this so I went so far as to provide documentation, a CLI, and created a PyPi project

Take a look for my attempt at making an useable open source project.

Even with all this done, there’s some weird holes in the data they output. One obvious issue is that the doorbell doesn’t output whether it’s online. I know they track this internally since I’ve been notified that it was having connectivity issues in the past. It really seems like this is about as shallow an API as they could get away with.

checkmk integration

After all that effort, setting up the checkmk integration was a snap. I updated with a nest script. The only change here was that I loaded the credentials from a shared json config file, and I specified each metric as a lambda function

So now I can keep an eye on all these devices. I can nice plots of how these devices function over time, and it would be pretty easy to output more information, or log to a different interface if I wanted.

Pretty happy with checkmk, and the open source community, less happy with Google and their senseless product strategies.