What Google Cloud Services Do I Need to use please...

ringo · 12-15-2022 07:10 AM

TL;DR - Having briefly explained my solution - is there an obvious collection of toolsets i should use in google cloud please ?

Hi,

I've build a data logger that has up to 8 sensors on it. It logs the sensor measurements up to 4 times a second in its flash memory.

Multiple data loggers are installed in a container (e.g. an off grid cabin) and are designed to run off the grid. The only way to connect to them is via Bluetooth.

When the owner of the data loggers connects to them via their smart phone (using an app i am developing), the application gets the data off the data loggers and stores it in the smartphone memory.

Then, when the smartphone has internet connectivity, the data will be sent to the cloud.

At this stage i want to be able to evaluate the data and also provide a webpage so the customer can log in and get their data that they uploaded.

I've got three different data sets:

Data defining what the "container" is. For example it will have a list of the data logger device ID's that the customer has (one entry per container)
Data defining the configuration of each data logger (one entry per data logger)
Sensor data (one entry per data logger). For example, this could be a CSV file with each row defining a timestamp and different columns storing the sensors value at that time (e.g. epoc, volts1, volts2, temperature). The CSV file will have thousands of rows.

An application user will register and log into Google Auth. The Google User UID will be stored with the above data so it can be identified.

Once the data is in the cloud I will want to analyse it using serverless operations. If i find that a sensor value is out of a pre-configured range, i will alert the user with a google notification to their smart phone.

I will create a website where the user logs in via Google Auth and is able to see their data and download it if they choose.

Having briefly explained my solution - is there an obvious collection of toolsets i should use in google cloud please ?

For example, i was thinking of using Firestore but then i started watching the youtube video "Managing IoT Storage with Google's Cloud Platform" and got completely confused.

Sorry for the long post and sorry for being such a noob. Please don't send me to AWS. Amplify is the work of the devil 😂

Kind regards,

Ringo

anjelab

As stated on the IoT-Core documentation, the product is being retired on August 16, 2023 hence using IoT storage pipeline + your choice of database is not the sensible choice in your scenario.

For your Dataset, the recommended database for your use case which involves IoT is Firestore. You can then extend Cloud Firestore with Cloud Functions for triggering and alert and use Pub-Sub for Notification.

ringo

Many thanks for this.

I have to say Databases are not my thing so i'm learning here.

A user will have several files of data as time goes on (they might upload
once a week). In my traditional way of DB thinking the data would be
translated into a DB where all the data can be queried. From what i've read
on Firestore, this doesn't happen - they stay as separate files.

I'll play around for a bit...

Thanks again,

Ringo

ringo

OK, so i've just implemented uploading a document to firestore. The document contains a header and nested data for the time (e.g. data samples taken every second) from 13:00 (21st December 2022) to 14:31 (22nd December 2022).

So now I want to append data to that document for the time from 14:31 (22nd December 2022) to 18:03 (23rd December 2022).

Reading the documentation, there is no "append" command - just "update". However, update only seems to let me change existing elements of the document - not append a whole new set of data to the document. So i'm thinking Firestore isn't the solution for me.

I can't be the only person in the world dealing with time series data - how do i use google cloud for this please ?

ringo

Looking into this a bit more. I mainly storing data for a time series - that is sensor values at points in time.

Therefore, the json would look something like this:

{
    "docType": "jts",
    "version": "1.0",
    "header": {
        "startTime": "2018-08-16T02:00:00.000Z",
        "endTime": "2018-08-16T02:20:43.000Z",
        "recordCount": 5,
        "columns": {
            "0": {
                "id": "541a5a129bc9b4035f906d70",
                "name": "Temperature",
                "dataType": "NUMBER",
                "renderType": "VALUE",
                "format": "0.###",
                "aggregate": "NONE"
            }
        }
    },
    "data": [
        {
            "ts": "2018-08-16T02:00:39.000Z",
            "f": { "0": {"v": 99, "q": 100, "a": "site maintenance"} }
        },
        {
            "ts": "2018-08-16T02:05:40.000Z",
            "f": { "0": {"v": 28.22 } }
        },
        {
            "ts": "2018-08-16T02:10:41.000Z",
            "f": { "0": {"a": "sensor recalibrated" } }
        },
        {
            "ts": "2018-08-16T02:15:42.000Z",
            "f": { "0": {"v": 29.2, "q": 100 } }
        },
        {
            "ts": "2018-08-16T02:20:43.000Z",
            "f": { "0": {"v": 29.18 } }
        }
    ]
}

And obviously over time, when more data is retrieved, more files will be sent.

To me this is a typical IoT application - yet i'm not sure Firestore can handle time series.

Am i going in the right direction? Is Firestore still the correct solution ?

Many thanks, Ringo

What Google Cloud Services Do I Need to use please - IoT project ?