This article provides a high level overview of how we have configured multiple Looker deployments to separate development and production environments for both LookML and dashboards. Having a Looker deployment on a single instance has many challenges which include:

• a dashboard can get edited when someone else is presenting it to a client or a stakeholder
• someone testing a large dashboard can slow down Looker which will affect other dashboards
• Looker users cannot test the effects of model changes to their dashboards without having to request for developer access which has cost implications

Multiple Looker instances

We have set up 3 Looker instances: a development instance where LookML updates are made and dashboards created and reviewed before being released to production, and two production instances. One of the production instances is for internal looks and dashboards and the other for creating dashboards that can be shared with external clients.

All three Looker instances have a standardized data connection to read from the same data warehouse. This is to ensure that dashboards and looks work the same access all environments. To ensure all development actually occurs on the development instance, both production instances enforce view only access for all users.

This set up allows us to test both data models and dashboards without affecting production content. It also leads to standardized data models across all production instances.

Linking development to production

To take advantage of the benefits of multiple Looker instances, while limiting the operational maintance, we have linked all three looker instances using GitHub, and use GitHub Releases to update the production instances whenever changes have been tested and approved in development.

To release dashboards from development to production we use Backstage, which is also our developer portal. We have created a template which dashboard builders use to release ready-to-use content to production instances, but any type of trigger would work. We have also integrated with slack for notifications.

This setup can be achieved using the steps outlined below.

Step 1: Create a new GitHub repository to host your LookML code.

Step 2: Enable GitHub integration in all Looker instances, and enable either Pull Requests Recommended or Pull Requests Required options. Steps to enable GitHub integration can be found in this link. All instances should point to the repository created in step 1.

Step 3: Configure deploy webhooks in GitHub, one for each instance. These are used to deploy the head commit from the production branch after a push event or a release. The URLs take the format <looker_instance_url>webhooks/projects/<looker_project_name>/deploy. For example https://company.looker.com/webhooks/projects/test/deploy. Steps to add the URLs to the GitHub repo can be found here.

Our current set up is to trigger the development instance deploy URL using a push event and the production instances deploy URLs using releases. When triggered, these webhooks deploy to the production mode in the respective instances. To change the triger event go to the LookML repo > Settings > Webhooks and click on the webhook you want to modify.

LookML and content development flow

The setup above allows us to achieve the development flow below.

LookML development flow

1. A developer modifies LookML code and creates a PR
2. PR is reviewed and merged. This auto-deploys the changes to the production mode in the development instance using the deploy webhook for the developer instance. Looker users and dashboard builders can test the updated models in the development instance without requiring developer permissions.
3. Finally, a release is created in GitHub which auto-deploys to the production mode in the production instances.

Content development flow

1. A Looker user creates a new dashboard in the development instance and has it reviewed.

2. The user makes a request in Backstage to release the dashboard to production. This request is forwarded to GitHub Actions which runs a Python script which uses Gazer to do the actual dashboard migration. Any trigger can be used in this step as long as the request is ultimately forwarded to Github Actions.

3. An alert is send to Slack on the status of the release, successful or failed. The message also includes a link to the released dashboard in the destination instance.

Below is a summary of the content development and release workflow.

Conclusion

Looker is extremely powerful and extensive as a Business Intelligence platform. We hope this article is useful to anyone trying to connect multiple Looker deployments, or anyone trying to separate development and production environments.

We are interested in hearing about your current Looker multi-instance setup so feel free to let us know in the comments.

Premise is constantly looking to hire top-tier engineering talent to help us improve our front-end, mobile offerings, data and cloud infrastructure. Come find out why we’re consistently named among the top places to work in Silicon Valley by visiting our Careers page.

Configuring a Multi-Instance Looker Deployment was originally published in Engineering at Premise on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this post, we’ll detail our solution for publishing, and consuming, Kotlin Multiplatform Swift Packages. Our solution leverages a custom Gradle plugin publishing XCFrameworks to Google Cloud Storage and a Google Cloud Run service to download Swift Package binaries requested by XCode. With this solution in place, we’ve been able to more efficiently serve multiple Kotlin Multiplatform libraries to our iOS application.

This is a part of an ongoing series on our usage of Kotlin Multiplatform at Premise:

• Part 1: Kotlin Multiplatform at Premise
• Part 2: Kotlin Multiplatform Project Structure for Integrating with Brownfield Applications
• Part 3: Building a CI Pipeline for Kotlin Multiplatform Mobile Using GitHub Actions
• Part 4: Publishing Kotlin Multiplatform Swift Packages Using Google Cloud Storage and Cloud Run — This Post
• Part 5: Generating BuildConfig Files for a Kotlin Multiplatform Library — Coming Soon
• Part 6: Optimizing Local Build Times for Kotlin Multiplatform Mobile Projects — Coming Soon

Premise and Kotlin Multiplatform Swift Packages

We’ve been using Kotlin Multiplatform in production since early 2021 in the form of our mobile-shared project.

During that time, we’ve consumed our shared code as a Swift Package within our iOS application. The integration of that Swift Package has gone through several iterations.

• v1: Use the multiplatform-swiftpackage plugin to build the Swift Package and store the XCFramework binary in GitHub
• v2: Use our own custom Gradle plugin to build the Swift Package and store the XCFramework binary in GitHub

These two solutions were very similar. Build the XCFramework. Generate the Package.swiftfile. Check both into git with the desired version tag.

These approaches worked fine for a while, but eventually we started to pay the price for our simple initial solution.

An XCFramework binary can be pretty large. Ours were in the ballpark of 100MB. So checking 2–3 of these into each commit (1 for each iOS architecture) started to really increase the overall download size of our git repo.

Why is this a problem?

Well, Swift Package Manager does a full clone of the git repository when it has to sync package dependencies or reset package caches. So, when our developers had to switch branches with difference library versions, or had to reset their Xcode caches, they had to download GBs worth of XCFramework binaries; thereby dramatically slowing down their Xcode sync.

Once we determined what was causing this slow down, we knew we needed a new approach. We decided to move to a remote url distribution model for our Kotlin Multiplatform Swift Packages. Rather than check the binary into source control, we would upload the binary to some remote location, acquire the url to that resource, and add that url, along with the checksum, to our Package.swift file.

This is conceptually pretty simple. However, there wasn’t an ideal out-of-the-box solution at the time. So, we built a solution ourselves which has been serving us well in managing the publication and integration of Swift Packages for multiple Kotlin Multiplatform projects.

The solution is broken down into three parts:

• Uploading XCFramework binaries to Google Cloud Storage
• Publishing the Package.swift file to GitHub
• Consuming the Swift Package from Xcode using Google Cloud Run

Uploading XCFramework Binaries to Google Cloud Storage

As part of our Kotlin Multiplatform project, we have a custom Gradle plugin named SwiftPackagePlugin that adds several key tasks to our build.

class SwiftPackagePlugin : Plugin<Project> {
    override fun apply(target: Project) {
        target.registerMoveXCFrameworkTask()
        target.registerZipFileTask()
        target.registerSwiftPackageTask()
        target.registerLocalSwiftPackageTask()
    }
}

registerMoveXCFrameworkTask moves a build XCFramework from the build directory to our desired output directory location

/**
 * Creates a moveXCFramework Gradle task to move files from the
 * /build/swiftpackage directory to a /swiftpackage output directory
 */
private fun Project.registerMoveXCFrameworkTask() {
    tasks.register("moveXCFramework", Copy::class.java) {
        group = TASK_GROUP_NAME
        description = "Moves XCFramework into swiftpackage output directory"

        dependsOn("assemble${rootProject.name.capitalized()}XCFramework")

        from("${this@registerMoveXCFrameworkTask.buildDir}/$OUTPUT_FILE_DIR_NAME")
        into(File(OUTPUT_FILE_DIR_NAME))
    }
}

registerZipFileTask creates a zip containing the build XCFrameworks

/**
 * Creates a zipXCFramework Gradle task
 */
private fun Project.registerZipFileTask() {
    tasks.register("zipXCFramework", Zip::class.java) {
        group = TASK_GROUP_NAME
        description = "Creates a ZIP file for the XCFramework"

        // dependsOn assembleMobilesharedXCFramework task
        dependsOn("assemble${rootProject.name.capitalized()}XCFramework")

        archiveFileName.set(project.getZipFileName())
        destinationDirectory.set(File("${this@registerZipFileTask.buildDir}/$OUTPUT_FILE_DIR_NAME"))
        from("${this@registerZipFileTask.buildDir}/XCFrameworks/release")
    }
}

registerLocalSwiftPackageTask constructs a Package.swift file useful for locally testing changes in our iOS project

/**
 * Creates a createLocalSwiftPackage Gradle task that generates a
 * Package.swift file needed for the generated XCFramework so it
 * can be consumed as a local Swift Package
 */
private fun Project.registerLocalSwiftPackageTask() {
    tasks.register("createLocalSwiftPackage") {
        group = TASK_GROUP_NAME
        description = "Creates a local Swift package to distribute the XCFramework"

        dependsOn("zipXCFramework")
        finalizedBy("moveXCFramework")

        doLast {
            val packageDotSwiftFile = createPackageDotSwiftFile(
                outputDirName = OUTPUT_FILE_DIR_NAME,
                filename = OUTPUT_PACKAGE_CONFIG_FILE_NAME
            )

            val packageConfiguration = SwiftPackageConfiguration.local(
                packageName = rootProject.name,
                zipFileName = project.getZipFileName()
            )

            packageDotSwiftFile.writePackageDotSwiftFile(packageConfiguration)
        }
    }
}

registerSwiftPackageTask uploads a zipped set of XCFrameworks to Google Cloud Storage, builds a SwiftPackageConfigurationand then constructs a Package.swift file pointing to this remote file.

/**
 * Generates the Package.swift file needed for the generated
 * XCFramework so it can be consumed as a Swift Package
 */
private fun Project.registerSwiftPackageTask() {
    tasks.register("createSwiftPackage") {
        group = TASK_GROUP_NAME
        description = "Creates the Swift package to distribute the XCFramework"

        dependsOn("zipXCFramework")

        doLast {
            val packageDotSwiftFile = createPackageDotSwiftFile(
                outputDirName = OUTPUT_FILE_DIR_NAME,
                filename = OUTPUT_PACKAGE_CONFIG_FILE_NAME
            )

            uploadeToCloudStorage(
                projectId = project.swiftPackageGCPProjectId,
                bucketName = project.swiftPackageBucketName,
                packageName = project.getZipFileName(),
                filePath = "${project.buildDir}/$OUTPUT_FILE_DIR_NAME/${project.getZipFileName()}"
            )

            val packageConfiguration = SwiftPackageConfiguration.remote(
                packageName = rootProject.name,
                zipFileName = project.getZipFileName(),
                zipChecksum = project.zipFileChecksum().trim(),
                distributionUrl = project.swiftPackageDistributionURL
            )

            packageDotSwiftFile.writePackageDotSwiftFile(packageConfiguration)
        }
    }
}

Uploading to Google Cloud Storage

The implementation of uploadToCloudStorage looks like the following. This relies on the com.google.cloud:google-cloud-storage:<version> dependency for interacting with Cloud Storage. By extension this also means the task must be run from an environment authenticated with Google Cloud. Locally, this generally means authentication via application-default credentials. In CI, we use the google-github-actions/auth GitHub Action to authenticate via a service account that has permissions to upload to our desired Cloud Storage bucket.

/**
 * Uploads a specified XCFramework zip file to a specified Cloud Storage bucket
 *
 * @param projectId The ID of your GCP project
 * @param bucketName The Cloud Storage bucket, within the specific project, to upload zip files to
 * @param packageName The name of the XCFramework zip file
 * @param filePath Path to the file to upload
 */
private fun uploadeToCloudStorage(projectId: String, bucketName: String, packageName: String, filePath: String) {
    val storage = StorageOptions.newBuilder().setProjectId(projectId).build().service
    val blobId: BlobId = BlobId.of(bucketName, packageName)
    val blobInfo: BlobInfo = BlobInfo.newBuilder(blobId).build()

    // Optional: set a generation-match precondition to avoid potential race
    // conditions and data corruptions. The request returns a 412 error if the
    // preconditions are not met.
    val precondition = if (storage[bucketName, packageName] == null) {
        // For a target object that does not yet exist, set the DoesNotExist precondition.
        // This will cause the request to fail if the object is created before the request runs.
        Storage.BlobWriteOption.doesNotExist()
    } else {
        // If the destination already exists in your bucket, instead set a generation-match
        // precondition. This will cause the request to fail if the existing object's generation
        // changes before the request runs.
        Storage.BlobWriteOption.generationMatch(
            storage[bucketName, packageName].generation
        )
    }

    storage.createFrom(blobInfo, Paths.get(filePath), precondition)
}

Building a Package.swift file

To build the desired Package.swift file we construct an instance of a SwiftPackageConfiguration which will hold the values we want to write out to disk

private data class SwiftPackageConfiguration private constructor(
    private val packageName: String,
    private val swiftToolsVersion: String,
    private val platforms: String,
    private val zipFileName: String,
    private val zipChecksum: String,
    private val isLocal: Boolean,
    private val distributionUrl: String,
) {

    val templateProperties = mapOf(
        "toolsVersion" to swiftToolsVersion,
        "name" to packageName,
        "zipName" to zipFileName,
        "platforms" to platforms,
        "isLocal" to isLocal,
        "checksum" to zipChecksum,
        "url" to "$distributionUrl/$zipFileName"
    )

    companion object {
        internal val templateFile =
            SwiftPackagePlugin::class.java.getResource("/templates/$OUTPUT_PACKAGE_CONFIG_FILE_NAME.template")

        fun local(
            packageName: String,
            swiftToolsVersion: String = DEFAULT_SWIFT_TOOLS_VERSION,
            platforms: String = DEFAULT_IOS_PLATFORM_VERSION,
            zipFileName: String
        ) = SwiftPackageConfiguration(packageName, swiftToolsVersion, platforms, zipFileName, "", true, "")

        fun remote(
            packageName: String,
            swiftToolsVersion: String = DEFAULT_SWIFT_TOOLS_VERSION,
            platforms: String = DEFAULT_IOS_PLATFORM_VERSION,
            zipFileName: String,
            zipChecksum: String,
            distributionUrl: String,
        ) = SwiftPackageConfiguration(packageName, swiftToolsVersion, platforms, zipFileName, zipChecksum, false, distributionUrl)
    }
}

With this class, we then configure our instance as follows

val packageConfiguration = SwiftPackageConfiguration.remote(
                packageName = rootProject.name,
                zipFileName = project.getZipFileName(),
                zipChecksum = project.zipFileChecksum().trim(),
                distributionUrl = project.swiftPackageDistributionURL
            )

With an instance of this SwiftPackageConfiguration we then build our Package.swift file by using a template and subsituting in our desired configuration values.

// Writes the Package.swift file by
// first loading from a template file and
// second subtituting in the config values
private fun File.writePackageDotSwiftFile(packageConfiguration: SwiftPackageConfiguration) {
    SimpleTemplateEngine()
        .createTemplate(SwiftPackageConfiguration.templateFile)
        .make(packageConfiguration.templateProperties)
        .writeTo(writer())
}

The base template file lives in /src/main/resources/templates/Package.swift.template

// swift-tools-version:$toolsVersion
import PackageDescription

let package = Package(
    name: "$name",
    platforms: [
        $platforms
    ],
    products: [
        .library(
            name: "$name",
            targets: ["$name"]
        ),
    ],
    targets: [
<% if (isLocal) print """        .binaryTarget(
            name: "$name",
            path: "./${zipName}"
        ),""" else print """        .binaryTarget(
            name: "$name",
            url: "$url",
            checksum: "$checksum"
        ),""" %>
    ]
)

When the configuration is applied to the template, it resulst in an output Package.swift file that looks like the following:

// swift-tools-version:5.8.0
import PackageDescription

let package = Package(
    name: "mobileshared",
    platforms: [
        .iOS(.v14)
    ],
    products: [
        .library(
            name: "mobileshared",
            targets: ["mobileshared"]
        ),
    ],
    targets: [
        .binaryTarget(
            name: "mobileshared",
            url: "https://<cloud run url>/swiftpackage/mobileshared-<version>.zip",
            checksum: "<XCFramework zip checksum>"
        ),
    ]
)

Note the url value. This is a url pointing to a Google Cloud Run service which will be described below. The purpose of this service is to give us a known location to build up our artifact urls so we can add them to this Package.swift file.

Additional plugin code

To round out the plugin implementation, we have a handful of of other configuration values and helper functions/classes.

private const val DEFAULT_SWIFT_TOOLS_VERSION = "5.8.0"
private const val DEFAULT_IOS_PLATFORM_VERSION = ".iOS(.v14)"

private val Project.swiftPackageGCPProjectId
    get() = System.getenv("SWIFT_PACKAGE_BUCKET_GCP_PROJECT_ID") ?: ""

private val Project.swiftPackageBucketName
    get() = System.getenv("SWIFT_PACKAGE_BUCKET_NAME") ?: ""

private val Project.swiftPackageDistributionURL
    get() = System.getenv("SWIFT_PACKAGE_DISTRIBUTION_URL") ?: ""

private fun Project.createPackageDotSwiftFile(outputDirName: String, filename: String): File {
    return File("$buildDir/$outputDirName", filename).apply {
        parentFile.mkdirs()
        createNewFile()
    }
}


private const val TASK_GROUP_NAME = "Premise Swift Package"
private const val OUTPUT_PACKAGE_CONFIG_FILE_NAME =  "Package.swift"
private const val OUTPUT_FILE_DIR_NAME = "swiftpackage"
private fun Project.getZipFileName() = "${rootProject.name}-${rootProject.version}.zip"

internal fun Project.zipFileChecksum(): String {
    val outputPath = "${buildDir}/$OUTPUT_FILE_DIR_NAME"
    logger.info("checksum for path: $outputPath")
    logger.info("checksum for file: ${getZipFileName()}")
    return File(outputPath, getZipFileName())
        .takeIf { it.exists() }
        ?.let { zipFile ->
            ByteArrayOutputStream().use { os ->
                project.exec {
                    workingDir = File(outputPath)
                    executable = "swift"
                    args = listOf("package", "compute-checksum", zipFile.name)
                    standardOutput = os
                }
                os.toString()
            }
        } ?: ""
}

Publishing the Package.swift File to GitHub

With our SwiftPackagePlugin applied to our project, we can run the createSwiftPacakge task from CI to build the binaries, upload them to Google Cloud Storage, and generate a Package.swift referencing that version.

However, we will need to publish that Package.swift file to GitHub so Xcode can reference it, and use it to eventually download the binaries.

To do this, we are using GitHub actions. The following, is a snippet from our release build that includes the neccessary job steps to build and publish the Swift Package. (this does not include the neccessary gcloud env setup)

# runs our Gradle tasks for building and uploading
- name: Create Swift Package
  uses: gradle/gradle-build-action@v2.4.2
  with:
    arguments: createSwiftPackage

# copies Package.swift to desired location in repo so Xcode sees it
- name: Prepare iOS Artifacts
  run: |
    cp -r shared/build/swiftpackage/Package.swift .

# commits the updated Package.swift and tags it so Xcode can find it
- name: Tag Release
  run: |
    git add --all
    git commit -m "[skip ci] Update Package.swift to version ${{ env.ARTIFACT_VERSION }}"
    git tag -a ${{ env.ARTIFACT_VERSION }} -m "mobile-shared v${{ env.ARTIFACT_VERSION }}"
    git push
    git push origin ${{ env.ARTIFACT_VERSION }}

Consuming the Swift Package from Xcode Using Google Cloud Run

From Xcode, we can now point our project at GitHub and ask it for a specific tagged version. Xcode and Swift Package Manager will then

• download the repo
• examine the Package.swift file for the tagged commit
• extract the binaryTarget url
• attempt to download the binary

When building the Package.swift file, we construct the url using the name of the shared artifact, the version of that artifact, and add those values to a known service url.

https://<cloud run service url>/swiftpackage/<artifact name>-<version>.zip

That service is our swift-package-distribution-service .

It’s a Google Cloud Run service that is permissioned to access our Cloud Storage bucket and stream the requested files back for download.

Implementing swift-package-distribution-service

How you would go about implementing your own version of this type of service is for you to decide. We built our service with Python using Goblet. Goblet is a framework designed to streamline the creation og Google Cloud Services using Python.

With Goblet, our entire service looks like the following:

import io
import os
from goblet import Goblet, goblet_entrypoint, Response
from flask import stream_with_context
from google.cloud import storage
import base64

CHUNK_SIZE = 8192
BUCKET_NAME = os.environ.get("CLOUD_STORAGE_BUCKET")
SWIFT_DISTRIBUTION_USERNAME = os.environ.get("SWIFT_DISTRIBUTION_USERNAME")
SWIFT_DISTRIBUTION_TOKEN = os.environ.get("SWIFT_DISTRIBUTION_TOKEN")

app = Goblet(function_name="swift-package-distribution-service", backend="cloudrun", routes_type="cloudrun")
goblet_entrypoint(app)


@app.route('/swiftpackage/{name}')
def home(name: str):
    auth_header = get_auth_token()

    if not is_token_valid(auth_header):
        return Response("Invalid token", None, 401)

    swiftpackage_filename = f'{name}'
    storage_client = storage.Client()

    bucket = storage_client.get_bucket(BUCKET_NAME)
    if bucket is None or not bucket.exists():
        missing_bucket_msg = f'Bucket {bucket} does not exist'
        app.log.info(missing_bucket_msg)
        return Response(missing_bucket_msg, None, 404)
    else:
        app.log.info(f'Located bucket {bucket}')

    swiftpackage_blob = bucket.get_blob(swiftpackage_filename)
    if swiftpackage_blob is None or not swiftpackage_blob.exists():
        missing_blob_msg = f'Requested Swift Package {swiftpackage_filename} does not exist'
        app.log.info(missing_blob_msg)
        return Response(missing_blob_msg, None, 404)
    else:
        app.log.info(f'Located file {swiftpackage_filename} and started download')

    swiftpackage_content = swiftpackage_blob.download_as_bytes()

    app.log.info("Streaming response")
    return stream_response(swiftpackage_content, swiftpackage_filename)


def stream_response(content: bytes, filename: str):
    return stream_with_context(
        read_bytes_chunks(content)), \
        200, \
        {
            "Content-Type": "application/octet-stream",
            "Content-Disposition": f'attachment; filename={filename}'
        }


def read_bytes_chunks(content: bytes):
    app.log.info("Reading chunks")
    content_stream = io.BytesIO(content)
    with content_stream:
        while 1:
            buffer = content_stream.read(CHUNK_SIZE)
            if buffer:
                app.log.info("chunk")
                yield buffer
            else:
                app.log.info("no chunks left")
                break


def get_auth_token() -> str:
    # We'll skip this specific implementation
    # We include a token taken from .netrc on developers' and CI machines
    # If the token is not present, or incorrect, we will reject the request
    # This is used due to Swift Package Manager not allowing us to auth via gcloud


def is_token_valid(auth_header: str) -> bool:
    # We'll skip this specific implementation


def decode_auth_header(auth_header: str) -> str:
    # We'll skip this specific implementation

With this service deployed, we can use its url when building our Package.swift file.

https://<cloud run service url>/swiftpackage/<artifact name>-<version>.zip

And now, when Xcode tries to download the file referenced in Package.swift it makes a request against this service, validates the auth token, and downloads the desired XCFramework binaries.

From there, our iOS build is off and running.

How Has This Solution Worked?

We’ve been using this solution for 6+ months now without issue.

By enabling us to move away from checking binaries into GitHub, it dramatically improved package sync time for our iOS devs that were regularly switching between branches requiring different versions of our mobile-shared project.

Because the plugin, and service, are general purpose, we were able to apply the same plugin and deployment model for our second Kotlin Multiplatform project in the org.

Because the plugin and service were both already in place, applying this approach to the second project was trivial; not requiring any real new development work beyond configuring the plugin in the new project.

What Are The Drawbacks To This Solution?

Perhaps the biggest drawback to this solution is that it requires cloud resources for storage and request handling. In an organization already using cloud resources this is likely not an issue. However, for open source projects or hobbyists, this could be a significant barrier to entry; both in terms of cost required to run these resources and the administration/security of cloud resources/organizations/etc.

On a similar note, another drawback is that it’s not an off-the-shelf solution. It required dev work leveraging knowledge of Swift Package Manager, Gradle plugins, and Google Cloud. This might not be something every time has access to and could be an additional barrier to entry. For a more, off-the-shelf option, I recommend looking into KMMBridge from TouchLab.

Finally, another, albiet smaller, limitation of the current implementation is the ordering of binary upload and Package.swift file creation. If uploading the XCFramework binary to Cloud Storage succeeds, but creation of the Package.swift file fails, it would result in a stranded artifact in our Cloud Storage bucket. In practice, this wouldn’t result in any significant cost unless it was happening repeatedly; and in 6+ months of using the current implementation I don’t believe we’ve ever encountered this issue.

Premise is constantly looking to hire top-tier engineering talent to help us improve our front-end, mobile offerings, data and cloud infrastructure. Come find out why we’re consistently named among the top places to work in Silicon Valley by visiting our Careers page.

Publishing Kotlin Multiplatform Swift Packages Using Google Cloud Storage and Cloud Run was originally published in Engineering at Premise on Medium, where people are continuing the conversation by highlighting and responding to this story.

Dataform is an amazing new offering from Google Cloud Platform (GCP) that allows for serverless orchestration for data pipelines. Previously, data engineering pipelines were either supported by separate data engineering teams, which many smaller organizations don’t have the resources to support, or were stitched together using a series of interrelated BigQuery scheduled queries, which could be time-consuming to build and error-prone to run and maintain.

Dataform solves these problems, especially for data scientists and data analysts, by including features such as

• Ability to build data pipelines using SQL
• Integrates with GitHub and GitLab for code versioning and deployment
• Monitor pipelines for failures and trigger alerts

The main limitation of Dataform is that it only supports SQL, with limited ability to add javascript, but this limitation can be overcome with BigQuery Remote Functions. BigQuery Remote Functions allow you to call external services running in Cloudrun or Cloudfunctions from BigQuery, making it possible to have a serverless data pipeline that can run custom logic in any programming language that you may need.

The rest of this blog post will go through a complete example on how to setup both Dataform and BigQuery Remote Functions in just a few steps.

Setup Dataform

We will begin by setting up a simple Dataform pipeline in the GCP console. Navigate to Dataform and create a repository.

Make sure to grant the default Dataform service account service-PROJECT_ID@gcp-sa-dataform.iam.gserviceaccount.com BigQuery read and write permissions. This can be accomplished using the roles/bigquery.user role.

Next create a new development workspace and allow GCP to initialize it, which will add the default files needed to run the pipeline.

Next create a new BigQuery dataset called tutorial , and inside a table called test with schema id: int, val: int, type: str .

We can insert some test data by running the following query

INSERT `PROJECT.tutorial.test` (id, val, type) VALUES (1,1,"double"),(2,2,"square"),(3,3,"zero") 

Now lets return back to Dataform and in the file dataform.json update the defaultSchema to be the dataset tutorial. This ensures that Dataform creates any new tables in this dataset, and will allow us to deploy our goblet application correctly.

Next we will update the definitions/first_view.sqlx file to create a new table from the table we selected in the previous step.

config { type: "table" }

SELECT * FROM PROJECT.DATASET.TABLE

We will now setup the BigQuery Remote Function before coming back and updating the second_view.sqlx.

Setup BigQuery Remote Function

Setting up the BigQuery Remote Function requires a few steps, including creating a Cloudrun or Cloudfunction endpoint, creating a BigQuery routine, creating a BigQuery connection, and setting up the IAM permissions.

Luckily we can leverage Goblet, an open source framework for writing, deploying, and managing serverless REST APIs and related GCP resources. With Goblet, we will be able to simply write our API code, while Goblet will handle all of the deployment steps. The source code for the following steps can be found in Github.

To setup our Goblet application we need a requirements.txt that includes goblet-gcp .

We also need to pip install goblet locally in order to deploy our application, which can be done by running the commandpip install goblet-gcp.

Next we will setup our main.py , which will contain our function logic. In our example we will create a remote function that takes in a val and a type and returns an int. Note that we can put any logic we would like in this function including calls to a machine learning model.

from goblet import Goblet, goblet_entrypoint

app = Goblet(function_name="goblet")
goblet_entrypoint(app)


@app.bqremotefunction(dataset_id="tutorial", location="US")
def tutorial(value: int, type: str) -> int:
    if type == "double":
        return value * 2
    if type == "square":
        return value**2
    if type == "zero":
        return 0
    else:
        return value

To deploy the cloudfunction, the BigQuery connection, the BigQuery routine, and the correct invoker permissions, we simple need to run goblet deploy -p PROJECT -l LOCATION .

(To read a full writeup on Goblet and BigQuery Remote Functions, check out Tutorial: Deploying BigQuery Remote Functions)

Setup Dataform with BigQuery Remote Function

Now that we have deployed our BigQuery Remote Function we are able to reference it in Dataform in defintions/second_view.sqlx , by using the following code

config { type: "table",
columns:{
             id: "id",
             val:"val",
             type:"type"
      } }

SELECT id, `PROJECT.tutorial`.goblet_tutorial(val, type) as val, type FROM ${ref("first_view")}

Note how we call the BigQuery remote function we created in the previous section by using PROJECT.tutorial`.goblet_tutorial(val, type) , where val and type fields are passed to our Goblet application, and an int is returned, which we save as val in our new table.

We can view the full pipeline by looking at the compiled graph in Dataform.

Executing the Dataform Pipeline

Now that we have created a BigQuery test dataset, a Dataform pipeline and deployed a BigQuery Remote Function, all that is left is to execute our pipeline!

This can be triggered in the console, by clicking the start execution dropdown and selecting all actions .

The results of the pipeline can been seen in our test dataset in the table second_view.

Conclusion

Dataform is a powerful new offering from GCP that allows for serverless orchestration for data pipelines. The main limitation of using Dataform compared to other workflow tools, is that Dataform is only able to run SQL queries. However, we have shown that we can actually add any custom logic into our pipelines using BigQuery Remote Functions in just a few simple steps. This makes Dataform a powerful tool, that can handle a wide variety of use cases, especially for data scientists and data analysts.

Feel free to leave a comment letting us know your interesting use cases for Dataform and BigQuery Remote Functions and don’t forget to star our open source project Goblet.

Premise is constantly looking to hire top-tier engineering talent to help us improve our front-end, mobile offerings and cloud infrastructure. Come find out why we’re consistently named among the top places to work in Silicon Valley by visiting our Careers page.

Serverless Data Pipelines in GCP using Dataform and BigQuery Remote Functions was originally published in Engineering at Premise on Medium, where people are continuing the conversation by highlighting and responding to this story.

I have recently been experimenting with mutual exclusion(Mutex), a concept in software engineering to solve race conditions when accessing shared resources.

Let’s assume we have a component with a refresh function that can be accessed from multiple threads or coroutines:

In the implementation of this component, the function retrieves data from a remote resource and inserts it into the local database. However, these operations need to be atomic. Now, imagine if 10 threads call this function simultaneously, or even a few milliseconds apart. What happens? If you guessed that we make 10 network calls and 10 database IO operations to save the items, then you would be right, as confirmed by the below test:

But that’s not what we want. It’s definitely not resource-friendly. To address this issue, we can use a Mutex.

As the name suggests, a Mutex provides mutual exclusion for a specific portion of your code, imposing restrictions on the access to that portion in situations where multiple threads or coroutines may attempt to access it concurrently. In essence, Mutex allows only one thread or coroutine to work within the confines of that portion of the code at any given time.

By introducing a Mutex in our refresh function, we can ensure that only one thread is executing the critical sections of the code, preventing race conditions and ensuring atomicity of the operations.

In the example above, whenever the refresh function is called, we check to see if the Mutex is already locked or been used. If not, we lock it and then perform our network call and save the results in our local database. This means that whenever another thread or coroutine calls this refresh function as it’s being used, the refresh is skipped, as confirmed by the test below:

There are some other cases where we do not want to skip but wait for the current lock to be released. Achieving this is very straightforward in Kotlin as the withLock lambda is a suspend function, meaning if the refresh function is called and the Mutex is already locked, the call suspends or waits for the Mutex to be unlocked:

class MutexEnabledComponent: Component {
    private val mutex = Mutex()

    override suspend fun refresh() {
        mutex.withLock {
            delay(1L) // Delay to simulate a long running operation
        }
    }
}

Using a Mutex has many advantages besides making your application run faster and use resources better. It helps keep your data safe and consistent by allowing only one thread or coroutine to access important sections at a time. This prevents any issues that can arise when multiple threads try to change the same data simultaneously.

However, it’s crucial to avoid overusing Mutex. When too much code is wrapped with Mutex locks, excessive serialization can occur, which can hinder parallelism and slow down your application. So, as engineers, it is important to identify the critical sections in our application that genuinely require exclusive access and limit the usage of Mutex locks to those specific sections.

Premise is constantly looking to hire top-tier engineering talent to help us improve our front-end, mobile offerings, data and cloud infrastructure. Come find out why we’re consistently named among the top places to work in Silicon Valley by visiting our Careers page.

Concurrent Programming in Kotlin: Ensuring Thread Safety with Mutex was originally published in Engineering at Premise on Medium, where people are continuing the conversation by highlighting and responding to this story.

Writing and deploying serverless applications has never been easier, especially on Google Cloud Platform (GCP). With a few clicks developers are able to deploy their application to the cloud and then trigger calls in a variety of ways from Cloud Schedulers and Pubsub Subscriptions to API Gateways.

Deploying your resources to GCP seems straightforward until you get a 403 error, permission denied. This error causes frustration as the root cause can be a number of issues, and often the solution requires knowing GCP roles and permissions in an in depth manner. The issue can be caused by the user not having the correct permissions to deploy their services or the services themselves not having the correct permissions to connect to each other.

Once you have identified the problem you will need to understand what roles have the required permissions, how to apply those roles, and where to apply them. Instead of application developers wasting their time researching GCP documentation, searching stack overflow, or reaching out to devops teams, IAM access should be as automated as much as possible.

Using the Goblet framework, we are now able to view exactly what permissions are needed to deploy our serverless applications, enable required GCP service API’s, create a new custom role with these permissions, create a service account with this role, and add the correct invoker bindings so that all serverless services can connect to each other.

Goblet is a python framework for writing serverless applications in GCP with the goal of making it as simple as possible to write and deploy REST applications. Goblet uses simple decorators, similar to flask, to create the necessary configurations and automatically deploy the required services and infrastructure.

For example, with code below we can deploy a simple cloudfunction that is triggered by a pubsub subscription and a cloud scheduler. Goblet will also take care of adding the invoker bindings so both the subscription and cloud scheduler will have the permissions needed to actually make the call to the cloudfunction.

import logging
from goblet import Goblet, goblet_entrypoint

# goblet setup
app = Goblet(function_name="goblet-iam-tutorial")
goblet_entrypoint(app)

# set debug level
app.log.setLevel(logging.DEBUG)

# Define resources 

# Pubsub Topic
app.pubsub_topic("test")

# Pubsub Subscription
@app.pubsub_subscription(topic="test", use_subscription=True)
def subscription(data):
    return "success"

# Cloud Scheduler
@app.schedule("1 * * * *")
def schedule():
    return "success"

In the rest of the tutorial we will go over how to write a simple Goblet application, how to enable the required GCP services, how to view the permissions needed to deploy the application, how to create a new custom role with these permissions, and finally how to create a deployment service account with this role. The code used for this tutorial can be found in github.

Prerequisites

• GCP Account
• Python environment (>3.8)
• Gcloud cli

Getting Started:

Before you can deploy an application, be sure you have GCP credentials configured. You should run gcloud auth login and sign in to the desired GCP project.

Next you will need to install goblet, which can be done by running the commandpip install goblet-gcp .

Writing your Application:

We will write a simple application that can be triggered by both a Pubsub topic and cron job. Our code will also create the Pubsub topic if it doesn’t already exist. See below for the complete application, which we will write to main.py .

import logging
from goblet import Goblet, goblet_entrypoint

# goblet setup
app = Goblet(function_name="goblet-iam-tutorial")
goblet_entrypoint(app)

# set debug level
app.log.setLevel(logging.DEBUG)

# Define resources 

# Pubsub Topic
app.pubsub_topic("test")

# Pubsub Subscription
@app.pubsub_subscription(topic="test", use_subscription=True)
def subscription(data):
    return "success"

# Cloud Scheduler
@app.schedule("1 * * * *")
def schedule():
    return "success"

Our requirements.txt will contain goblet-gcp .

goblet-gcp

And finally we will need a .goblet/config.json where we can specify our service accounts for our Pubsub topic and Cloud Scheduler.

{
    "pubsub": {
        "serviceAccountEmail": "sa_pubsub@goblet.iam.gserviceaccount.com"
    },
    "scheduler": {
        "serviceAccount": "sa_scheduler@goblet.iam.gserviceaccount.com"
    }
}

That is all that is needed for actually writing our application.

Enabling Services

Before we can deploy our application and the dependent resources, we need to ensure that the required GCP apis are enabled. This can be done by running goblet services check . This will check to see what service API’s are needed for the application code you have written.

If there are some services not enabled you can run goblet services enable and goblet will handle enabling the APIs. Note that it will take a few minutes for the API’s to be enabled.

Generating an IAM Policy, an IAM Role, and a Service Account

We can view the required permissions needed to deploy our application by using Goblet to generate a deployment IAM policy. This is done by running goblet services autogen_iam . This will create a autogen_iam_role.jsonfile in the .goblet directory.

{
    "roleId": "Goblet_Deployment_Role_goblet_iam_tutorial",
    "role": {
        "title": "Deployment role for goblet-iam-tutorial",
        "description": "Goblet generated role",
        "includedPermissions": [
            "cloudfunctions.functions.create",
            "cloudfunctions.functions.delete",
            "cloudfunctions.functions.get",
            "cloudfunctions.functions.getIamPolicy",
            "cloudfunctions.functions.list",
            "cloudfunctions.functions.setIamPolicy",
            "cloudfunctions.functions.sourceCodeSet",
            "cloudfunctions.functions.update",
            "cloudfunctions.operations.get",
            "cloudscheduler.jobs.create",
            "cloudscheduler.jobs.delete",
            "cloudscheduler.jobs.get",
            "cloudscheduler.jobs.list",
            "cloudscheduler.jobs.update",
            "pubsub.subscriptions.create",
            "pubsub.subscriptions.delete",
            "pubsub.subscriptions.get",
            "pubsub.subscriptions.list",
            "pubsub.subscriptions.update",
            "pubsub.topics.create",
            "pubsub.topics.delete",
            "pubsub.topics.get",
            "pubsub.topics.list",
            "pubsub.topics.update"
        ],
        "stage": "GA"
    }
}

This role contains all the required permissions needed to deploy our application. This is useful if you are using an external service to handle deployments, such as Github Actions, and want to create a service account that has the minimal permissions needed for deployment.

Goblet can also handle the creation of a custom role based on the IAM policy that was autogenerated, the creation of a deployment service account, and attaching the role to the service account. This is done by running goblet services create_service_account -p PROJECT .

You can now view the service account in the IAM console and verify that the custom deployment role is attached.

Deploying the Application

Now that we have enabled the required GCP service API’s and created a custom deployment role and custom deployment service account, we are ready to deploy our application. Now using our custom deployment service account, we can simply run goblet deploy -p PROJECT -l LOCATION .

Triggering the Application

You can verify everything is working correctly by submitting a test message in the Pubsub topic, which can be done via the CLI or the cloud console. You can also trigger the Cloud Scheduler manually if you do not want to wait for the cron schedule to trigger a scheduled event.

Cleaning up

You can easily delete all resources created by running goblet destroy -p PROJECT -l LOCATION . The only resources you will need to delete manually are the service account and custom role.

Conclusion

Using the Goblet framework, we are now able to view exactly what permissions are needed to deploy our serverless applications, enable required GCP service API’s, create a new custom role with these permissions, create a service account with this role, and add the correct invoker bindings so that all serverless services can connect to each other.

Check out the full Goblet documentation for more information on supported backends, infrastircture, and handlers. If you enjoyed this blog, then check out our other tutorials or explore additional content on our newly launched engineer blog!

Premise is constantly looking to hire top-tier engineering talent to help us improve our front-end, mobile offerings and cloud infrastructure. Come find out why we’re consistently named among the top places to work in Silicon Valley by visiting our Careers page.

Easily Manage IAM Policies for Serverless REST Applications in GCP with Goblet was originally published in Engineering at Premise on Medium, where people are continuing the conversation by highlighting and responding to this story.

Google Cloud Platform (GCP) is a powerful cloud computing platform that allows businesses to run their applications and workloads with ease. However, as the number of services and applications increases, it becomes challenging to keep track of the usage of each service and ensure they are cost optimized. To address this issue, we can deploy a Python-based Cloud Function that will monitor the low usage of GCP services and notify service owners weekly via Slack that their service can be safely downscaled to minimize costs.

This tutorial will walk you through setting up this entire flow in a few quick and easy steps, while allowing you to easily customize the service thresholds, Slack message ui, and the cron schedule. The code for this tutorial can be found on our gcp-tutorials GitHub repository.

Architecture

Get CPU/Memory Usage Metrics

We can use the Metrics Explorer UI to build a MQL query to retrieve the desired data from the monitoring metrics API. In this tutorial we will monitor Cloud Run services, but you can monitor other resources by using a different MQL query.

In our example we selected the CPU and Memory Cloud Run metrics:

We group by service name and location and select the alignment for the 99th percentage to get the max usage over the 1 week duration.

Then click on CODE EDITOR to generate a sample MQL query:

fetch cloud_run_revision
| metric 'run.googleapis.com/container/cpu/utilizations'
| group_by 1w,
    [value_utilizations_percentile: percentile(value.utilizations, 99)]
| every 1w
| group_by [resource.service_name, resource.location],
    [value_utilizations_percentile_max: max(value_utilizations_percentile)]

fetch cloud_run_revision
| metric 'run.googleapis.com/container/memory/utilizations'
| group_by 1w,
    [value_utilizations_percentile: percentile(value.utilizations, 99)]
| every 1w
| group_by [resource.service_name, resource.location],
    [value_utilizations_percentile_max: max(value_utilizations_percentile)]

Code

Once we have our MQL queries we will use the Metrics API to run the query within our Cloud Function. We leverage goblet to deploy the scheduled Cloud Function that will be running each week to do the check. Our code also uses the goblet-gcp-client library to easily create our metrics client. This client is used to fetch the requested CPU/Memory data points.

We do additional checks after requesting the data to see if the service is already at the minimum thresholds, since downscaling is not possible in that case. If the service can be downscaled our Cloud Function will send a Slack message notifying the owner of the service that the service can be downscaled to save costs.

Deploy it!

The sample code can be cloned from gcp-tutorials.

Rename .goblet/config.json.sample to .goblet/config.json and replace all values that have {} with your environment details. This includes a SLACK_CHANNEL_ID, SLACK_TOKEN, and SERVICE_ACCOUNT_EMAIL.

{
  "function_name": "{FUNCTION_NAME}",
  "cloudfunction": {
      "environmentVariables": {
          "CLOUDRUN_CPU_MIN_VALUE": "1000",
          "CLOUDRUN_MEMORY_MIN_VALUE": "512",
          "CLOUDRUN_CPU_MIN_THRESHOLD": "10%",
          "CLOUDRUN_MEMORY_MIN_THRESHOLD": "10%",
          "SLACK_CHANNEL_ID": "{SLACK_CHANNEL_ID}",
          "CRON_EXPRESSION": "0 10 * 1 *",
          "DEBUG": "true"
      },
      "secretEnvironmentVariables":[
          {
              "key": "SLACK_BOT_TOKEN",
              "secret": "{SLACK_TOKEN}",
              "version": "latest"
          }
      ],
      "serviceAccountEmail": "{SERVICE_ACCOUNT_EMAIL}",
      "timeout": "540s"
  }
}

Next you can run the following two commands to install the required dependencies and deploy the application:

pip install -r requirements.txt
goblet deploy --project=<PROJECT> --location=<LOCATION>

When the command finishes successfully you will have a Cloud Function in the chosen project and a Cloud Scheduler that will execute the function every week, which results in slack messages that post which Cloud Run services that can be downscaled because they were below the configured thresholds for the entire week.

The schedule can be adjusting by changing the CRON_EXPRESSION variable in config.json.

Conclusion

We have shown how to build and deploy a simple solution that can have a significant effect on your cloud cost savings. This is especially important as you scale up your cloud environment, as costs can quickly spiral if you are not careful.

If you found this tutorial helpful you should check out another tutorial on cloud cost spike alerting.

Tutorial: Low Usage Alerting On Slack for Google Cloud Platform (GCP) was originally published in Engineering at Premise on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

CloudTasks is a Google Cloud Platform (GCP) service that allows users to enqueue tasks in a queue. It is somewhat similar to PubSub, but with CloudTasks, the queue has features such as rateLimits and retryConfig. This can be very useful when having to enforce a rate limit over an external API or retrying tasks that have a known probability of failure.

If you want to use CloudTasks you will have to manually deploy cloud infrastructure, understand the CloudTask object, deal with client libraries and figure out multiple IAM role bindings. Or you can use Goblet for all the heavy lifting and in just a few minutes be able push a task and later process the task in a Python function of our choosing.

Goblet is a Python based cloud framework for building serverless microservices on Google Cloud Platform. It enables developers to quickly and reliably deploy cloud resources together with the code that will make use of those resources.

In this blog post we will review how Goblet deploys CloudTaskQueues and provides the user with an simple code interface to enqueue and handle CloudTasks⁶.

Using the short snipped of code below we can define a queue, create a client, push a task to the queue, and handle tasks from the queue!

See the rest of the tutorial for setting up your Goblet environment, setting permissions, and deploying you infrastructure and application in a few quick steps.

Infrastructure

For this tutorial we will use an example GCP project called goblet-cloudtask and deploy region specific resources to us-central1. If you would like to follow along the complete code can be found at gcp-tutorials. The account used to run this example must have enough IAM role binding to run all the gcloud commands described in the Set-Up section below.

Now, let’s take a look at the infrastructure Goblet will create. We won’t have to take care of any of these steps ourselves. All we need to do is write our code in main.pyand Goblet will take care of the rest when we run goblet deploy.

1. Deploy Queue: Goblet will register line 5³ in main.py as an instruction to create or update a CloudTaskQueue with id my-queue.
2. Create CloudBuild: Upload all files needed to build a Docker image that will be able to run main.py in a CloudRun Revision.
3. Store Image: the Docker image created in the previous step is stored in Container Registry.
4. Deploy CloudRun: Create a new CloudRun Revision pulling our previously built docker image from Container Registry (5).

The last diagram, Runtime Flow shows how the example will run after the infrastructure and code are deployed.

1. We make an HTTP request to the /enqueue path of the CloudRun Revision running our main.py code.
2. Because we decorated function enqueue of our main.py with @app.route("/enqueue", methods=["GET]) , Goblet will route the request to the enqueue function. In line 10 we use client.enqueue⁴ to push a new CloudTask (with a payload and a target) to the queue. As soon as the task in enqueued, client.enqueue returns.
3. The CloudTask now lives in the queue. Based on the queue’s configuration, it will independently decide when to make an HTTP request to the target with the payload.
4. The CloudTaskQueue decides to process the CloudTask. Notice how in line 10 and line 13 of our example the new task has target="my_target" and the app.cloudtasktarget decorator has name="my_target". This links the CloudTask with the function that will handle it when called from the CloudTaskQueue.⁵

When we test the deployment, we will be able to see these requests in the service’s log.

Set-Up

For Goblet to be able to apply all the actions described in the previous section, we will need to enable APIs, create service accounts with the required IAM policy bindings and create a container repository for our Docker image.

APIs

gcloud services enable run.googleapis.com
gcloud services enable cloudfunctions.googleapis.com
gcloud services enable cloudbuild.googleapis.com
gcloud services enable iam.googleapis.com
gcloud services enable artifactregistry.googleapis.com
gcloud services enable cloudresourcemanager.googleapis.com
gcloud services enable apigateway.googleapis.com
gcloud services enable cloudtasks.googleapis.com

IAM Policy Bindings

The Goblet client goblet uses GCP’s REST API to achieve the infrastructure state described in the Goblet Deploy diagram. This is the list of roles at the project level needed by the user running the goblet command-line client.

roles/apigateway.admin
roles/cloudbuild.serviceAgent
roles/cloudfunctions.developer
roles/run.developer
roles/run.serviceAgent
roles/cloudtasks.queueAdmin

In this example, we will use a service account instead of a user account to run the goblet command. By the time we are ready to deploy — when we need to run goblet deploy— we will impersonate the service account so API calls are authenticated as coming from the service account.²

Service Accounts

The first service account we need is the deployer. This is the service account described in the previous paragraph. It is the account that will run steps 1 through 5 in the Goblet Deploy diagram.

gcloud iam service-accounts create deployer --display-name="deployer"
     
gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:deployer@goblet-cloudtask.iam.gserviceaccount.com" \
  --role="roles/apigateway.admin"

gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:deployer@goblet-cloudtask.iam.gserviceaccount.com" \
  --role="roles/cloudbuild.serviceAgent"

gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:deployer@goblet-cloudtask.iam.gserviceaccount.com" \
  --role="roles/run.serviceAgent"

gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:deployer@goblet-cloudtask.iam.gserviceaccount.com" \
  --role="roles/cloudfunctions.developer"

gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:deployer@goblet-cloudtask.iam.gserviceaccount.com" \
  --role="roles/run.developer"

gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:deployer@goblet-cloudtask.iam.gserviceaccount.com" \
  --role="roles/cloudtasks.queueAdmin"

Next we need the cloudtask service account. When the CloudTask runs, it will use this service account to authenticate itself against CloudRun¹. Goblet implements an easy way to create CloudTasks that will be handled by a function in the same CloudRun service. The cloudtask service account needs roles/run.invoker to make requests to CloudRun endpoints.

gcloud iam service-accounts create cloudtask --display-name="cloudtask"

gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:cloudtask@goblet-cloudtask.iam.gserviceaccount.com" \
  --role="roles/run.invoker"

The last account we need is the cloudrun service account. This is the account that runs our CloudRun service and calls the CloudTaskQueue API to enqueue tasks. Therefore it needs the roles/cloudtasks.enqueuer and roles/run.viewer roles. It also needs roles/iam.serviceAccountUser bound to service account cloudtask to be able to create CloudTasks that will authenticate with the cloudtask service account.

gcloud iam service-accounts create cloudrun --display-name="cloudrun"

gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:cloudrun@goblet-cloudtask.iam.gserviceaccount.com" \
  --role="roles/cloudtasks.enqueuer"

gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:cloudrun@goblet-cloudtask.iam.gserviceaccount.com" \
  --role="roles/run.viewer"

gcloud iam service-accounts add-iam-policy-binding \
  cloudtask@goblet-cloudtask.iam.gserviceaccount.com \
  --member="serviceAccount:cloudrun@goblet-cloudtask.iam.gserviceaccount.com" \
  --role="roles/iam.serviceAccountUser"

Container Registry

gcloud artifacts repositories create cloud-run-source-deploy \
   --location=us-central1 --repository-format=docker

Deploy

First we will pull our example code.

$ git clone https://github.com/premisedata/gcp-tutorial
$ cd gcp-tutoriales/015-goblet-cloudtask
$ tree
.
├── .goblet
│   └── config.json.sample
├── Dockerfile
├── README.md
├── main.py
└── requirements.txt

To deploy our Goblet app, we need to create a config.json file that links our service accounts with the services that they will run. Our code example already has a template configuration file in .goblet/config.json.sample. We just need to replace the {PROJECT} boilerplate variable.

$ pwd
./gcp-tutorials/015-goblet-cloudtask

$ export GOBLET_PROJECT="goblet-cloudtask"
$ export GOBLET_LOCATION="us-central1"
$ sed "s/{PROJECT}/$GOBLET_PROJECT/g" .goblet/config.json.sample > .goblet/config.json

$ cat .goblet/config.json
{
  "cloudrun_revision": {
    "serviceAccount": "cloudrun@goblet-cloudtask.iam.gserviceaccount.com"
  },
  "cloudtask": {
    "serviceAccount": "cloudtask@goblet-cloudtask.iam.gserviceaccount.com"
  },
  "cloudtaskqueue": {
    "my-queue": {
      "rateLimits": {
        "maxDispatchesPerSecond": 500,
        "maxBurstSize": 100,
        "maxConcurrentDispatches": 1000
      },
      "retryConfig": {
        "maxAttempts": 10,
        "minBackoff": "0.100s",
        "maxBackoff": "3600s",
        "maxDoublings": 16
      }
    }
  }
}

Almost all of Goblet’s features can be configured in the config.json file. Here we see how the cloudrun service account is set to run the CloudRun Revision. We have also set the cloudtask service account as the service account the CloudTask will use to authenticate itself.

We are almost ready to deploy. As mentioned at the beginning of the post, we will deploy impersonating the deployer service account. We will need an additional IAM policy binding to be able to do that before authenticating.

export MY_GCP_ACCOUNT=`gcloud auth list --filter=status:ACTIVE --format="value(account)"`

# IAM binding to impersonate deployer service account
gcloud iam service-accounts add-iam-policy-binding \
  deployer@goblet-cloudtask.iam.gserviceaccount.com \
  --member="user:$MY_GCP_ACCOUNT" \
  --role="roles/iam.serviceAccountTokenCreator"

# authenticate impersonating deployer service account
gcloud auth application-default login \
  --impersonate-service-account=deployer@goblet-cloudtask.iam.gserviceaccount.com

We can now install Goblet and deploy.

$ pwd
./gcp-tutorials/015-goblet-cloudtask

$ export GOBLET_PROJECT="goblet-cloudtask"
$ export GOBLET_LOCATION="us-central1"

$ python3 -m venv venv
$ . venv/bin/activate
(venv) $ pip install -r requirements.txt
(venv) $ goblet deploy -l $GOBLET_LOCATION -p $GOBLET_PROJECT

INFO:goblet.app:deploying infrastructure
INFO:goblet.decorators:deploying cloudtaskqueue
INFO:goblet.deployer:CloudTask Queue [my-queue] deployed
INFO:goblet.decorators:deploying apigateway
INFO:goblet.app:preparing to deploy with backend cloudrun
INFO:goblet.backend:zipping source code
INFO:goblet.backend:uploading source zip to gs......
INFO:goblet.backend:source code uploaded
INFO:goblet.deployer:creating cloudbuild
INFO:goblet.deployer:creating cloudrun
INFO:goblet.decorators:deploying cloudtasktarget
INFO:goblet.decorators:deploying route
INFO:goblet.deployer:deploying api......
INFO:goblet.deployer:api successfully deployed...
INFO:goblet.deployer:api endpoint is cloudtask-example-xxxxxxxx.uc.gateway.dev

Test

To test that everything is working, we will make a request to the /enqueue endpoint in our deployed URL as described in the Runtime Flow diagram. Since we are impersonating the deployer service account and the request has to be authenticated, we will bind roles/run.invoker to deployer so we can use it to make requests to the CloudRun endpoint.

gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:deployer@goblet-cloudtask.iam.gserviceaccount.com" \
  --role="roles/run.invoker"

CLOUDRUN_URL=`gcloud run services list --filter=SERVICE:cloudtask-example --format="value(URL)"`
curl $CLOUDRUN_URL/enqueue \
  -H "Authorization: Bearer $(gcloud auth print-identity-token)"  

We can see from the logs that the CloudRun gets deployed by the deployer service account. Then we see the curl GET request to /enqueue and at last the POST request from the CloudTaskQueue and the log {"title": "enqueue"} from the response when then CloudTask is processed.

Conclusion

Even though GCP has a huge number of useful services and features, orchestrating them together can be a time consuming and error prone task. Goblet solves this problem by reliably orchestrating and deploying the infrastructure and code needed for you application to work.

TL;DR

$ export GOBLET_PROJECT="goblet-cloudtask"
$ export GOBLET_LOCATION="us-central1"
$ export MY_GCP_ACCOUNT=`gcloud auth list --filter=status:ACTIVE --format="value(account)"`
##### set the project
$ gcloud config set project $GOBLET_PROJECT
##### enable APIs
$ gcloud services enable run.googleapis.com
$ gcloud services enable cloudfunctions.googleapis.com
$ gcloud services enable cloudbuild.googleapis.com
$ gcloud services enable iam.googleapis.com 
$ gcloud services enable artifactregistry.googleapis.com
$ gcloud services enable cloudresourcemanager.googleapis.com
$ gcloud services enable apigateway.googleapis.com
$ gcloud services enable cloudtasks.googleapis.com
##### service account to run goblet deploy
$ gcloud iam service-accounts create deployer --display-name="deployer"     
$ gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:deployer@${GOBLET_PROJECT}.iam.gserviceaccount.com" \
  --role="roles/apigateway.admin"
$ gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:deployer@${GOBLET_PROJECT}.iam.gserviceaccount.com" \
  --role="roles/cloudbuild.serviceAgent"
$ gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:deployer@${GOBLET_PROJECT}.iam.gserviceaccount.com" \
  --role="roles/run.serviceAgent"
$ gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:deployer@${GOBLET_PROJECT}.iam.gserviceaccount.com" \
  --role="roles/cloudfunctions.developer"
$ gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:deployer@${GOBLET_PROJECT}.iam.gserviceaccount.com" \
  --role="roles/run.developer"
$ gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:deployer@${GOBLET_PROJECT}.iam.gserviceaccount.com" \
  --role="roles/cloudtasks.queueAdmin"
##### service account to authenticate the CloudTask against CloudRun
$ gcloud iam service-accounts create cloudtask --display-name="cloudtask"
$ gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:cloudtask@${GOBLET_PROJECT}.iam.gserviceaccount.com" \
  --role="roles/run.invoker"
##### service account to run the CloudRun Revision
gcloud iam service-accounts create cloudrun --display-name="cloudrun"
gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:cloudrun@${GOBLET_PROJECT}.iam.gserviceaccount.com" \
  --role="roles/cloudtasks.enqueuer"
gcloud projects add-iam-policy-binding goblet-cloudtask \
  --member="serviceAccount:cloudrun@${GOBLET_PROJECT}.iam.gserviceaccount.com" \
  --role="roles/run.viewer"
gcloud iam service-accounts add-iam-policy-binding \
  cloudtask@${GOBLET_PROJECT}.iam.gserviceaccount.com \
  --member="serviceAccount:cloudrun@${GOBLET_PROJECT}.iam.gserviceaccount.com" \
  --role="roles/iam.serviceAccountUser"
##### impersonate deployer service account
$ gcloud iam service-accounts add-iam-policy-binding \
  deployer@${GOBLET_PROJECT}.iam.gserviceaccount.com \
  --member="user:${MY_GCP_ACCOUNT}" \
  --role="roles/iam.serviceAccountTokenCreator"
$ gcloud auth application-default login \
  --impersonate-service-account=deployer@${GOBLET_PROJECT}.iam.gserviceaccount.com
##### fetch example and deploy
$ git clone https://github.com/premisedata/gcp-tutorial
$ cd gcp-tutoriales/015-goblet-cloudtask
$ python3 -m venv venv
$ . venv/bin/activate
$ sed "s/{PROJECT}/$GOBLET_PROJECT/g" .goblet/config.json.sample > .goblet/config.json
(venv) $ pip install -r requirements.txt
(venv) $ goblet deploy -l $GOBLET_LOCATION -p $GOBLET_PROJECT
# test
$ SERVICE_URL=`gcloud run services list \
  --filter=SERVICE:cloudtask-example --format="value(URL)"
$ curl $SERVICE_URL/enqueue \
  -H "Authorization: Bearer $(gcloud auth print-identity-token)" 

[1] The httpRequest of the CloudTask expects a service account for authentication. https://github.com/goblet/goblet/blob/608512c7fccec6cb29ce1b6d9353c2b6e4977d3b/goblet/infrastructures/cloudtask.py#L30

[2] This makes it much easier later when integrating with CI/CD.

[3] During goblet deploy line 5 creates or updates a queue. During runtime Goblet returns an object that is wrapper for CloudTasks REST API.

[4] Since the client is a Goblet wrapper for the CloudTask REST API, the CloudTask target is set to the URL of the CloudRun Revision. The client also added headers that will let Goblet route the request to the the function decorated with a matching cloudtasktarget.

[5] The client adds an HTTP Header to the CloudTask with the name of the target that handles the task. In this example it would be X-Goblet-CloudTask-Target: my_target. When the CloudTaskQueue makes a request to the target it does so with User-Agent: Google-Cloud-Tasks. The combination of these two headers allows Goblet to route the CloudTask to the defined target/function.

[6] CloudTasks Documentation

Deploy and Handle GCP CloudTasks with Goblet in minutes. was originally published in Engineering at Premise on Medium, where people are continuing the conversation by highlighting and responding to this story.

Table Of Contents

Introduction

Frontend Plugin (dynamic-pick-extension)

Backend Plugin (form-data-backend)

Introduction

If you’ve ever built a Backstage template, you know that creating user-friendly forms is a must. One of the most common form elements is the enumcomponent. It allows users to choose from a list of options, which is especially useful when you have a large number of choices. But what happens when you have to dynamically populate that list with data from an external API? That's where the dynamic-data-form-extension plugin collection comes in.

The dynamic-data-form-extension plugin collection allows you to populate a <Select> component with data fetched from an HTTP endpoint. It's a simple, yet powerful plugin that can save you a lot of time and effort.

Frontend Plugin (dynamic-pick-extension)

This plugin is a Custom Field Extension that allow you to create <Select> components that fetches data dynamically from an endpoint. This can be used together with the form-data-backend plugin to write custom logic to fill the field.

Installation

yarn add --cwd packages/app @premise/plugin-dynamic-pick-extension

Configuration

Add the import to your App.tsx on the frontend package of your backstage instance:

import { DynamicPickFieldExtension } from '@premise/plugin-dynamic-pick-extension';

Then add the imported field extension as a child of ScaffolderFieldExtensions.

<ScaffolderFieldExtensions>
  <DynamicPickFieldExtension />
</ScaffolderFieldExtensions>

Usage

To use the extension on a Backstage Template Action just add the ui-field and ui-options fields to the parameter.

Basic Usage:

parameters:
  - category:
      title: Category
      type: string
      ui:field: DynamicPickExtension
      ui:options:
        # IMPORTANT: The endpoint needs to return a JSON array of strings.
        external_data: https://dummyjson.com/products/categories

Using the form-data-backend plugin:

parameters:
  - team:
      title: Github Team to add as admin of the repository
      type: string
      ui:field: DynamicPickExtension
      ui:options:
        # This is a provider added on the form-data-backend plugin
        form_data: github/teams

Backend Plugin (form-data-backend)

This plugin is a Backstage Backend Plugin that allows you to write custom providers to add custom logic and build the data to be shown on the dynamic-field-extension for example for APIs that require authentication or to parse data that doesn’t come on the required JSON Array String format.

Installation

yarn add --cwd packages/backend @premise/plugin-form-data-backend

Configuration

Create a form-data.ts file under packages/backend/src/plugins with the following content:

import { createRouter } from '@premise/plugin-form-data-backend';
import { Router } from 'express';
import { PluginEnvironment } from '../types';
import { exampleRouter } from '../providers';
export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  return await createRouter(
    {
      logger: env.logger,
    },
    [
      {
        path: '/example',
        router: exampleRouter,
      }
    ],
  );
}

Create a providers folder under packages/backend/src containing all of your custom providers for example:

example.ts

import express from 'express';
import Router from 'express-promise-router';
import { RouterOptions } from '@premise/plugin-form-data-backend';

export async function exampleRouter(
  options: RouterOptions,
): Promise<express.Router> {
  const { logger } = options;
  const router = Router();

  // You can add in here all of the endpoints you want
  router.get('/ping', async (_, response) => {
    logger.info('Pong');
    response.json(['pong']);
  });

  return router;
}

Then under packages/backend/src/index.ts import the plugin:

import formData from './plugins/form-data';
// …
const formDataEnv = useHotMemoize(module, () => createEnv('form-data'));
// …
apiRouter.use('/form-data', await formData(formDataEnv));

Usage with dynamic-pick-extension

parameters:
  - example:
      title: Example
      type: string
      ui:field: DynamicPickExtension
      ui:options:
        # This is a provider added on the form-data-backend plugin
        form_data: example/ping

Dynamic Data On Backstage Templates was originally published in Engineering at Premise on Medium, where people are continuing the conversation by highlighting and responding to this story.

There are a number of quality Postgres open source administration tools such as pgAdmin or DBeaver, but these tools require a persistent server to run. This means that each user needs to install the tool locally and setup their connections, or you need to host the tool in a cloud server, which can get expensive.

A much cheaper alternative is to use Pgweb, a lightweight web-based database explorer for PostgreSQL written in Go, and deploy to Cloud Run. This allows you to utilize a web-based tool, for minimal cost, that can scale for any number of users.

We will leverage several Google Cloud Platform services for the complete setup starting with a http load balancer to allow for a custom DNS name as well as enable Identity Aware Proxy (IAP), which is what we will be using for authentication. We will host the service in Cloud Run and connect directly to CloudSQL using a CloudSQL connector. You can bypass the load balancer and IAP if you would like to use username and passwords for authentication.

Cloud Setup

The first step is setting up your CloudSQL instance. This can be done via terraform or in the GCP console. The main requirement is that you enable a public ip address, which will allow us to use the CloudSQL Auth proxy to connect our Cloud Run instance to our CloudSQL instance. If this is not possible you can still connect to your CloudSQL instance to Cloud Run through a VPC Connector instead.

Next we will need to save our database connection credentials in GCP’s Secret Manger by creating a new secret called PGWEB_DATABASE_URL in the format of postgres:///DB_NAME?host=/cloudsql/PROJECT:REGION:INSTANCE_NAME&user=DB_USER&password=DB_PASSWORD .

Now that we have our CloudSQL instance and connection secret we will deploy our Cloud Run instance using the gcloud cli. First we will need to push the desired Pgweb container to GCP’s Artifact Registry, which we can do using the following commands.

gcloud auth configure-docker us-central1-docker.pkg.dev

export PROJECT=PROJECT

docker pull sosedoff/pgweb:0.14.0
docker tag sosedoff/pgweb:0.14.0  us-central1-docker.pkg.dev/$PROJECT/cloud-run-source-deploy/pgweb:0.14.0
docker push us-central1-docker.pkg.dev/$PROJECT/cloud-run-source-deploy/pgweb:0.14.0

Next you can run the following gcloud command to deploy your Cloud Run instance. You will need to replace the following variables (PROJECT, SERVICE_ACCOUNT, INSTANCE_NAME).

export PROJECT=PROJECT
export SERVICE_ACCOUNT=SERVICE_ACCOUNT
export INSTANCE_NAME=INSTANCE_NAME

gcloud alpha run deploy pgweb --set-secrets PGWEB_DATABASE_URL=PGWEB_DATABASE_URL:latest --no-allow-unauthenticated --project=$PROJECT --region=us-central1 --platform managed --service-account=$SEVICE_ACCOUNT --min-instances=1 --max-instances=1 --port=8081 --image=us-central1-docker.pkg.dev/$PROJECT/cloud-run-source-deploy/pgweb:0.14.0 --set-cloudsql-instances=$PROJECT:us-central1:$INSTANCE_NAME

Note that your service account will need CloudSQL user permissions for your instance. The min instance is set to 1 to make sure that an instance is always running.

The next step is deploying the HTTP Load Balancer. This can be deploying in the console, by selecting a global load balancer and creating a route to your Cloud Run instance.

Once this is created you will need to go to the IAP page in the GCP console and enable IAP on your http load balancer. If this is your first time using IAP you will need to follow the IAP setup steps, which requires adding a firewall rule to your project.

In order to grant users permission to Pgweb, you will need to grant specific users IAP-Secured Web App User permission in IAP.

An additional step to make it easier on users to access is to setup a DNS record to point to your load balancer. For example you can create a pgweb.DOMAIN.com DNS record to point to the ip address of your load balancer.

Conclusion

We now have a way to deploy a lightweight web-based database explorer for PostgreSQL hosted on Cloud Run. This is great for rapid development and keeping costs low.

If you enjoyed this blog, then check out our other tutorials or explore additional content on our newly launched engineer blog!

Premise is constantly looking to hire top-tier engineering talent to help us improve our front-end, mobile offerings and cloud infrastructure. Come find out why we’re consistently named among the top places to work in Silicon Valley by visiting our Careers page.

Hosting a fully Serverless Web-Based Postgres Admin Client on GCP using Pgweb, Cloud Run, & IAP was originally published in Engineering at Premise on Medium, where people are continuing the conversation by highlighting and responding to this story.

Google Cloud Platform’s (GCP) Private Services Access allows you to connect to GCP services without an external IP being assigned. Redis is an in-memory DB that is used for caching and is a service that supports private access. In order to take advantage of this feature, a private connection is necessary. For serverless environments, this can be done using a VPC Connector.

VPC Connectors allow serverless environments to connect to VPC Networks by handling traffic between the network and environment, restricting requests to use only internal IP addresses. This can be beneficial for applications that do not require exposure to the public internet or if latency for requests across services needs to be improved.

Currently you can provision a connector using the console, cli, or terraform but you will still need to configure your serverless environment to utilize this connector. Instead of manually going through the segmented process of creating a connector and updating your environment, you can leverage Goblet to provision your connector and inject needed configuration values into your deployment.

This tutorial will walk you through the steps of deploying a Cloud Run service that privately connects to a Redis Instance using a VPC Connector.

Prerequisites:

• GCP Account
• Python Environment (≥ 3.7)
• Gcloud CLI

You will also need to have goblet and redis installed which can be done by running pip install goblet-gcp && pip install redis

Getting Started:

Ensure you have credentials configured by running gcloud auth login and sign in to the desired project. Make sure to have the correct services enabled in your GCP project which include:

• Cloud Run
• Cloud Build
• Artifact Registry
• Serverless VPC Access
• Memorystore for Redis

To get started, clone the code from the gcp-tutorials repository and navigate to the directory. This can be done with the Github CLI by running:

gh repo clone premisedata/gcp-tutorials/014-goblet-private-services

Configuring the VPC Connector and Redis Instance:

Adding a VPC Connector and Redis Instance to your Goblet deploy is as simple as:

app.vpcconnector("goblet-vpcconnector")
app.redis("goblet-redis")

For Redis configuration, update the PROJECT in the authorizedNetwork field within the redis key of theconfig.json file in the .goblet folder.

The default range for the VPC Connector is 10.32.1.0/28 if this range is already taken by another VPC connector, update the ipCidrRange field within the vpcconnector key of the config.json file in the .goblet folder

Deploying the VPC Connector and Redis Instance:

Deploying your infrastructure and backend can be done with:

goblet deploy -p PROJECT -l LOCATION

To deploy just the VPC connector and Redis Instance add the --skip-backend flag

After a few minutes you will have the following:

• Redis Instance with private service access connection mode enabled

• VPC Connector for your default network

• Cloud Run instance that is connected to your default VPC Network through the connector with environment variables set for your Redis HOST, PORT, and INSTANCE_NAME.

If you want to update your local config file with your recent infrastructure updates you can easily write out the config using --write-config flag.

Testing:

Make sure you have Cloud Run Invoker permissions for the Cloud Run service. We will call the /redis endpoint, this will connect to our Redis instance and increment a counter every time the endpoint is hit and return the value.

The URL for your service can be found on the Cloud Run details page.

curl $URL/redis  -H "Authorization: Bearer $(gcloud auth print-identity-token)"

On first call you will see the count is set to 1, on every subsequent call this should increase for the lifespan of the Redis Instance:

> curl $URL/redis  -H "Authorization: Bearer $(gcloud auth print-identity-token)"
{"Visitor number":1}
> curl $URL/redis  -H "Authorization: Bearer $(gcloud auth print-identity-token)"
{"Visitor number":2}

Cleaning up the VPC Connector:

To cleanup all deployed resources, including the VPC Connector and Redis instance you can run the goblet destroy command:

goblet destroy -p PROJECT -l LOCATION

Conclusion:

We now have an extremely simple way to deploy infrastructure for your serverless environment. By leveraging VPC Connectors you can easily connect to a Redis instance using private access, improving the security of your application. Check out the full Goblet documentation for more information on supported backends, infrastructures, and resources. If you enjoyed this blog, then check out our other tutorials or explore additional content on our newly launched engineer blog!

Premise is constantly looking to hire top-tier engineering talent to help us improve our front-end, mobile offerings and cloud infrastructure. Come find out why we’re consistently named among the top places to work in Silicon Valley by visiting our Careers page.

Tutorial: Connecting Cloudrun and Cloudfunctions to Redis and other Private Services using Goblet was originally published in Engineering at Premise on Medium, where people are continuing the conversation by highlighting and responding to this story.