Docker Hub is a popular registry for hosting public container images. Earlier this summer, Docker announced it will begin rate-limiting the number of pull requests to the service by “Free Plan” users. For pull requests by anonymous users this limit is now 100 pull requests per 6 hours; authenticated users have a limit of 200 pull requests per 6 hours. When the new rate limits take effect on November 1st, they might disrupt your automated build and deployment processes on Cloud Build or how you deploy artifacts to Google Kubernetes Engine (GKE), Cloud Run or App Engine Flex from Docker Hub. 

This situation is made more challenging because, in many cases, you may not be aware that a Google Cloud service you are using is pulling images from Docker Hub. For example, if your Dockerfile has a statement like “FROM debian:latest”or your Kubernetes Deployment manifest has a statement like “image: postgres:latest” it is pulling the image directly from Docker Hub. To help you identify these cases, Google Cloud has prepared a guide with instructions on how to scan your codebase and workloads for container image dependencies from third-party container registries, like Docker Hub.

We are committed to helping you run highly reliable workloads and automation processes. In the rest of the blog post, we’ll discuss how these new Docker Hub pull rate limits may affect your deployments running on various Google Cloud services, and strategies for mitigating against any potential impact. Be sure to check back often, as we will update this post regularly. 

Impact on Kubernetes and GKE

One of the groups that may see the most impact from these Docker Hub changes is users of managed container services. Like it does for other managed Kubernetes platforms, Docker Hub treats GKE as an anonymous user by default. This means that unless you are specifying Docker Hub credentials in your configuration, your cluster is subject to the new throttling of 100 image pulls per six hours, per IP. And many Kubernetes deployments on GKE use public images. In fact, any container name that doesn’t have a container registry prefix such as gcr.io is pulled from Docker Hub. Examples include nginx, and redis.

Container Registry hosts a cache of the most requested Docker Hub images from Google Cloud, and GKE is configured to use this cache by default. This means that the majority of image pulls by GKE workloads should not be affected by Docker Hub’s new rate limits. Furthermore, to remove any chance that your images would not be in the cache in the future, we recommend that you migrate your dependencies into Container Registry, so that you can pull all your images from a registry under your control.

In the interim, to verify whether or not you are affected, you can generate a list of DockerHub images your cluster consumes:

You may want to know if the images you use are in the cache. The cache will change frequently but you can check for current images via a simple command:

It is impractical to predict cache hit-rates, especially in times where usage will likely change dramatically. However, we are increasing cache retention times to ensure that most images that are in the cache stay in the cache.

GKE nodes also have their own local disk cache, so when reviewing your usage of DockerHub, you only need to count the number of unique image pulls (of images not in our cache) made from GKE nodes: 

• For private clusters, consider the total number of such image pulls across your cluster (as all image pulls will be routed via a single NAT gateway). 

• For public clusters you have a bit of extra breathing room, as you only need to consider the number of unique image pulls on a per-node basis. For public nodes, you would need to churn through more than 100 unique public uncached images every 6 hours to be impacted, which is fairly uncommon.

If you determine that your cluster may be  impacted, you can authenticate to DockerHub by adding imagePullSecrets with your Docker Hub credentials to every Pod that references a container image on Docker Hub.

While GKE is one of the Google Cloud services that may see an impact from the Docker Hub rate limits, any service that relies on container images may be affected, including similar Cloud Build, Cloud Run, App Engine, etc.

Finding the right path forward 

Upgrade to a paid Docker Hub account

Arguably, the simplest—but most expensive—solution to Docker Hub’s new rate limits is to upgrade to a paid Docker Hub account. If you choose to do that and you use Cloud Build, Cloud Run on Anthos, or GKE, you can configure the runtime to pull with your credentials. Below are instructions for how to configure each of these services:

• Cloud Build: Interacting with Docker Hub images

• Cloud Run on Anthos: Deploying private container images from other container registries

• Google Kubernetes Engine: Pull an Image from a Private Registry

Switch to Container Registry

Another way to avoid this issue is to move any container artifacts you use from Docker Hub to Container Registry. Container Registry stores images as Google Cloud Storage objects, allowing you to incorporate container image management as part of your overall Google Cloud environment. More to the point, opting for a private image repository for your organization puts you in control of your software delivery destiny. 

To help you migrate, the above-mentioned guide also provides instructions on how to copy your container image dependencies from Docker Hub and other third-party container image registries to Container Registry. Please note that these instructions are not exhaustive—you will have to adjust them based on the structure of your codebase.

Additionally, you can use Managed Base Images, which are automatically patched by Google for security vulnerabilities, using the most recent patches available from the project upstream (for example, GitHub). These images are available in the GCP Marketplace.

Here to help you weather the change

The new rate limits on Docker Hub pull requests will have a swift and significant impact on how organizations build and deploy container-based applications. In partnership with the Open Container Initiative (OCI), a community devoted to open industry standards around container formats and runtimes, we are committed to ensuring that you weather this change as painlessly as possible.

2020 has brought with it some tremendous innovations in the area of cloud security. As cloud deployments and technologies have become an even more central part of organizations’ security program, we hope you’ll join us for the latest installment of our Google Cloud Security Talks, a live online event on November 18th, where we’ll help you navigate the latest thinking in cloud security.

We’ll share expert insights into our security ecosystem and cover the following topics

• Sunil Potti and Rob Sadowski will open the digital event with our latest Google Cloud security announcements.

• This will be followed by a panel discussion with Dave Hannigan and Jeanette Manfra from Google Cloud’s Office of the CISO on how cloud migration is a unique opportunity to dismantle the legacy security debt of the past two decades.

• Kelly Waldher and Karthik Lakshminarayan will talk about the new Google Workspace and how it can enable users to access data safely and securely while preserving individual trust and privacy.

• We will present our vision of network security in the cloud with Shailesh Shukla and Peter Blum, where we’ll talk about the recent innovations that are making network security in the cloud powerful but invisible, protecting infrastructure and users from cyber attacks. 

• Sam Lugani and Ibrahim Damlaj will do a deeper dive on Confidential Computing, and more specifically Confidential GKE Nodes and how they can add another layer of protection for containerized workloads.

• You will also learn how Security Command Center can help you identify misconfigurations in your virtual machines, containers, network, storage, and identity and access management policies as well vulnerabilities in your web applications, with Kathryn Shih and Timothy Peacock.

• Anton Chuvakin and Seth Vargo will talk about the differences between key management and secret management to help you choose the best security controls for your use cases.

• Finally, we will host the Google Cloud Security Showcase, a special segment where we’ll focus on a few security problems and show how we’ve recently helped customers solve them using the tools and products that Google Cloud provides. 

We look forward to sharing our latest security insights and solutions with you. Sign-up now to reserve your virtual seat.

In recent years, stateless middle-tiers have been touted as a simple way to achieve horizontal scalability. But the rise of microservices has pushed the limits of the stateless architectural pattern, causing developers to look for alternatives.

Stateless middle-tiers have been a preferred architectural pattern because they helped with horizontal scaling by alleviating the need for server affinity (aka sticky sessions). Server affinity made it easy to hold data in the middle-tier for low-latency access and easy cache invalidation. The stateless model pushed all “state” out of the middle-tier into backing data stores. In reality the stateless pattern just moved complexity and bottlenecks to that backing data tier. The growth of microservice architectures exacerbated the problem by putting more pressure on the middle tier, since technically, microservices should only talk to other services and not share data tiers. All manners of bailing wire and duct tape have been employed to overcome the challenges introduced by these patterns. New patterns are now emerging which fundamentally change how we compose a system from many services running on many machines.

To take an example, imagine you have a fraud detection system. Traditionally the transactions would be stored in a gigantic database and the only way to perform some analysis on the data would be to periodically query the database, pull the necessary records into an application, and perform the analysis. But these systems do not partition or scale easily. Also, they lack the ability for real-time analysis. So architectures shifted to more of an event-driven approach where transactions were put onto a bus where a scalable fleet of event-consuming nodes could pull them off. This approach makes partitioning easier, but it still relies on gigantic databases that received a lot of queries. Thus, event-driven architectures often ran into challenges with multiple systems consuming the same events but at different rates.

Another (we think better) approach, is to build an event-driven system that co-locates partitioned data in the application tier, while backing the event log in a durable external store. To take our fraud detection example, this means a consumer can receive transactions for a given customer, keep those transactions in memory for as long as needed, and perform real-time analysis without having to perform an external query. Each consumer instance receives a subset of commands (i.e., add a transaction) and maintains its own “query” / projection of the accumulated state. For instance:

By separating commands and queries we can easily achieve end-to-end horizontal scaling, fault tolerance, and microservice decoupling. And with the data being partitioned in the application tier we can easily scale that tier up and down based on the number of events or size of data, achieving serverless operations. 

Making it work with Cloudstate

This architecture is not entirely uncommon, going by the names Event Sourcing, Command Query Response Segregation (CQRS), and Conflict-Free Replicated Data Types. (Note: for a great overview of this see a presentation titled “Cloudstate – Towards Stateful Serverless” by Jonas Bonér.) But until now, it’s been pretty cumbersome to build systems with these architectures due to primitive programming and operational models. The new Cloudstate open-source project attempts to change that by building more approachable programming and operational models.

Cloudstate’s programming model is built on top of protocol buffers (protobufs) which enable evolvable data schemas and generated service interaction stubs. When it comes to data schemas, protobufs allow you to add fields to event / message objects without breaking systems that are still using older versions of those objects. Likewise, with the gRPC project, protobufs can be automatically wrapped with client and server “stubs” so that no code needs to be written for handling protobuf-based network communication. For example, in the fraud detection system, the protobuf might be:

The `Transaction` message contains the details about a transaction and the `user_id` field enables automatic sharding of data based on the user.

Cloudstate adds support for event sourcing on top of this foundation so developers can focus on just the commands and accumulated state that a given component needs. For our fraud detection example, we can simply define a class / entity to hold the distributed state and handle each new transaction. You can use any language, but we use Kotlin, a Google-sponsored language that extends Java.

With the exception of a tiny bit of bootstrapping code, that’s all you need to build an event-sourced system with Cloudstate!

The operational model is also just as delightful since it is built on Kubernetes and Knative. First you need to containerize the service. For JVM-based builds (Maven, Gradle, etc.) you can do this with Jib. In our example we use Gradle and can just run:

This creates a container image for the service and stores it on the Google Container Registry. To run the Cloudstate service on your own Kubernetes / Google Kubernetes Engine (GKE) cluster, you can use the Cloudstate operator and a deployment descriptor such as:

There you have it—a scalable, distributed event-sourced service! 

And if you’d rather not manage your own Kubernetes cluster, then you can also run your Cloudstate service in the Akka Serverless managed environment, provided by Lightbend, the company behind Cloudstate.

To deploy the Cloudstate service on Lightbend Cloudstate simply run:

It’s that easy! Here is a video that walks through the full fraud detection sample:

You can find the source for the sample on GitHub: github.com/jamesward/cloudstate-sample-fraud

Akka Serverless under the hood

As an added bonus, Akka Serverless itself is built on Google Cloud. To deliver this stateful serverless cloud service on Google Cloud, Cloudstate needs a distributed durable store for messages. With the open-source Cloudstate you can use PostgreSQL or Apache Cassandra. The managed Akka Serverless service is built on Google Cloud Spanner due to its global scale and high throughput. Lightbend also chose to build their workload execution on GKE to take advantage of its autoscaling and security features.

Together, Lightbend and Google Cloud have many shared customers who have built modern, resilient, and scalable systems with Lightbend’s open source and Google’s Cloud services. So we are excited that Cloudstate brings together Lightbend and Google Cloud and we look forward to seeing what you will build with it! To get started check out the Open Source Cloudstate project and Lightbend’s Akka Serverless managed cloud service.

Roughly one billion people, or 15% of the world’s population, have some form of a disability¹. Here at Google, one of our core values is providing help to our users and with that, making our products as accessible as possible. In fact, my job is dedicated to building technology that makes Chrome Browser and Chrome OS more accessible. In recognition of National Disability Employment Awareness Month in October, we wanted to highlight new and existing accessibility features that you or teams can use while they work on the web and on Chromebooks. 

Before we get started, don’t forget to explore and enable accessibility features in your settings first. At the bottom right of your Chromebook, select the time, or press Alt + Shift + s. Then select “Settings.” Scroll to the bottom of the screen and choose “Advanced.” In the “Accessibility” section, select “Manage accessibility features.” If you want to enable them even quicker, turn on “Always show accessibility features” in the system menu of your Chromebook to skip some of these steps in the future.

Explore new enterprise accessibility policies 

In Chrome 84, fifteen new accessibility policies were added to the Google Admin console for Chrome OS devices. They allow IT to manage these settings centrally for their users. The new policies include: ChromeVox spoken feedback, Select-to-speak, High contrast, Screen magnifier, Sticky keys, Virtual keyboard, Dictation, Keyboard focus highlight, Caret highlight, Auto-click enabled, Large cursor, Cursor highlight, Primary mouse button, Mono audio and Accessibility shortcuts. Head to the Admin console to enable them in your organization.

Change your cursor color 

Another new feature allows your workforce to update the color of their cursor to improve its visibility on Chrome OS. There are seven new colors available: red, yellow, green, cyan, blue, magenta and pink, in addition to default black. To change the cursor, go to the “Mouse and touchpad” section of Settings.

Shade background text in Select-to-speak 

Select-to-speak lets users select text on a specific part of their screen and have it spoken aloud—extremely helpful for folks with low vision or learning disabilities. To make it easier to focus on the spoken text, you can now shade the background text that is not being highlighted. To enable this select-to-speak feature, search for “Select-to-speak settings” within Settings. 

Use Voice Switching and other ChromeVox enhancements

Recent enhancements to the ChromeVox screen reader help users with visual impairments use Chrome OS.  Employees can now utilize Voice Switching which automatically changes the screen reader’s voice based on the language of the text being read. Some additional updates include: new speech customization options, Smart Sticky Mode, and improved navigation on ChromeVox menus. To start using the screen reader, check out this help center article for more info.

Export accessible PDFs through Chrome 

Chrome now generates more accessible PDFs. Users can save web pages as a PDF that will include metadata like the page’s headings, lists, tables, paragraphs, and image descriptions. This makes the web more accessible for people with low vision or who are blind that use a screen reader to access PDF files.

Customize web page content and font sizes in Chrome

Chrome allows your workforce to change the size of everything on the website they visit, including text, images, and videos. It’s also possible to change only the size of the font on a web page. Read this help center article to learn how.

Zoom and magnify your screen

Employees with limited vision can use greater levels of zoom to make the screen more visible. They can also magnify their entire screen or specific parts of your screen. For more information on both of these options, read this help center article. 

Leverage extensions to improve accessibility 

There are many extensions that can also help users with disabilities enjoy the web. Check out this list of extensions for other tools that can alter Chrome to your users’ unique needs. As an admin, don’t forget you can manage and pre-install extensions of Chrome for your workforce. 

We’re continuously making improvements to Chrome and Chrome OS to ensure our products are as accessible as possible for everyone. We encourage you to share these features with your workforce to improve accessibility and enable them to be more productive. To stay updated on accessibility news from our team, check out the Accessibility Help Center. We also offer a Disability Support team who can help answer questions about using assistive technology and accessibility features with Google products. To chat with support agents, visit g.co/disabilitysupport.

To discover other benefits that Chrome and Chrome OS can bring to your organization, visit our website. 

^{1. Source: The World Bank, Disability Inclusion Overview, October 2020}