Incident Report

Summary

On Monday, March 10, 2025, at 03:50 US/Pacific, NotebookLM experienced an outage where users were unable to use chat, upload sources, or use interactive audio features globally for 2 hours and 40 minutes.

We deeply regret the disruption our Google Workspace customers experienced. We know you rely on our services, and we fell short of meeting your expectations. We are committed to improving our platform's reliability and performance, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

The root cause has been determined to be permission errors that impeded NotebookLM's access to the necessary underlying model.

Subsequent to a recent configuration change, the model was assigned to incorrect access control lists (ACLs) by the internal router that manages the models. This change resulted in the NotebookLM model being associated with a default model family possessing more stringent access controls.

Therefore, the NotebookLM model encountered permission errors, and incoming requests were rejected, culminating in customer inability to utilize the impacted features.

Remediation and Prevention

Google engineers were alerted of the outage from our internal monitoring system on March 10, 2025, at 03:50 US/Pacific, and immediately started an investigation. Once the underlying cause was identified, Google engineers added the required access controls to the default model at 06:30 US/Pacific to fully mitigate the impact.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• Implement a failsafe mechanism on the internal system that manages the Machine Learning (ML) models. This will prevent similar issues from occurring due to unintended configuration changes.

• Updating the configuration of the router to bypass the access control check when models are moved to the internal system.

Detailed Description of Impact

Between 03:50 US/Pacific and 06:30 US/Pacific on Monday, 10 March 2025, NotebookLM and Enterprise users were unable to use chat, upload sources, or use interactive audio. Customers experienced errors up to 100% due to this incident related to permission errors.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 10 March 2025 03:50

Incident End: 10 March 2025 06:30

Duration: 2 hours, 40 minutes

Affected Services and Features:

• Notebook LM

Regions/Zones: Global

Description:

NotebookLM users were unable to use chat, upload sources, or use interactive audio globally for a duration of 2 hours and 40 minutes.

The preliminary root cause of the issue was due to a recent change where a service was missing permissions to interact with another service. As a result of these missing permissions, we rejected incoming requests, leading to users being unable to use the affected features.

Engineers identified the root cause within the permissions system, and implemented a code change to resolve the issue.

We apologize for any inconvenience this may have caused. Google will complete a full Incident Report in the following days that will provide a full root cause analysis.

Customer Impact:

• NotebookLM users were unable to use chat, upload sources, or interactive audio.