Office 365 Security Alerts: Automated Risk Analysis
OFFICE 365
The Challenge
Manually monitoring Office 365 security logs is an exhausting and error-prone task. With thousands of events generated daily, identifying real threats—such as login attempts on disabled accounts or compliance failures—amidst operational "noise" (like KMSI interrupts) is a constant challenge for IT teams.
The goal was to:
- Identify active security risks in the tenant
- Filter irrelevant events (false positives)
- Obtain clear, immediate action recommendations
- Detect compromised credentials or unauthorized access attempts
The Prompt Sent to Myrmex
Using the integration with the Office 365 tenant, we sent:
Retrieve the 5 most recent Office 365 security alerts from the tenant.
For each alert, analyze the threat level and provide a concise list
of remediation steps. Focus on immediate actions to mitigate risk,
such as password resets or device isolation.
Note: Myrmex acts as a senior security analyst, cross-referencing raw log data with threat intelligence to deliver actionable insights.
What Myrmex Did
Myrmex connected to the Microsoft Entra ID Sign-in Logs API (since the advanced alerts API had license restrictions) and scanned the latest authentication events.
Summary of Analyzed Events
| Timestamp (UTC) | User | Event Detected | Threat Level | Recommended Action |
|---|---|---|---|---|
| 19:20 | Robson Severo | Sign-in on Disabled Account | 🟡 Medium | Revoke Sessions |
| 19:19 | Robson Severo | Sign-in on Disabled Account | 🟡 Medium | Audit IP Source |
| 18:58 | Larissa Faustino | KMSI Interrupt | 🔵 Info | No Action |
| 18:36 | Larissa Faustino | Password Expired | 🟡 Low | Reset Password |
| 18:02 | Vinicius Silva | KMSI Interrupt | 🔵 Info | No Action |
Analysis and Actions Taken
1. Detection of Access on Disabled Account
Myrmex identified multiple failed login attempts for user robson.severo@ainext.com.br, originating from a Windows 10 device (GRS).
Risk Identification: Even with the account disabled, a device or local client (Outlook/Teams) kept trying to authenticate. This indicates the device was not properly wiped or disconnected upon employee offboarding.
Myrmex Recommended Actions:
- Immediate: Execute "Revoke sessions" in Entra ID to invalidate Refresh Tokens.
- Device: Perform remote wipe on device
GRSif managed by Intune. - Network: Monitor IP
2804:1e68:8401:d71a:519d:7d32:3b03:70e5.
2. Noise Filtering (KMSI)
"KMSI Interrupt" (Keep Me Signed In) events were automatically classified as Info, preventing the security team from wasting time investigating standard system behaviors.
3. Credential Management
For the "Password Expired" error, Myrmex suggested a self-service flow for the user, reducing the load on Level 1 support tickets.
Benefits of the Approach
🚀 Proactive Detection
Instead of waiting for a support ticket or a severe incident, the automated audit detected an "orphan" device attempting authentication, allowing preventive action.
📉 Alert Fatigue Reduction
Automatic risk classification (Medium vs. Info) allows the team to focus only on what matters (unauthorized access) and ignore noise (KMSI).
Result
With Myrmex analysis:
- ✅ Risk Mitigated: Identification of device attempting unauthorized access.
- ✅ Visibility: Clarity on which accounts are generating error logs.
- ✅ Efficiency: Clear separation between technical issues (password) and security (disabled account).
Prompt Variations
The same pattern can be used for other security searches:
To monitor administrators:
Analyze the last 20 logins from users with 'Global Administrator' role
and alert on access outside business hours or from unusual countries.
For device compliance:
List all devices that attempted login in the last 24h
that are not marked as 'Compliant' in Intune.