Siemserva Evaluation Guide

A scavenger hunt. We describe what is seeded in the built-in demo tenant so you know which scenarios to hunt for. We do not tell you what Siemserva will flag, how severe it will rate the finding, or which compliance controls it will cite. The scanner says that. You watch it say it.

How to read this guide. Every section below describes what is in the demo Senserva security context graph, not what Siemserva will find. When you open the dashboard, the red badges, the severity ratings, the compliance citations, the remediation scripts, all of that is the scanner's work. If a scenario in this guide doesn't produce a finding in your run, that's a bug in Siemserva, not a gap in your reading. Evaluate us on whether our scanner sees what an attacker would see, not on whether we hyped it in a marketing page.

Reading offline? The full Evaluation Guide ships as SiemservaEvaluationGuide.pdf inside every release zip, next to the scanner binary. Same content, print-friendly.

Demo at a glance Personas Risky entities CA scenarios Sign-in signals Reports What-If Closing note
New to Siemserva? Walk through the Quick Start first. This page assumes you have the tool installed and a demo database built. See the Build the demo database card below if you just need the commands.

What's in the demo, at a glance

Three tenants, roughly 1,300 users and 1,400 devices, 260 groups, 130 applications, and 370 service principals. One tenant carries a detailed narrative; the other two provide realistic noise so every dashboard view has more than one bar on the chart. Device counts run slightly above user counts on purpose so BYOD, shared workstations, conference-room devices, and legacy kiosks are present. Group, app, and service-principal ratios match industry norms for enterprise tenants.

Canonical narrative

Zava Demo

Technology company, US. Carries every planted scenario: the No-MFA GA, the stale GA, the jailbroken iPhone, the Storm-0558 analog, the helpdesk compromise.

Users1,000 Groups180 Devices1,100 Apps100 Service principals300
Small tenant

Contoso Retail

Retail, US. Baseline scan-style noise so multi-tenant views always have comparison data.

Users90 Groups14 Devices100 Apps10 Service principals22
Mid-size tenant

Fabrikam Engineering

Manufacturing, Germany. More devices per user than Zava; a heavier app estate.

Users220 Groups40 Devices240 Apps20 Service principals48

Build the demo database

Two commands, no tenant permissions required. The demo Senserva security context graph lives entirely in a local SQLite file. For the full walkthrough with install links, trust-store setup, and troubleshooting, see Quick Start.

  1. Get Siemserva from the Microsoft Marketplace. Register on the free plan and we will email you a login location with the signed binaries. One file per platform, no installer. Every build is digitally signed: Notarized for macOS, Azure Artifact Signed for Windows.
  2. Pick your surface.
    Interactive dashboard
    siemserva-win-x64.exe demo dashboard
    Ask Claude in plain English
    siemserva-win-x64.exe demo claude
    Both commands build the same zava.sqlite in the current folder with all three tenants and every planted scenario. dashboard auto-launches the interactive TUI. claude wires Claude Desktop over MCP so you can ask questions in plain English; pass --cli to target Claude Code instead. Takes a minute or two on first run.
Already have a scan? Reopen the dashboard on a specific file with siemserva-win-x64.exe --reporter-db zava.sqlite --reporter --reporter-dashboard. See the Claude page for the full MCP story.

Personas to investigate

Every persona below is seeded into Zava Demo. We tell you who they are and what's unusual about their account. We don't tell you which finding rules will pick them up, what severity will be assigned, or which compliance controls will cite them. That's what the scanner is for.

Global Admin

Cameron Kline

What's planted: Global Administrator with a YubiKey registered. The Conditional Access policy that would require phishing-resistant MFA for Global Admins exists, but is in report-only mode.

Where to look: Users tab, sort by role. Open Cameron's drill-down. Check Authentication Methods, then the Conditional Access tab for "Phishing-Resistant".

Compromised helpdesk

Marcus Henderson

What's planted: Helpdesk role. Bulk password-reset burst against ~40 accounts from an external IP in a short window. No Conditional Access policy evaluated the session.

Where to look: Directory audit log for "Change user password" clustered in one window. Cross-reference the What-If scenarios. The Business Review report calls this out.

Correctly configured

Break-Glass Alpha / Beta

What's planted: Two emergency-access accounts. Excluded from every enabled CA policy. Two FIDO2 keys each, monitored via a dedicated alert path.

Where to look: Users tab, filter for "break-glass". Inspect CA exclusions. The one persona where green badges are the right answer.

Also seeded: Priya Raman (stale GA with standing role, PIM Privileged Role Admin eligibility, High user risk, Brussels sign-in attempt), Jordan Acosta (guest account holding an elevated directory role, non-compliant BYOD device), and an auto-forward user (inbox rule forwarding external mail outside the tenant, classic BEC-after-compromise).

Risky entities to hunt

Applications, service principals, devices, and SharePoint sites that were deliberately configured in dangerous states. Same rule as before: we tell you what was planted, we don't tell you how Siemserva will rate it.

Over-privileged OAuth app

MarketingToolsConnector

What's planted: Third-party OAuth app holding full_access_as_app, Mail.ReadWrite.All, and Sites.FullControl.All. Guest-owned, 400+ day credential. Classic Storm-0558 blast-radius shape.

Where to look: Applications tab, sort by risk. Drill down for owners, scopes, and credential expiry.

Risky service principal

ReportSyncEnterpriseApp

What's planted: Service principal granted Directory.ReadWrite.All and Application.ReadWrite.All. App-role assignments allow it to register new applications in your tenant.

Where to look: Service Principals tab, filter for risky. The Microsoft Enhancement report calls out SP privilege explicitly.

SharePoint sharing

Anonymous-share site

What's planted: A SharePoint site configured for anonymous (no-login) sharing. Tenant-wide sharing policy doesn't prohibit it. Classic leak path: a contractor link still live a year later.

Where to look: SharePoint tab. Also the SharePoint tenant admin settings row controlling cross-site sharing.

Also seeded: ExpiredVendorIntegration (vendor gone, credentials expired months ago, app still enabled with scopes), LegacyInvoiceImporter (orphan OAuth app with no owner, reads mail and files), and SALES-IPHONE-044 (iPhone 12 on iOS 15.4.1, non-compliant in Intune, guest-owned, recent sign-in attempts).

Conditional Access scenarios

The Conditional Access surface is where most posture issues live. The demo tenant carries deliberate gaps, deliberate misconfigurations, and one deliberate correct configuration. Let Siemserva tell you which is which.

Report-only trap

Policies stuck in report-only

What's planted: Multiple policies sit in report-only. Look protective in the list, enforce nothing. Includes the phishing-resistant-MFA-for-GA policy that would otherwise cover Cameron.

Where to look: Conditional Access tab, filter by state=reportOnly. Cross-reference the What-If scenarios.

Open door

No policy evaluated

What's planted: Marcus Henderson's external-IP sign-in hit an app-flow combination where zero Conditional Access policies applied. Silent failure most teams miss.

Where to look: What-If scenarios tab. Sign-in logs for the helpdesk user. The Business Review report surfaces this by name.

Correctly configured

Break-glass exclusion audit

What's planted: Break-glass accounts are excluded from every enabled CA policy. Intentional and correct. Siemserva should recognize this as a healthy pattern, not flag it as a gap.

Where to look: Conditional Access tab, filter any policy, inspect Excluded Users. Expect a positive-signal row.

Also seeded: Finance group exclusion (the quarter-end MFA exclusion that never came off), sign-in + user risk combined in one policy (Microsoft guidance splits them), and a country allowlist/block list with a named "Minneapolis HQ" trusted range.

Sign-in and audit log signals

The demo carries realistic telemetry: failed sign-ins, push-fatigue bursts, anomalous locations, legacy auth traffic, bulk admin actions. If the scanner correlates any of this, you'll see it surface. We just tell you what patterns are there.

Push-fatigue pattern

Cameron's midnight burst

What's planted: Six failed MFA prompts in a five-minute window overnight for Cameron Kline. From an IP never seen before. A seventh attempt later that night succeeded.

Where to look: Sign-In Logs tab. Filter for Cameron. Sort by time. Look for the cluster crossing midnight.

Anomalous location

Priya from Brussels

What's planted: A sign-in attempt for Priya Raman originating from Brussels, inconsistent with her normal pattern. Identity Protection flagged it as High user risk.

Where to look: Sign-In Logs tab, filter Priya, filter to Brussels. Cross-reference the Risky Users view.

Bulk admin action

Bulk password-reset burst

What's planted: Marcus Henderson ran ~40 "Change user password" actions from an external IP in a short window. No CA policy evaluated the session.

Where to look: Directory audit log. Filter on "Change user password". The cluster is obvious. The Business Review report calls this out.

Also seeded: IMAP and SMTP attempts against the tenant (legacy-auth traffic that should be refused), Jordan's non-compliant device blocked (BYOD device with OS under floor hitting a CA compliance gate), and a report-only canary failure (a policy in report-only would have blocked a known benign action, so the canary surfaces the silent would-be enforcement).

Reports to generate

Press R in the dashboard and pick a report type. Each has a different frame. We describe what each one is for. We do not preview the contents.

Detailed

Every finding, every entity, every compliance code the scanner mapped. For the engineer who wants to verify a scan thoroughly before the first demo call, and for the History view of entity change timelines.

Compliance

Findings organized by CISA SCuBA, Microsoft MCSB, CIS, NIST 800-53, and ISO 27001. For the auditor asking which controls pass, which fail, and which are partial.

Remediation

What to fix first, this week, and this quarter, with Graph PowerShell SDK scripts ready to run. Pair with the Business Review for executive framing and Microsoft Enhancement for license upgrade deltas.

The What-If panel (pre-evaluated)

Five Conditional Access scenarios are replayed against the full policy set in the canonical tenant. The results are written into the scan so you can drill in. The ones where the verdict is "allowed" are the interesting cells.

Scenario Who Verdict
Priya signs in from Brussels Stale GA, High risk allowed
Cameron activates GA without MFA No-MFA GA report-only
Helpdesk bulk password-reset burst Compromised helpdesk allowed (no policy)
Jailbroken iPhone hits Exchange Online Guest-owned BYOD blocked
Break-glass sign-in during Azure outage Break-Glass Alpha allowed (by design)

Where to look: Conditional Access tab, What-If panel. Open each scenario for the full list of policies evaluated, which matched, which would grant, which would block.

From the blog

Reading from the Senserva team.