Blog/NEWS//4 min read

9,791 AI findings, 99 reports: what GitHub and Microsoft's MDASH pilot says about AI security testing

GitHub and Microsoft's MDASH pilot cut 9,791 AI findings to 99 checked reports, and 55 are fixed. Why verified findings beat raw volume when you buy.

WRITTEN BY, Security Engineer
LENGTH710 words

GitHub and Microsoft ran an AI security scan across six major open-source projects. It flagged 9,791 possible issues, and only 99 survived checking and reached maintainers. That is about one report per hundred raw findings. If you are buying AI-driven security testing, the number that matters is the hundred-to-one filter, not the 9,791. The filtering is where most of the value was created.

What happened

On 17 November, GitHub published results from a pilot with Microsoft called Codename MDASH. The process had five steps:

  • 1. Find. MDASH, Microsoft's multi-model agentic scanning harness, scanned each project using Microsoft's MAI Cyber model and suggested fixes.
  • 2. Filter. An AI triage agent from GitHub Security Lab removed duplicates, checked validity and focused on high-impact bugs.
  • 3. Verify. Seven researchers from Microsoft FORGE Lab checked every remaining finding by hand.
  • 4. Report privately. Each report went to maintainers through GitHub private vulnerability reporting, with a proof of concept and a suggested fix.
  • 5. Fix. Maintainers decided what to fix and when.
Five steps of the MDASH pilot: find, filter, verify by hand, report privately, fix
The pilot's five steps. Source: GitHub Blog, 17 November 2026.

The six projects were AutoGPT, Caddy, Express, FastMCP, ImageMagick and Netty. As of 15 November, maintainers had fixed 55 of the 99 reports, and 14 of those fixes had shipped in a release. Caddy's maintainers fixed all 22 of their reports in a week or less. One ImageMagick fix took 15 minutes.

Log-scale bar chart of the MDASH pilot: 9,791 possible issues flagged, 99 private reports sent to maintainers, 55 fixed, 14 fixes already in a release, as of 15 November 2026
From 9,791 possible issues to 99 reports, 55 fixes and 14 shipped fixes (as of 15 November 2026; log scale). Source: GitHub Blog, 17 November 2026, with 9,791 from the post's results infographic.

GitHub is candid about the limits. Six projects is a small sample. The researchers spent about two days per project checking findings by hand, and GitHub says that work "doesn't scale yet."

Why it matters for security buyers

The pilot puts numbers on something every security team already feels. AI has made finding possible issues cheap. Checking them is still expensive, and someone has to pay that cost. In this pilot it was paid before anything reached a maintainer. A Netty maintainer put it this way: "Finding vulnerabilities is no longer our bottleneck. Fixing them is the hard part, and that depends on maintainers."

The same problem shows up after big public disclosures. Google's threat intelligence team reported on 16 November that in the days after the PHP flaws patched on 21 October were fixed, hundreds of claimed exploits were published that did not work, likely generated with AI help. Someone had to check each one, and that work fell on defenders.

When you evaluate an AI pentest or scanning vendor, the risk is that the filtering step gets skipped and handed to you as a long list. Volume can look like value in a demo. It costs your engineers time in practice.

What to ask a vendor

  • 1. What share of findings reach us, and who checked them? Ask whether a human or an exploit confirmed each one.
  • 2. Does every high or critical finding come with a proof that your team can reproduce?
  • 3. Is there a suggested fix, written for the engineer who owns the code?
  • 4. What happens after we fix it? Ask whether the vendor reruns the same test and records the result.
  • 5. How do you measure false positives, and will you share that number for our environment?

If you maintain open source, GitHub's advice is to turn on private vulnerability reporting. If you depend on open source, turn on Dependabot alerts so you hear about fixes like these.

Where HOG3 fits

HOG3 is built around the same principle at the application level. Within your authorized scope, it chains weaknesses into a working attack path. Every high or critical finding comes with a reproducible proof and steps to fix it. After your fix, HOG3 retests the original issue and checks related paths before closing it. We were not part of the MDASH pilot. We share its view that a checked finding is worth more than a long list. See how HOG3 works.

Sources

All accessed 18 November 2026.