About the Cayosoft Guardian password security database

Cayosoft Guardian ships a password security database: a curated list of password hashes from publicly known data breaches. Cayosoft Guardian uses it to tell you whether accounts in your Active Directory are using passwords that attackers already know.

This article explains what the database contains, why it is much smaller than the original breach dataset, how complete its coverage is, and which threat signatures depend on it.

How Cayosoft Guardian uses the database

Cayosoft Guardian never sends your password hashes anywhere. The comparison runs where the hashes already live:

1. Cayosoft downloads the full Have I Been Pwned (HIBP) NTLM hash dataset to a Cayosoft cloud server.
2. Cayosoft filters that dataset down to the most frequently used password hashes and publishes the result as a password security database release.
3. Your Cayosoft Guardian server downloads the release.
4. Cayosoft Guardian compares account password hashes locally, on your domain controllers, against the downloaded list.

Cayosoft Guardian does not store or transmit password hashes. All password-related comparison operations run locally on your domain controllers, so password hashes stay inside your environment and are never sent to the internet or to external systems. Password hashes are not collected, exported, or exposed outside your Active Directory domain controllers during backup, recovery, or change monitoring operations.

What the database contains

Item	Detail
Hash type	NTLM (not SHA-1). Cayosoft Guardian requests the NTLM variant of the HIBP Pwned Passwords dataset so that hashes can be compared directly against Active Directory password hashes.
Source	Have I Been Pwned - Pwned Passwords
Contents	Password hashes only - one hash per line, sorted alphabetically.
Not included	Plaintext passwords, user names, email addresses, domain names, breach names, or any data from your environment.
Approximate size	About 1.5 GB uncompressed, shipped as a compressed archive.

The hashes in the database are not tied to any individual. A hash appears in the list because that password was seen in one or more breach corpora - not because a particular person used it.

Why the database is about 1.5 GB and not about 100 GB

The original HIBP source data is roughly 100 GB and contains about 1.3 billion distinct password hashes.

Shipping all of that to every Cayosoft Guardian server is not practical: it would mean a very large download, a large amount of disk on the Cayosoft Guardian server, and a much longer threat job run.

So Cayosoft ships a subset rather than the full set. The subset keeps the most frequently used password hashes and drops the long tail of hashes that were each seen only a handful of times. The result is roughly 1.5 GB - about 1.5% of the original size - while still covering the majority of real-world password use.

The reason this works is that password reuse is extremely concentrated. In the source data:

• About 71% of all distinct hashes were seen only 1-5 times each, but together they account for only 6.7% of all password occurrences.
• Conversely, the hashes seen more than 100 times each are only about 2.9% of all distinct hashes, yet they account for about 66% of all password occurrences.

In other words, dropping the rare hashes removes most of the size but very little of the risk.

Hash frequency statistics

"Frequency" is the number of times a given password hash was observed across the breach corpora. A higher frequency means the password is more commonly used - and therefore more likely to be tried first in a password-spraying or dictionary attack.

The source dataset contains 1,297,509,369 distinct hashes with a combined frequency of 28,636,824,846 occurrences, distributed as follows:

Frequency range	Distinct hashes	% of all hashes	Total occurrences	% of all occurrences
1-5	919,607,640	70.87%	1,922,223,371	6.71%
6-10	128,612,129	9.91%	962,388,691	3.36%
11-20	82,876,791	6.39%	1,228,236,555	4.29%
21-50	88,497,535	6.82%	2,874,067,985	10.04%
51-100	40,841,625	3.15%	2,850,968,758	9.96%
101-1,000	34,064,055	2.63%	8,476,983,939	29.60%
1,001-5,000	2,657,441	0.20%	5,147,320,472	17.97%
More than 5,000	352,153	0.03%	5,174,635,075	18.07%
Total	1,297,509,369	100%	28,636,824,846	100%

What this means for coverage

The shipped database keeps roughly the top 40 million hashes by frequency - about 3% of all distinct hashes. Those hashes account for approximately 66% of all recorded password occurrences.

Put simply: if an account in your environment is using a password that appears in a breach corpus, there is roughly a two-in-three chance that the specific hash is in the shipped database and Cayosoft Guardian will flag it.

Hashes shipped	Share of all distinct hashes	Approximate coverage of all password occurrences
~2.4 million (earlier, "100 MB" selection)	0.2%	~33%
~40 million (current)	~3%	~66%
All 1.3 billion (not shipped)	100%	100%

Cayosoft plans to increase the size of the database in future releases so that it covers more passwords.

Which threat signatures use the database

Cayosoft Guardian's password security threat signatures are:

Signature	Name	Uses the password security database?
CTD-000178	AD user with compromised password	Yes. Compares each account's password hash against the shipped breach hash list. Coverage of this signature is determined directly by what is in the database.
CTD-000179	AD user has a password that matches its sAMAccountName	No. Cayosoft Guardian derives a hash from the account's own sAMAccountName and compares it locally.
CTD-000180	AD user with blank password	No. Cayosoft Guardian checks for an empty password.
CTD-000181	AD user with identical password	No. Cayosoft Guardian compares accounts' password hashes with each other, within a single domain.

Only CTD-000178 depends on the contents of the password security database. If a compromised password is not detected, the most likely explanation is that its hash falls in the low-frequency tail that the shipped subset does not include - not that the check failed.

The other three signatures work entirely from data already in your directory, so a database update does not change their results.

Note: Duplicate password detection (CTD-000181) has a per-domain scope in the current release, and Cayosoft Guardian does not expose which specific accounts share a password.

What is included in each password security database release

Each release consists of:

• A compressed archive containing a single file of NTLM password hashes, one per line, sorted alphabetically. The archive is named with the release version and the archive checksum.
• A manifest entry that records the release version, the download location, the target path, and the archive checksum, so that the Cayosoft Guardian server can verify what it downloaded.

Each release is a full replacement, not an incremental patch: the new file supersedes the previous one entirely.

A new release is produced when the underlying HIBP dataset is refreshed or when the selection criteria change, for example, when the number of hashes shipped is increased.

Frequently asked questions

Does Cayosoft Guardian send our password hashes to Cayosoft or to Have I Been Pwned?
No. The breach hash list is downloaded to your Cayosoft Guardian server, and all comparisons run locally on your domain controllers. No hash from your environment leaves your environment.

Does the database contain actual passwords?
No. It contains NTLM hashes only. There is no plaintext, and no user or account information of any kind.

Why was a password we know is weak not flagged by CTD-000178?
CTD-000178 flags passwords whose hashes are in the shipped subset - the most frequently breached passwords. A weak password that has never appeared in a breach corpus, or that appears only a few times, will not be in the subset. Weak passwords of other kinds are covered by CTD-000179, CTD-000180, and CTD-000181.

Will the database get bigger?
Yes. Increasing coverage is on the roadmap, along with a planned check that matches breached accounts by domain suffix.

How much disk space does the database need on the Cayosoft Guardian server?
About 1.5 GB for the extracted file, plus space for the downloaded archive.