Finding the right hashcat -m mode (and what to do when you get it wrong)
Hashcat won't autodetect anything. Here is how to pick the correct -m mode, disambiguate look-alike hashes, and read the errors that mean you chose wrong.
Hashcat does not look at your hash and figure out what it is. You tell it, with -m, and if you tell it wrong it will happily run for hours and crack nothing. That is the single most common reason people think a password is "uncrackable" when it was a one-word dictionary hit the whole time.
So the real skill is not running hashcat. It is choosing the mode.
The mode number is the algorithm, not the hash
-m 0 is MD5. -m 1000 is NTLM. -m 1800 is sha512crypt. The number encodes the exact construction, including how the salt is mixed in, so md5($pass) and md5($salt.$pass) are different modes (0 and 20) even though both produce 32 hex characters. Get the construction wrong and every candidate you try hashes differently from the target. No match, ever.
This is why identifying the hash comes first. The format tells you the family. The source tells you the specific mode.
Use --identify, then ignore most of what it says
Recent hashcat ships with a shortlist tool:
hashcat --identify 5f4dcc3b5aa765d61d8327deb882cf99
That returns the modes whose structure matches the string. For a bare 32-hex value you will get a long list: MD5, NTLM, MD4, the double-MD5 variants, and so on. It is correct and not very helpful, because all of those are structurally identical. The tool can only see the string. It cannot see that the hash came out of secretsdump.py against a domain controller, which would make it NTLM and nothing else.
So treat --identify as a sanity check. The decision comes from context:
- Pulled from
NTDS.dit, the SAM hive, or asecretsdumpline that looks likeuser:1001:aad3b435...:b4b9b02e...:::? That right half is NTLM,-m 1000. The left half is LM, and if it isaad3b435b51404eeaad3b435b51404eeit is the empty LM hash, skip it. - Sitting in a
userstable in a web app dump? Overwhelmingly raw MD5 (-m 0) or SHA-1 (-m 100), occasionally stored ashash:salt. - Starts with
$2y$or$2b$? bcrypt,-m 3200, and you should lower your expectations accordingly. - Starts with
$6$? sha512crypt out of/etc/shadow,-m 1800.
Length narrows the family. Provenance picks the mode.
The errors that mean you guessed wrong
Two messages come up constantly and people misread both.
"Token length exception" means the hash does not parse under the mode you picked. The structure is wrong. Nine times out of ten it is a trailing newline from a bad copy-paste, a username column you forgot to strip, or simply the wrong -m. Run it through xxd if you are unsure what is actually in the file:
xxd hash.txt | tail -2
A 0a at the end is a newline. Hashcat tolerates one trailing newline per line, but two hashes glued together or a Windows 0d 0a will break parsing.
"Separator unmatched" shows up when you use --username or a salted mode and the colon count is off. Salted modes expect hash:salt. If the salt itself contains a colon, or you left the user: prefix on, the parser splits in the wrong place.
Neither error means the hash is exotic. It almost always means a formatting problem or the wrong mode.
The silent failure is the dangerous one
There is a worse case than an error: the wrong mode that still parses. md5($pass) and md5($pass.$salt) both accept a 32-hex token. Point -m 0 at a salted MD5 and hashcat runs cleanly, churns through your whole wordlist, and reports nothing cracked. No error. Just a wrong answer that looks like a hard hash.
The defense is a known-answer test. Before you trust a long run, hash a password you already know with the same construction and confirm hashcat cracks it in seconds:
echo -n "password" | md5sum
# 5f4dcc3b5aa765d61d8327deb882cf99
echo "5f4dcc3b5aa765d61d8327deb882cf99" > test.txt
hashcat -m 0 -a 0 test.txt <(echo password)
If that does not return password, your mode is wrong and there is no point launching the real attack.
When in doubt, let John triage
John the Ripper autodetects, and that makes it a good first pass when you genuinely do not know what you are holding. Throw the file at john with no format flag and read what it picked:
john hash.txt
john --show hash.txt
If John settles on Raw-MD5 you have your answer, and you can move the work to hashcat with -m 0 for the GPU throughput. I cover when to stay in John versus move to hashcat in the tooling comparison.
One more thing about optimized kernels
If you run with -O (optimized kernels) hashcat caps the candidate password length, often at 31 characters or less depending on the mode. For most dictionary work that is irrelevant. But if you are running a mask that generates long candidates and nothing cracks, the optimized kernel quietly skipped everything past the limit. Drop -O and rerun before you conclude the password is long and random. It might just be 33 characters of something you already had in your list.