- Test sets included real world casings similar to those
you would see in practice, with a diversity of toolmark
type and quality.
- Matching based on 0-1 confidence score which separately
quantifies similarity between the true breech-face
impression and the linear profiles of the aperture shears.
- No False Positives are seen across
all experiments (more than 100,000 comparisons).
- For a single ammunition type, the algorithm identifies a
correct known-match above the significance threshold for
approximately 84% of casings.
- When the casings are well marked (eg. Fadul study), 100%
of casings have a correct match above threshold.
- These results are current as of early 2015. Many improvements are currently being tested.
Several experiments were conducted
to validate the TopMatch imaging system and matching algorithms.
The following summarizes the most recent experiments. Full
details as well as additional studies will be described in a
journal publication currently undergoing peer review. NOTE: It
has always been our goal to maximize the number of identified
Known Matches while maintaining ZERO False Positives. That is,
we err on the side of no false positives which means that the
algorithm may fail to recognize a known match if the algorithm
doesn't have high confidence in the match.
The four datasets used in testing. The Firearms
column is the number of firearms represented in the dataset.
The Casings column lists the number of
casings in the dataset. The Ammo column
graphically shows the number of ammunition brands present in
each dataset. The Marking column lists the
type of markings seen on the casings: Real-world casings
represent the range of casings typically encountered in
actual casework. It includes casings that are well marked or
poorly marked and which contain granular, milled, and filed
breech-face impressions. Some casings have a visible
aperture shear. Glock casings represent real-world Glock
casings which often rely on aperture shear visualization and
matching for identification or elimination. Well-Marked
casings represent casings with very strong visible marks.
Data Set 1: Forty Seven Firearms
Forty seven "real world" firearms were selected without
preference to their ability to mark cartridge casings. The
intent was to select firearms that would represent real-world
conditions. In reality, the toolmarks left on a casing range
from being extremely reliable and
interpretable to being unreliable, irreproducible, and
barely present. The firearms include: 2x Colt, 5x Hi-Point, 7x
Fabrique Nationale, 5x S&W, 5x Radom, 16x Ruger (including
10 with consecutively manufactured breech-faces), 5x Norinco, 1x
FEG, 1x Springfield Armory. Most firearms came from the Oakland
reference collection. Some firearms were obtained from our
collaborator Andy Smith (SFPD).
The first dataset included three test fires of PMC brand (115GR
bullet, brass casing, brass primer) from each of the 47
firearms. A second set included test fires from three different
types of ammunition. Match scores range from 0 (low-similarity)
to 1 (high-similarity) and incorporate both the breech-face
impression and aperture shear similarity (aperture shear
comparison is based on the comparison of the linear striation
The results table shows that 82% of the casings have a correct
match with match score above the threshold of 0.7. There are no
false positives among 9,729 known non-matches.
All-vs-all match scores for the forty seven firearm
experiment. "Casings with Match": the
number of casings in the set that have a correct known
match, "Recall": the percent of casings
with a match for which a known match is identified, "FPR":
False Positive Rate, the percent of known non-matches that
have a match score above the specified threshold, "KNM":
Known Non-Matches, the number of known non-matches in the
Most firearms match well, some firearms simply do not mark
well. Note there are no false positives.
We note that some of the poorly marked casings are difficult to
match. These difficult matches can have very low match scores,
scores that overlap with the Known Non-Matches. These matches
would be considered missed hits. However, it's most important to
stress that we have no false positives across all experiments
performed. That is, we never call a Known Non-Match as a match.
Stated another way, a KM involving two poorly marked casings may
have a small match score, but a KNM never has a large match
score. While we are working on algorithmic improvements to
increase the number of identified known matches, there an
inherent upper bound; some firearms simply do not mark well
enough to be identified.
Data Set 4: Miami-Dade Test Set 8
We next explored the Miami-Dade Study test fires (materials
kindly provided by Dr. Thomas Fadul). We included this set to
demonstrate TopMatch's performance on a set of well marked
casings (in contrast to the real-world test sets above). The
Miami-Dade set consists of ten pairs of matched knowns and
fifteen individual 'questioned' (or unknown) casings. The
examiner is tasked with matching each questioned case with one
of the known pairs. All test fires use Federal Cartridge
ammunition (brass casing, nickel primer). In contrast to our
other data sets, the casings in the Miami-Dade Study are all
strongly marking. This is reflected in the match score results.
The confusion matrix is shown below. All questioned cases are
correctly identified with their matching known pairs.
Well-marked casings produce 100% matching accuracy and
perfect separation between known matches and known
Data Set 2: Large Collection
Set 2 includes the forty seven firearms from Set 1 plus
additional real world casings collected by our collaborators.
The casings were not selected based on their ability to mark
well. Altogether 101 firearms are represented among 337 casings
from the following firearm manufacturers: Armi Fratelli, Baikal,
Beretta, Browning Arms, Bryco Arms, Colt, Hi-Point, Fabrique
Nationale, FEG, Heckler & Koch, Intratec, Kahr Arms, Keltec,
S&W, Radom, Ruger (including 10 with consecutively
manufactured breech-faces), Norinco, Sig Sauer, Springfield
Armory, Star, Taurus, Uzi, and Walther. A total of 328 of these
337 casings have a known match in the test set, 9 casings do not
have a known match. Seven different ammunition types are
represented in the set.
Approximately 70% of casings have identified known
matches. The slight decrease in performance from Set 1 is
likely due to two factors. 1) This set includes seven
different ammunition types, for many pairs one casing of a
known match may have been collected with one ammunition type
and the sister-casing (known match) may have been collected
with a different ammunition type. 2) Some of these scans are
a bit 'dirty' in that a lower-resolution scanning gel was
used during scan acquisition. We are looking into this
possibility. Most importantly there are no false positives
across more than 56,000 known non-matches.
Data Set 3: Glock
The final set includes 328 paired test fires from 164 Glock
firearms (materials kindly provided by Dr. Jim Hamby). Ten
different ammunition types are represented in this set. For many
known matches, one casing may have been collected with one
ammunition type and the sister-casing (known match) may have
been collected with a different ammunition type. Glock casings
heavily rely on the ability to match aperture shear profiles.
Matching using breech-face impressions only (no shears) had an
approximately 20% lower recall rate. These results use our
prototype aperture shear matching algorithm. The algorithm is
currently being updated. Most importantly there are no false
positives across more than 53,000 known non-matches.
Glock matching performance across 10 ammunition types. No
false positives are reported across more than 53,000 known