TopMatch-GS 3D

Some Experimental Results.
  • Home
  • /
  • Experiments

Takeaways:

  • Test sets included real world casings similar to those you would see in practice, with a diversity of toolmark type and quality.
  • Matching based on 0-1 confidence score which separately quantifies similarity between the true breech-face impression and the linear profiles of the aperture shears.
  • No False Positives are seen across all experiments (more than 100,000 comparisons).
  • For a single ammunition type, the algorithm identifies a correct known-match above the significance threshold for approximately 84% of casings.
  • When the casings are well marked (eg. Fadul study), 100% of casings have a correct match above threshold.
  • These results are current as of early 2015. Many improvements are currently being tested.

Several experiments were conducted to validate the TopMatch imaging system and matching algorithms. The following summarizes the most recent experiments. Full details as well as additional studies will be described in a journal publication currently undergoing peer review. NOTE: It has always been our goal to maximize the number of identified Known Matches while maintaining ZERO False Positives. That is, we err on the side of no false positives which means that the algorithm may fail to recognize a known match if the algorithm doesn't have high confidence in the match.



Experiment Summaries


The four datasets used in testing. The Firearms column is the number of firearms represented in the dataset. The Casings column lists the number of casings in the dataset. The Ammo column graphically shows the number of ammunition brands present in each dataset. The Marking column lists the type of markings seen on the casings: Real-world casings represent the range of casings typically encountered in actual casework. It includes casings that are well marked or poorly marked and which contain granular, milled, and filed breech-face impressions. Some casings have a visible aperture shear. Glock casings represent real-world Glock casings which often rely on aperture shear visualization and matching for identification or elimination. Well-Marked casings represent casings with very strong visible marks.



 

Data Set 1: Forty Seven Firearms

Forty seven "real world" firearms were selected without preference to their ability to mark cartridge casings. The intent was to select firearms that would represent real-world conditions. In reality, the toolmarks left on a casing range from being extremely reliable and interpretable to being unreliable, irreproducible, and barely present. The firearms include: 2x Colt, 5x Hi-Point, 7x Fabrique Nationale, 5x S&W, 5x Radom, 16x Ruger (including 10 with consecutively manufactured breech-faces), 5x Norinco, 1x FEG, 1x Springfield Armory. Most firearms came from the Oakland reference collection. Some firearms were obtained from our collaborator Andy Smith (SFPD).

The first dataset included three test fires of PMC brand (115GR bullet, brass casing, brass primer) from each of the 47 firearms. A second set included test fires from three different types of ammunition. Match scores range from 0 (low-similarity) to 1 (high-similarity) and incorporate both the breech-face impression and aperture shear similarity (aperture shear comparison is based on the comparison of the linear striation profile).

The results table shows that 82% of the casings have a correct match with match score above the threshold of 0.7. There are no false positives among 9,729 known non-matches.


All-vs-all match scores for the forty seven firearm experiment. "Casings with Match": the number of casings in the set that have a correct known match, "Recall": the percent of casings with a match for which a known match is identified, "FPR":  False Positive Rate, the percent of known non-matches that have a match score above the specified threshold, "KNM": Known Non-Matches, the number of known non-matches in the dataset.
Most firearms match well, some firearms simply do not mark well. Note there are no false positives.


We note that some of the poorly marked casings are difficult to match. These difficult matches can have very low match scores, scores that overlap with the Known Non-Matches. These matches would be considered missed hits. However, it's most important to stress that we have no false positives across all experiments performed. That is, we never call a Known Non-Match as a match. Stated another way, a KM involving two poorly marked casings may have a small match score, but a KNM never has a large match score. While we are working on algorithmic improvements to increase the number of identified known matches, there an inherent upper bound; some firearms simply do not mark well enough to be identified.




 

Data Set 4: Miami-Dade Test Set 8

We next explored the Miami-Dade Study test fires (materials kindly provided by Dr. Thomas Fadul). We included this set to demonstrate TopMatch's performance on a set of well marked casings (in contrast to the real-world test sets above). The Miami-Dade set consists of ten pairs of matched knowns and fifteen individual 'questioned' (or unknown) casings. The examiner is tasked with matching each questioned case with one of the known pairs. All test fires use Federal Cartridge ammunition (brass casing, nickel primer). In contrast to our other data sets, the casings in the Miami-Dade Study are all strongly marking. This is reflected in the match score results. The confusion matrix is shown below. All questioned cases are correctly identified with their matching known pairs.


Well-marked casings produce 100% matching accuracy and perfect separation between known matches and known non-matches.



 

Data Set 2: Large Collection

Set 2 includes the forty seven firearms from Set 1 plus additional real world casings collected by our collaborators. The casings were not selected based on their ability to mark well. Altogether 101 firearms are represented among 337 casings from the following firearm manufacturers: Armi Fratelli, Baikal, Beretta, Browning Arms, Bryco Arms, Colt, Hi-Point, Fabrique Nationale, FEG, Heckler & Koch, Intratec, Kahr Arms, Keltec, S&W, Radom, Ruger (including 10 with consecutively manufactured breech-faces), Norinco, Sig Sauer, Springfield Armory, Star, Taurus, Uzi, and Walther. A total of 328 of these 337 casings have a known match in the test set, 9 casings do not have a known match. Seven different ammunition types are represented in the set.


Approximately 70% of casings have identified known matches. The slight decrease in performance from Set 1 is likely due to two factors. 1) This set includes seven different ammunition types, for many pairs one casing of a known match may have been collected with one ammunition type and the sister-casing (known match) may have been collected with a different ammunition type. 2) Some of these scans are a bit 'dirty' in that a lower-resolution scanning gel was used during scan acquisition. We are looking into this possibility. Most importantly there are no false positives across more than 56,000 known non-matches.



 

Data Set 3: Glock

The final set includes 328 paired test fires from 164 Glock firearms (materials kindly provided by Dr. Jim Hamby). Ten different ammunition types are represented in this set. For many known matches, one casing may have been collected with one ammunition type and the sister-casing (known match) may have been collected with a different ammunition type. Glock casings heavily rely on the ability to match aperture shear profiles. Matching using breech-face impressions only (no shears) had an approximately 20% lower recall rate. These results use our prototype aperture shear matching algorithm. The algorithm is currently being updated. Most importantly there are no false positives across more than 53,000 known non-matches.


Glock matching performance across 10 ammunition types. No false positives are reported across more than 53,000 known non-matches..