The changing face of image comparison technology

After a bumpy start, Facial Recognition Technology is suddenly very topical in policing circles. No longer just regarded as a term that’s synonymous with breaching human rights, it’s also being seen in a positive light in regard to the benefits of its application. The outgoing Biometrics and Surveillance Commissioner Prof. Fraser Sampson is recently on record as stating: “In two years people will not be asking why police are using Facial Recognition Technology (FRT), but why aren’t they using it!”

So how did the dial move? It started with South Wales Police and the Metropolitan Police legitimately asking “what can we do?” The response was a deafening silence from legislators and policy makers. Maybe this was because the subject was untested ground, but perhaps it was also that those opposed already owned the narrative.

Faced with a blank canvas, a risk-based approach was adopted by both forces and decisions were made with the best intentions. Guidance and the law were crafted in stated cases such as R v Bridges following legitimate challenges. The findings were not all negative either, even though reading the mainstream media you might be forgiven for thinking they were. For example in R v Bridges, the Judge reinforced the right of the police in Common Law to take and use photographs for a policing purpose. This in effect gives legitimacy in law to police use of facial recognition, as long as stringent policies covering consultation, proportionality and certain legalities are in place and adhered to.

It is worth mentioning police leaders such as Chief Inspector Scott Lloyd of South Wales Police and various Human Rights Groups for making risk-based decisions about deployment and indeed challenging those deployments respectively. Without both necessary steps, the law, policy and technology would not have the credibility that they now have. That is not to say more can and will be done as deployments gather pace.

As I’ve mentioned, the narrative has to date been dominated by those opposed to facial recognition. This has been partly assisted by a media who still pay little attention to the benefits, the learnings now incorporated in the College of Policing Approved Police Practice (APP), or significant advances in the accuracy of the technology.

Police and their technology suppliers should not be afraid to be challenged over the issues of AI bias, proportionality or the legal use of FRT. In fact they should welcome it. This is because it presents an opportunity to examine the other side of the coin, where technology is improving, humans are making better decisions, victims are getting justice and vulnerable people are being safeguarded.

In addition to the progress made by those early adopters to force more robust policy and clarification of the law, the current Policing Minister Chris Philp has been vociferously championing the technology. However, even though the benefits are increasingly on the agenda, the argument still needs to be won in many quarters.

Agenda moving towards RFR and OIFR

What will certainly help this is the fact that the agenda is moving away from Live Facial Recognition (LFR) to Retrospective Facial Recognition (RFR) and Operator Initiated Facial Recognition (OIFR). So why is this important?

One of the main criticisms of LFR is collateral intrusion – by which an innocent member of the public walking down the street is having their biometric data scanned and used in a search. It is understandable that this will raise concerns with the public. This will no doubt be the most difficult battle.

However, these particular concerns are not relevant in RFR and OIFR since both rely on searching a single image of a specific person for a policing purpose. Yet whenever RFR and OIFR are discussed in the media, opponents are not challenged when they make the collateral intrusion argument. Similarly, I have heard the technology being discussed in a retail environment with the claim that we are now all on shop watchlists. You might well be – if you’re the sort of person who’s previously walked out with a joint of beef and forgotten to pay, but not because you’re just doing your shopping there.

Perhaps FRT might have found favour sooner if those early adopters had begun exploring RFR and OIFR with the same enthusiasm first, rather than their potentially more contentious cousin.

It’s much easier to make a case for RFR when you are seeking to identify an unknown suspect responsible for a reported crime, rather than the perception of LFR, that you are just throwing a net wide and hoping to catch something. This is especially when you are searching against legally held images of previously convicted persons.

I submit that it’s also far more palatable for the public to understand that identifying someone correctly in a face-to-face encounter using OIFR could prevent unnecessary detentions or ensure that a high risk missing person is safeguarded. It is also worth mentioning that due to the R v Bridges ruling, OIFR can be used for any policing purpose (subject to the force policy), whereas mobile biometric fingerprints are restricted by Sec 61A Pole and Criminal Evidence Act 1984 which states that you need an offence and doubt identity and therefore can be quite limited.

RFR and OIFR will be challenged more vigorously in due course when their use inevitably increases. For example, a case of an unlawful arrest in the US as a result of FRT is frequently quoted as having led to a miscarriage of justice. In those cases where this is not adhered to, we often find that it is the technology which is being blamed when in fact it has acted as expected and the user is at fault. This is why policy and training are so important. This is precisely why there should always be a human in the loop whichever form of FRT is used.

With LFR attracting the lion’s share of the coverage – and the recent announcement about greater use of RFR – you’d be forgiven for thinking that RFR is new. In fact, as many readers will know, forces have been using RFR within the Police National Database (PND) for a number  of years.

The most recent announcement with regard to OIFR being linked to national custody images is welcome. Perhaps talk of a later phase of gaining access to passport and DVLA images is less helpful at this time. I suspect the public will be largely on board with using legally held custody images to solve crime and protect vulnerable people. When the successes start to be seen as a result, then there may be scope to talk about other databases. As we have seen with LFR, if we try to run before we can walk, we lose the narrative and put ourselves back years. We may still be some way off a national OIFR becoming a reality, but even if it is delivered as early as this year, it will leave some gaps which only a local solution can fill.

ComparisonManager

Motorola Solutions has both an RFR solution, and through its Pronto mobile digital policing platform, a OIFR capability too.

The RFR capability available through PND is widely accepted as being much improved and under-used in a large number of forces, hence the Policing Minister’s recent comments. However, it lacks tools to achieve the best search results and indeed to review “Human in the loop” and report on those results.

At Motorola Solutions we are now looking to move the narrative away from “Facial Recognition” to “Image Comparison.” This is because faces are not the only identifiable images that you may wish to compare to develop investigative leads (notice here the emphasis on investigative leads rather than evidential). The reason our solution is called ComparisonManager is not just a play on words to distance ourselves from the increasingly unfashionable FRT terminology. It is because we don’t just compare faces, we also do retrospective comparison of objects and tattoos. And we’re not going to stop there either. There is much more to come.

Aside from these specialist tools in our ComparisonManager solution, we are also able to fill one of the biggest gaps that a national OIFR solution cannot. Specific groups such as high risk missing persons and specialist offender management categories can be created in local galleries and these will inevitably contain images that are not available in custody image databases.

When used as an OIFR search in Pronto, officers will not only be able to identify previously arrested individuals, but also those that the force wish to manage and protect locally too. In addition, and this is very important – an OIFR search can be carried out as a federated search against PNC, Interpol, DVLA, Voters Register and RMS. This gives the officer the maximum amount of information to make that “Human in the loop” decision.

Pre searching tools

Investigators and identification officers are constrained by the quality of the probe image available. Often this reflects the quality of the CCTV system from which it is obtained or because of the angle of the camera or distance from the subject etc. It follows that the RFR search within PND is either not considered or is unable to return a match that the identification officer is able to confirm.

For this reason Comparison Manager has a number of tools to develop a probe image to a point where the subject is likely to be returned within a batch of likely matches rather than at a high threshold which a good image will almost certainly do.

I can hear deep intakes of breath already! However, this is just the first stage of a process to get the right person returned in a batch that can be forensically examined by an identification expert – yes, a human. More than that, this is not intended to deliver an identification that will stand up in Court, just an investigative lead that can be used to develop further circumstantial and real evidence.

Think about it like a sheepdog trial. A skilled sheepdog can probably separate a specifically marked sheep and guide it through a hoop (the analogy being a high quality probe image). A young enthusiastic dog might separate half of the sheep in a group but, however, still contain the target sheep which the shepherd can then assist separating and guiding through the hoop (equating to the ComparisonManger enhancement tools) So the question is, if you were an SIO with no leads in a murder enquiry other than a really poor image, would you send for the enthusiastic young dog or would you trudge up the mountain and look for your own sheep? Anyway, it’s not about the sheep in the hoop – it’s the human in the loop that matters. (Did you see what I did there?)

These tools include:

• Eye locator (prior to face rotation to ensure accurate frontal view)
• Adjusting face to be front facing
  • Horizontal tilt
  • Lateral lit
• Eye marking
• Mirror duplication (missing half face)
• Eraser (e.g. removing a tattoo which has been duplicated)
• Full audit of changes made

If the subject being sought is in the database, they are likely to be returned at a lower threshold and may appear further down the list than other images. But this is where the “Human in the loop” really matters, using the “viewing” tools mentioned below and other sources of data to make an identification.

There are occasions when a number of people must be identified from a single video feed as part of an investigation. Think of a serious crime scene at a pub where numerous people have left prior to police arrival. ComparisonManager has a VideoExtract tool to assist here. It works by saving the best image of each face in a video clip. These can then be used individually in the RFR search by the identification officer. All images are immediately deleted once the tool is closed.

The search

Once a probe image has been identified and if necessary developed for optimal results, the solution can then perform a search. Which galleries are searched are a matter for the customer to determine, but in a police environment this will almost certainly be local custody images. As we have discussed above this may include national custody images outside of the PND in future.

However, a locally operated solution such as ComparisonManager will also enable user defined galleries such as high risk missing persons or unknown wanted suspects. This becomes even more relevant when we talk about linking OIFR later. The important thing to consider here is that you have a mechanism to ensure only legally held images are used, and not those which should have been deleted.

It is imperative that the algorithms used in any solution are both accurate and unbiased. The global standard is generally accepted to be the US National Institute of Standards and Technology (NIST) tables, where algorithms are submitted for testing on an annual basis. There are various categories, some leaning towards border force with one to one to one “visa” searches and others catering for police one to “public mugshots” searches. There are in excess of 500 algorithms on the list and being near the top really does matter. I am not going to state here which algorithm we use, but you won’t need to scroll to see it!

The search results are delivered in hierarchical order of accuracy, but that really is the end of the automated process of identification and where the human takes over. Being top of the list means nothing, other than to prioritise how the human identification officer assesses the results and makes their decision, especially if they have developed a really poor probe image prior to searching.

Post search assessment tools

Comparison Manager has a variety of tools to aid the identification officer. These include:

• Side-by-side probe and returned images as you scroll and hover over search results
• After clicking on a result the probe image can be viewed against the returned image in a variety of ways including:
  • Horizontal and vertical tilt of returned image to match probe image aspect
  • Side-by-side 3D facial with lateral and vertical tilt
  • Side-by-side wireframe with lateral and virtual tilt. This is particularly useful to clear the “white noise” of facial colourings and hair which can change with time. The identification officer can clearly compare the biometric measurements of the features
  • Side-by-side magnifier by hovering over a feature in the probe image the same feature is magnified in both images
  • Probe and returned image overlay with horizontal and vertical sliders
  • As an algorithm grid. This can then be rotated to focus on specific features for closer inspection. Other tools include a magnifier where aspects of both images can be enhanced by simply moving the cursor over one image.

This process should always be used in conjunction with other databases and information. This could include where the offending took place in relation to a subject’s associations. However MO and prison location are other examples of information that may lead to an identification.

The report

RFR in PND is often used by frontline police officers because it’s a simple “image in, image out” process. However, there are dangers to this approach where the officer simply relies on the highest hit because a hit is usually of high quality. But this is how miscarriages of justice have occurred in the past where an inexperienced officer has total reliance on the technology.

Apart from the fact that it is best practice to have a small number of highly trained specialist Identification officers, this is actually essential with ComparisonManager because of the nature of its advanced tools. At Motorola Solutions we have not only made it easier for these specialist officers to find a match by providing them with these advanced tools, but we have also simplified the process of reporting their findings.

The investigative report is partially automated. The probe image is presented alongside the subject’s gallery image selected and matched by the identification officer. If the probe image was enhanced before a search, the report will also present that alongside the other two. But crucially, it will list all of the changes made to the original image as an audit trail which the identification officer may be called upon to justify in Court.

The next section has further manual tools which enable the identification officer to callout and annotate the similarities between the original image and the selected image. Finally the identification officer can summarise other information such as data from other systems which will support the identification.

The result is that the investigating officer will receive a concise and accurate report highlighting why a subject’s identification should be considered as an investigative lead. It is the responsibility of the identification officer to make a professional judgement on the match, and the investigating officer’s decision as to what action should be taken. Contrary to popular belief – the computer does not have a say in either.

Operator Initiated Facial Recognition (OIFR)

Deploying ComparisonManager also gives a force the ability to deliver a locally managed OIFR to its frontline officers. Put simply, this is where an officer equipped with a smartphone whilst dealing with a subject face to face can take their photo and submit it for a search for the purpose of identifying them.

As we have discussed earlier, the taking of photos for a policing purpose is covered by Common Law as reinforced by R v Bridges. So OIFR can already be used in far wider circumstances than mobile fingerprint ID where you must both suspect an offence and doubt identity. However, whilst police have the power to use force to take a fingerprint of someone in those circumstances, I have never heard of anyone doing so because the optics would be somewhat embarrassing and almost certainly disproportionate in most circumstances. Therefore if someone resists a fingerprint they will probably be arrested. Taking a photo obviously requires no force at all in most circumstances even when non compliant.

It is also worth noting that this is the most accurate form of facial recognition. This is because the officer has control of the image capture and is almost certainly using a high quality camera. In most cases they will also be searching against high quality custody images.

You may remember I mentioned earlier one of the main differences between a national PND face search and the capabilities presented by ComparisonManager was the ability to control your own watchlists. This is where you are really starting to save lives and going beyond tracking down suspects for offences. High risk missing persons and other offenders being managed locally can be identified in encounters with police more easily using OIFR.

If a force has a gallery of images of unknown subjects from previously unsolved crimes, these can be dealt with at the same time as the offence for which they are being arrested for, thus creating efficiencies in the investigative process.

If you have a mobile solution such as Pronto, now used in 23 UK police forces, you can operate the OIFR search in a federated search alongside other databases. This ensures that the identification is made by the officer based on all of the information and not just the image search. Not only that: once verified with other local and national databases, the result can then be used in other workflows to reduce duplication of effort. This data could be used in an interim custody record, so that on arrival the custody officer is already aware of the risks and the time taken to book in is reduced.

Our OIFR is also available to non Pronto forces as a federated search or indeed as a silo application in its own right.

By default, officers with access to OIFR also have a mobile RFR. This is akin to a PND RFR search without the specialist tools. An example might be where an officer is presented with a Facebook image of an unknown suspect when reporting a harassment crime.

Both ComparisonManager and Pronto are able to create email and text alerts notifying OIC’s when a subject that they are managing has been searched. This happens within seconds so that interventions can be put in place if necessary covertly.

Other comparison searches

We talked earlier about ComparisonManager being the basis for other image matching. Current capabilities include a tattoo comparison and an object comparison. An example of this is where a suspicious rucksack has been left at a vulnerable location and the subject has only been captured on CCTV from behind. Here ComparisonManager can be used to match the image of the rucksack and link the subject.