At the start of the pandemic, the COVIDSafe app was designed as a tool to improve contact tracing. And a multimedia campaign promoted it as helping to find more COVID contacts faster.

Today, nearly two years after its launch in April 2020, we are publishing in The Lancet Public Health our assessment of the app’s effectiveness and usefulness in New South Wales.

We analyzed data from 619 recorded COVID cases in NSW aged over 12 and their contacts between May and November 2020. We also surveyed contact tracers to find out if they found the app useful.

Our analysis showed that the app did not add much value to the existing conventional contact tracing system.

A total of 22% of cases used the app. Most (61%) of the contacts registered by the app as “close contacts” turned out not to be epidemiologically linked to a case. The app only detected 15% of true close contacts identified by conventional contact tracing.

In total, COVIDSafe detected only 17 additional true close contacts in New South Wales during the six-month evaluation period.

This has resulted in substantial extra work for contact tracers and overall has not made a significant contribution to the COVID response in NSW.

Read more: From COVID control to chaos – what now for Australia? Two paths are open to us

Tracing takes time

Interviewing new cases to identify, trace and isolate their close contacts is a critical public health activity. But that takes time, and contact tracers can be quickly overwhelmed when the number of cases increases.

Digital contact tracing apps were enthusiastically adopted at the start of the COVID pandemic in many countries, including Australia.

Like many other tracing apps, COVIDSafe uses smartphones’ built-in Bluetooth feature to exchange signals between phones.

The duration, frequency and transmission strength of these “digital handshakes” are used to determine whether two smartphone users have come into “close contact” with each other.

Poor ability to correctly detect and identify close contacts

During our study period, less than a quarter of all cases used the app. That’s half the proportion of the general Australian public who have done so.

Of the cases that used the app, many did not have a single contact recorded by the app. For others, contact tracers were unable to access app data.

Contact tracers could use app data for 32 (5%) of 619 cases during our study period. Of these cases, only 79 (39%) of the 205 contacts recorded by the app as “close contacts” could be verified as true close contacts. This suggests a low positive predictive value of the app.

Here are examples of fake close contacts recorded by the app:

• neighbors in different apartments in apartment buildings

• office workers in adjoining rooms

• customers from neighboring restaurants

• people waiting in separate cars at drive-through COVID testing clinics.

The vast majority (85%) of close contacts identified by conventional contact tracing were not detected by the app, indicating low sensitivity.

Some additional contacts detected

In the six months of our study, there were only 17 true close contacts identified by COVIDSafe who would otherwise have been missed – a tiny fraction of the more than 25,300 close contacts detected and tracked through conventional contact tracing in New South Wales during the same period.

None of those 17 contacts have tested positive. COVIDSafe therefore did not contribute to preventing new exposures in New South Wales during our evaluation period.

Read more: Free rapid antigen tests make economic sense for governments, our analysis shows

Technological defects

Contact tracers didn’t find the app easy to use. Some said the app doesn’t seem to work as reliably on all types of phones. The number of contacts on iPhones was grossly underestimated, while those on Android phones were overestimated.

Contact tracers have also noted the app’s apparent inability to properly save contacts unless it’s open. This could explain the large number of cases without any contact recorded in our study.

The process of matching close contacts identified by the app with those identified during case interviews was considered time-consuming, especially since most of the contacts detected by the app were not actually close contacts.

Staff interviewed said it could easily overwhelm the contact tracing system if the number of cases had been higher, paradoxically reducing the usefulness of the app when it was needed most.

Overall, contact tracers’ perceptions of the app ranged from “not having much impact” to being an extra step that increased the workload without adding much value.

Little added value at high costs

In our study, COVIDSafe did not make a significant contribution to the COVID response in New South Wales in 2020. Instead, the app created a high workload without any clear benefit.

This compares to the AU$7.7 million it cost to develop and operate COVIDSafe until the end of April 2021, with estimated maintenance between $60,000 and $75,000 per month since.

The arrival of the highly transmissible variant of Omicron at the end of 2021 in Australia, coinciding with the lifting of most public health restrictions, has led to a massive increase in the number of cases, forcing a series of major adjustments in the search for contacts.

Read more: Australia has all but ditched the COVIDSafe app in favor of QR codes (so be sure to check in)

In most parts of Australia, contact tracing now plays a very limited role in controlling COVID. Nevertheless, it is likely to remain a key public health intervention for infectious diseases in the future.

For digital contact tracing apps to be effective and useful, it will be important to involve contact tracers when designing the system, testing the underlying technology under real-world conditions, and evaluating the app regularly. after deployment.

Dr Anthea Katelaris, a public health physician who worked at the Western Sydney Local Health District Public Health Unit at the time of the study, co-authored the research referenced in this article.