By: Emily M. Bender and Decca Muldowney At a recent appointment, Emily’s physical therapist (who knows some about her research) said, “Before we get started,...

At a recent appointment, Emily’s physical therapist (who knows some about her research) said, “Before we get started, there’s something I want to ask you about.” The something was an automatic “scribing” system their office is trialling for two weeks and deciding whether to purchase. These systems take in a (presumably audio-only) recording of the patient encounter and then output a draft patient note for the chart.

So what’s the big deal with “AI” charting? Here are nine reasons why we recommend refusing to consent to the use of scribing tools in healthcare settings:

  • chicken@lemmy.dbzer0.comEnglish
    11·
    2 months ago

    These systems always involve third-party software, where recordings of provider-patient interactions are sent to some other company.

    Not to dismiss the other problems with this practice, but it seems especially crazy it works this way since transcription and summarization in particular don’t need powerful hardware and they keep coming out with smaller models that can do these things. There’s no justifiable reason for any of this data to leave the doctor’s office and a lot of reasons for it not to.

    • obtoxious@lemmy.mlEnglish
      6·
      2 months ago

      I work in a place that is using these. The reason they are sending it out is that there is a very low level of comfort with tech and the outside company is “taking care of” a lot of messiness. The fact of the matter is there is no in-house capability for this, You should have seen us trying to replace the toner (or was it the drum?) in the fax machine yesterday. The first company with a “nice” UI will have market dominance. Not for technical, but for social reasons.

      • chicken@lemmy.dbzer0.comEnglish
        4·
        2 months ago

        I think at this point you could probably even do it as an offline mobile app, no extra technical competence needed to use. If it needs syncing with other devices have third party servers, but it’s end to end encrypted and all processing is clientside so they don’t actually see anything. But they want to see it, because that data is a valuable asset, even though ethically they really should not have it.

        • obtoxious@lemmy.mlEnglish
          2·
          2 months ago

          I agree but someone (?) needs to get that app out and certified and known ASAP because the ones already in the market

          If you want to give your health care provider a small panic attack, just mention the word “migration” as is “migrate to a new computer system”. Nobody wants to do it. Once something “works” (using the broadest definition of the term) it will be kept as long as possible.

    • Phoenixz@lemmy.caEnglish
      1·
      2 months ago

      Wellllll…

      I’m fully against these systems, and fully against my data leaving the doctors office, but the “summarization in particular don’t need powerful hardware” is a bit of a stretch to put it mildly, especially when you start looking at what computers most doctors have to work with. The average doctor computer that I see still uses VGA cables for crying out loud, you won’t see any AI on those machines ever.

      • chicken@lemmy.dbzer0.comEnglish
        2·
        2 months ago

        Summarization is known for being one of the things weaker models can handle competently, and there are LLMs with very low system requirements:

        239 tok/s decode on AMD CPU, 82 tok/s on mobile NPU. Runs under 1GB of memory

        I don’t know exactly what the minimum performance a model would have to have to do this specific task, but it seems very likely it could be done by something that can run on a phone, or other low powered hardware.