This study focused on developing and validating two NLP systems aimed at identifying social support (SS) and social isolation (SI) in clinical notes from psychiatric patients. The rule-based system demonstrated superior accuracy compared to the LLM approach, a result that was unexpected given the general trend of LLMs outperforming rule-based systems. The systems were portable, open-source, and tailored for precise labeling of risk factors, highlighting both the complexity of clinical text and the need for robust annotation strategies. The findings advocate for further research to enhance the performance of these models.
The study successfully established two open-source NLP systems to categorize social support and isolation in psychiatric clinical notes, affirming their potential utility in clinical settings.
Despite expectations, the rule-based system's accuracy surpassed that of LLMs in identifying social isolation and support, emphasizing the effectiveness of rule-based annotation in clinical contexts.
The divergence in performance between rule-based systems and LLMs illustrates that both methods can validly address the task, revealing different approaches to categorizing clinical data.
The findings suggest a need for further enhancement of model accuracy, particularly when training fine-tuned LLMs for specific social interaction categories in clinical notes.
Collection
[
|
...
]