If the reason you don't want to be someone's reference is that you haven't worked with them closely, you can tell them exactly that, suggesting someone else who may be better suited.
When a former employee didn't excel at their job, you can tactfully explain that you can't provide a recommendation but HR can confirm their role, and suggest seeking a reference from another department.
Collection
[
|
...
]