Value(s) | Describe the risk | Possible consequences | Priority |
Transparency | The robot’s output results are not presented in a manner that is easily understandable by elderly users or their caregivers. | Users and caregivers may feel confused or frustrated, leading to dissatisfaction; employees might leave due to increased support demands; customers may opt for competitors; potential for discrimination lawsuits if outputs are not clearly explained; difficulty in identifying and correcting errors. | High |
Trust, transparency | The subjective nature of categorizing output based on probability scores makes it challenging for users to understand or correct issues. | Loss of trust from families and caregivers, leading to customer attrition; increased difficulty for customer service representatives (CSRs) who cannot adequately explain the system’s decisions; potential legal disputes over perceived discrimination. | High |
Fairness | Input data may be biased towards caregivers willing to travel to suburbs or other specific areas. | Perceived unfairness among caregivers, damaging company reputation; potential discrimination against less affluent caregivers who may not have access to transportation. | Med-High |
Use of demographic data such as location, gender, race, and age could result in discriminatory practices. | Significant legal risks; need to eliminate demographic biases from the system. | Very High | |
Different stakeholders, including families, caregivers, and data scientists, have varying definitions of fairness. | Internal conflicts and confusion, negatively impacting company morale and operations. | Low-Med | |
CSRs’ personal biases in matching caregivers with families could influence the system’s recommendations over time. | Risk of biased outcomes, causing caregiver dissatisfaction, harming the company’s reputation, and potential legal actions. | High | |
The system’s reliance on English-language bios might exclude non-English speaking caregivers. | Non-English speaking caregivers and families might feel discriminated against, limiting the company’s ability to serve diverse communities effectively. | Med | |
Privacy | Caregivers can opt out of sharing their data, but doing so might disadvantage them compared to those who share data. | Caregivers can opt out of sharing their data, but doing so might disadvantage them compared to those who share data. | Med-High |
Autonomy and job security | Increased automation may threaten the job security of CSRs without proper planning and transition strategies. | Lower employee morale; proactive job-seeking by employees; loss of trust and dissatisfaction among customers and caregivers who value their relationship with CSRs. | Med |