JemPI can be installed in two ways:

• Standalone Installation: This is useful for development or custom implementations

• Jembi Platform Installation: This is the recommended way to install JemPI, it uses the Jembi Platform deployment tooling which allows JemPI to be installed with a single command. It also includes other useful software like the OpenHIM to authenticate API and route API request.

JeMPI is an entity matching (commonly used for patient matching) and linking technology that supports batch and transactional matching via advanced techniques though a combination of deterministic and probabilistic matching.

Overview

The Jembi MPI, also known as JeMPI, is a standards-based client registry (CR) or master patient index (MPI).

JeMPI facilitates the exchange of patient information between different systems and holds patient identifiers that may include patient demographic information. This is a necessary tool for public health to help manage patients, monitor outcomes, and conduct case-based surveillance.

JeMPI’s primary goal is to act as a tool in order to solve the issue of multiple or duplicated patient records that are submitted from multiple point of service systems such as electronic medical records, lab systems, radiology systems and other health information systems.

This is achieved by matching the various patient records from different systems under a Master Patient record with a unique ID. This allows for downstream applications, such as surveillance, to accurately display data and information on patient records without the worry that the data contains multiple records for the same patient.

The JeMPI Client Registry is a system that incorporates a microservice architecture, each microservice has a specific task such as data cleaning, data storing, etc. These various services communicate through a Kafka message bus, meaning that every service is storing and retrieving data from a specific Kafka topic.

Below the synchronous and asynchronous flow diagram.

Asynchronous flow

JeMPI_AsyncReceiver

Description: A microservice that sends the content of an uploaded csv file to the JeMPI_ETL service. the JeMPI_AsyncReciever service produces kafka messages where each message has a row from the CSV file uploaded. it will then be saved under a kafka topic.

The base version uses a reference implementation with the fields below:

String Fields

String auxId, String givenName, String familyName, String gender, String dob, String city, String phoneNumber, String nationalId

Input

1. A CSV file located in the JeMPI_AsyncReciever associated volume, under */app/csv_ directory (this can be done through HTTP request). Example of input file:

ID,Given_Name,Family_Name,Gender_at_Birth,Date_of_Birth,City,Phone_Number,National_ID
rec-00000000-aaa-0,Endalekachew,Onyango,male,20171114,Nairobi,091-749-4674,198804042874913
rec-00000001-aaa-0,Fikadu,Mwendwa,male,19840626,Nairobi,022-460-8846,199403050409528
rec-00000002-bbb-0,Biniyam,Maalim,male,20191022,Nairobi,098-119-7244,200006231841948

Output

The service will save the data from the CSV file, one line at a time. Kafka topic: TOPIC_INTERACTION_ETL="JeMPI-interactions-etl"

JeMPI_ETL

Description: A microservice that pocesses the input coming from the JeMPI_AsyncReceiver. The JeMPI_ETL service will perform some data trasformation e.g. lower case the values for name of the patient or unformat the date for the date of birth. The resulting data will be sent as JSON (JSON Streaming) to the JeMPI_Controller service.

Input:

Data coming from the JeMPI_AsyncReciever service. Kafka topic: _TOPIC_INTERACTION_ETL="JeMPI-interaction-etl"*

Output:

Data transformed into JSON that will be sent to the JeMPI_Controller. It will be stored in the Kafka topic: _TOPIC_INTERACTION_CONTROLLER="JeMPI-interaction-controller"*

Example or a Kafka message coming from the interaction controller topic:

{
  "contentType": "BATCH_INTERACTION",
  "tag": "import-5334297603633827819uploadConfig",
  "stan": "2024/08/07 08:24:07:0000001",
  "interaction": {
    "sourceId": {
      "facility": "FA2",
      "patient": "patient_id"
    },
    "uniqueInteractionData": {
      "auxDateCreated": "2024-08-07T08:24:08.174750419",
      "auxUserFields": [
        {
          "scTag": "aux_id",
          "tag": "auxId",
          "value": "rec-0000000708-02"
        },
        {
          "scTag": "aux_clinical_data",
          "tag": "auxClinicalData",
          "value": "RANDOM DATA(865)"
        }
      ]
    },
    "demographicData": {
      "fields": [
        {
          "tag": "given_name",
          "value": "patricia"
        },
        {
          "tag": "family_name",
          "value": "solis"
        },
        {
          "tag": "gender",
          "value": "female"
        },
        {
          "tag": "dob",
          "value": "19821106"
        },
        {
          "tag": "city",
          "value": "chicago"
        },
        {
          "tag": "phone_number",
          "value": "0133705553"
        },
        {
          "tag": "national_id",
          "value": "198211065001099"
        }
      ]
    }
  },
  "sessionMetadata": {
    "commonMetaData": {
      "stan": "2024/08/07 08:24:07:0000001",
      "uploadConfig": {
        "reportingRequired": false,
        "uploadWorkflow": 0,
        "minThreshold": 0.65,
        "linkThreshold": 0.7,
        "maxThreshold": 0.75,
        "marginWindowSize": 0.1
      }
    },
    "uiMetadata": {
      "timeStamp": null
    },
    "asyncReceiverMetadata": {
      "timeStamp": "2024/08/07 08:24:08"
    },
    "etlMetadata": {
      "timeStamp": "2024/08/07 08:27:05"
    },
    "controllerMetadata": {
      "timeStamp": null
    },
    "linkerMetadata": {
      "timeStamp": null
    }
  }
}

JeMPI_Controller

Description: The JeMPI_Controller service has multiple tasks:

• Send the data coming from the JeMPI_ETL to either the JeMPI_Linker or the JeMPI_EM services, based on the workflow selection made by user on import. The data will be stored in their respective Kafka topics accessed (consumed) by those service.

Input: Data coming from the JeMPI_ETL service Kafka topic: _TOPIC_INTERACTION_CONTROLLER="JeMPI-interaction-controller"*.

Values of the M & U computed in the JeMPI_EM service Kafka topic: _TOPIC_MU_CONTROLLER="JeMPI-mu-controller"*

Output:

1. Send the data to the JeMPI_EM

• Kafka topic: TOPIC_INTERACTION_EM="JeMPI-interaction-em"

• MU process: Kafka topic: TOPIC_MU_LINKER="JeMPI-mu-linker"

1. Send the data to the JeMPI_Linker

• Kafka topic: TOPIC_INTERACTION_LINKER="JeMPI-interaction-linker"

JeMPI_EM

Description: A microservice that will create an object containing m&u of a patient against patient records that go into the EM algorithm (quality (m) and the uniqueness (u) per field). This object is used in the linker for matching patients. It uses a machine learning called Estimation maximisation (EM) algorithm to optimize that value, it is launched after receiving a number of records specified in the configuration.

Input: Kafka topic: TOPIC_INTERACTION_LINKER="JeMPI-interaction-linker"

Output: Kafka topic: TOPIC_MU_CONTROLLER="JeMPI-mu-controller"

JeMPI_Linker

Description: A microservice that will interact with Dgraph database to do the matching of the patients. The Linker uses thresholds to drive the linking and notifications for review processes. These thresholds are the following:

• A single match or no match threshold : the interaction will automatically be linked to the highest golden record candidate above the threshold. If no candidate has a score above the threshold, a new golden record is created. This is typically used for fully autonomous linking.

• Window around the match/no match threshold : if the highest score generated for the candidates falls within this window, a notification is sent for Admin to review the interaction.

• Margin threshold : if another candidate falls within a margin from the highest score and this highest score is above the match threshold, a notification for review is sent for the Admin to review the linked interaction.

Input: Kafka topic: TOPIC_INTERACTION_EM="JeMPI-interaction-em"\

Output:

• Interact with the Dgraph database using GraphQL queries/mutations, save the interactions and the links.

• Send response of either the link info or the list of candidates to the Controller

• Save response to Kafka topic: TOPIC_notifications=”JeMPI_notifications”

JeMPI_Dgraph

Description: The Dgraph database used for JeMPI to store the patient records. it is a graph database.

Component linked:

• Dgraph Ratel: A tool for data visualization and cluster management. Ratel can be used with Dgraph to manage cluster settings, run DQL queries and mutations and see results of the mentioned operations.

• Dgraph Alpha: Expose and host endpoints of the indexes.

• Dgraph Zero: it is like a Zookeeper/KRaft in Kafka, it will control the instances of Alpha by assigning them to a group, and re-balances the data between them.

JeMPI_Kafka

Description: Kafka the message queue bus, it contains all the topics used previously in the other components.

JeMPI_API

Description: The JeMPI_API service contains the endpoints needed to interact with JeMPI.

It performs the following functions:

• Read data from the Kafka topic TOPIC_notifications=”JeMPI_notifications”

• Save data related to the administration in PostgeSQL DB

• Get the data from PostgreSQL when the JeMPI Web requests data.

Prerequisites

In the following, we will introduce the software prerequisites to be able to run the client registry jempi on your machine using the platform.

Docker

Please refer to the in order to install docker on your machine. It is best to follow the post installation process so that you grant docker sudo access.

[for Windows users] WSL2

Installing WSL2 is required to be able to develop and test the project. It is recommended to limit the memory usage of WSL2.

Local Setup

1. Create /tmp/logs directory

1. Create the docker platform image

1. Initialise Docker Swarm

1. Run 'go cli' binary to launch the project

1. Launch the client registry jempi package profile a. all packages and profiles are configured in the ./config.yaml field b. updates to environment variable can be made in the profile env file ie: mpi.env

1. Access : http://localhost:3033/login

1. Sign in with Keycloak user credentials

Prerequisites

In the following, we will introduce the software prerequisites to be able to run the JeMPI client registry on your machine.

Docker

Please refer to the in order to install docker on your machine. It is best to follow the post installation process so that you grant docker sudo access.

[for Windows users] WSL2

Installing WSL2 is required to be able to develop and test the project. It is recommended to limit the memory usage of WSL2.

Build Utilities

Follow the steps to install sdk , check if you already have it by running sdk. To install it, run the two following commands:

To check the installation, you can check the version by running: sdk version

We should install the following:

• Maven: Command: sdk install maven Version: 3.9.8

• Scala Build Tool: Command: sdk install sbt

• Java: Command: sdk install java 21.0.3-tem Version: Temerin 21.0.3-tem (See list by running: sdk list java)

Note: when installing with a non-root user set java directory

Check the version of java by running: java --version. We should get: Temurin-21.0.3+9.

Starting JeMPI

In the following section, we will discuss the steps for running JeMPI on you machine, start by cloning the JeMPI repository on your machine and navigate to JeMPI by running the following command in you terminal of choice

#1 Semi-Automated - Local Setup

Overview

This Bash script is designed for deploying JeMPI locally with various options. It performs tasks such as installing Docker, SDKMAN, Java, Maven, and SBT, setting up the environment configuration, creating a Docker registry, pulling and pushing Docker images, initializing the Docker Swarm, building the entire stack, rebooting, restarting, tearing down, Backup & Restore Databases and destroying JeMPI.

Usage

Location of file - JeMPI/devops/linux/docker/deployment File Name - local-deployment.sh

Set following variables JAVA_VERSION=21.0.3-tem JEMPI_ENV_CONFIGURATION=create-env-linux-low-1.sh

Installation Process

This script must be run from the following path and will not work if executed from a different location

Location of file - JeMPI/devops/linux/docker/deployment

Option 1: Deploy JeMPI (For Fresh Start) This Option used to install JeMPI from Scratch or Fresh setup

• Set up hostname and IP address in the Hosts file.

• Docker Swarm Initialization.

• Creates a Docker registry, pulls Docker images from the hub, and pushes them to the local registry.

• Builds and reboots the entire JeMPI stack

Option 2: Build and Reboot

• Builds and reboots the entire JeMPI stack.

Option 3: Restart JeMPI

• Reboots the entire JeMPI stack

Option 4: Down JeMPI

• Stop entire stack

Option 5: Backup Postgres & Dgraph

• Postgres backup process creates a folder with a timestamp, and inside it, SQL files are generated for each postgres database.

• Backup Directory: JeMPI/devops/linux/docker/docker_data/data/backups/postgres

• Dgraph backup process creates a folder with a timestamp, and inside it generates the Json file of data.

• Backup Directory: JeMPI/devops/linux/docker/docker_data/data/backups/dgraph

Option 6: Restore Postgres & Dgraph

• Users need to confirm with “ctr + Y” for restore.

• This process will wipe all existing data from both Postgres and Draph DB’s and restore new from backup.

• Users need to enter the folder name of the backup directory to initiate the restore process.

Option 7: Re-Deploy JeMPI

• Updates environment configuration settings

• Update HAProxy settings

• Pulls Docker images from the hub, and pushes them to the local registry.

• Builds and reboots the entire JeMPI stack

Option 8: Install Prerequisites

• Install SDKMAN - SDK Manager

• Install Docker

• Install Java, Maven, and SBT using SDKMAN

Option 9: Destroy JeMPI (This process will wipe all data)

• This process will remove all stack from swarm and leave the swarm.

• Remove all data and volumes

Notes

• The script prompts for user input to select an option.

• Confirmations are requested for critical actions.

• Use Ctrl+Y for "Yes" confirmation to Destroy all systems and Restore DB.

• Customize the script as needed for your specific deployment requirements.

#2 Manual - Local Setup

Setup an IP address Before starting the process of running JeMPI, you will need to setup an IP address for your machine.

On your terminal of choice, run the ip a command and retrieve the ip address from your wi-fi or ethernet interfaces

In our case, we are using the enp0s3 interface, the IP address that we will need is 192.168.1.137.

Next, you will need to set up the hostname for your machine. to do so, run the command bellow, it will open the hosts file under /etc/ directory using the nano text editor (you can use any other editor e.g. VIM, VI, Emacs, Helix, etc.) :

Keep the localhost IP and Comment any other IP address, follow the screenshot bellow.

Initialize the environment variables In the JeMPI directory, navigate to: docker/conf/env/ directory.

if you have less than 32Gbs of ram run the ./create-env-linux-low-1.sh. If you have 32Gb or more, run the ./create-env-linux-high-1.sh. both those script will create conf.env file that we will need.

Note: for server installations, manually set SERVER_IP for environment variable before executing the script

Pull the latest images

Pull the latest image versions form docker hub using the a-images-1-pull-from-hub.sh

Make sure you have a clean docker swarm

It is fine to keep the images, you can either remove all the services, containers, volumes, configs and secrets.

Run the b-swarm-2-leave.sh to leave your current swarm

After running the previous script, initialize a new swarm by running the b-swarm-1-init-node1.sh script locacted in the JeMPI/docker/ directory.

Add the ability to use local registries

Now, we need to tell docker that it is okay to run on the local registry because it is http and not https.

Go to devops/linux/docker/helper/scripts.

Run ./x-swarm-a-set-insecure-registries.sh (you need to grant it executable access first by running: chmod +x ./x-swarm-o-set-insecure-registries.sh), it will edit the file /etc/docker/daemon.json and will restart docker to make changes take effect.

NB: The script will edit the access grants of the /etc/docker/daemon.json file.

Create a local registry Now that you can use local Docker registries, run the c-registry-1-create.sh script to create a registry service. This service will host the docker images that we will use in our stack.

Push the images to the local registry We will need to pull images from docker hub then push them to the local registry :

Run the stack

After pushing the images into the local registry, we are ready to run the app, we have several options, we can run the whole stack (UI + Backend) by running the d-stack-1-build-all-reboot.sh, Or run each of the backend (d-stack-1-build-java-reboot.sh) and the UI (d-stack-1-build-ui-reboot) seperatly.

Other scripts

• d-stack-2-build-java.sh: This script will build and push the backend services to the local docker registry.

• d-stack-3-down.sh: This script will remove all services from the stack.

• d-stack-3-reboot.sh: This script will only remove everything and start again.

That's it 🚀

Testing

Check the deployment sanity

To check for running containers you can run: docker container ls To check for running services you can run: docker service ls To list all the containers you can run: docker ps -a

Or you can go to devops/linux/docker/helper/scripts and run d-stack-ps.sh and it will run: docker stack ps <NAME_STACK>

Example of the stack when running with local docker registry: (docker stack ps jempi)

Example of the stack when running with docker hub: (docker service ls)

Stop or remove the stack

To remove everything in the swarm: you can go to devops/linux/docker/ and run b-swarm-2-leave.sh

To shut down the stack: you can go to devops/linux/docker/ and run: d-stack-3-down.sh

We encourage any contributions and suggestions!

We use Discord as our preferred communications and support platform for the JemPI. We subscribe to and are bound by the OpenHIE Community Guidelines (https://discourse.ohie.org/faq). If you have a question about the JemPI or would like to get involved, please join the conversation on Discord in the #jempi channel.

For developers looking to log bugs and/or feature requests, you can also add an issue on Github, or submit pull requests for changes that you'd like to see.

Next Release:

• UI/UX Enhancements

• Performance improvements e.g. vector search

• Role-based Access Control

Future Goals:

• AI-driven matching algorithm selection

• Aim to increase performance by an order of magnitude by exploring new technology options.

• Enhance UX to make JeMPI the easiest MPI to install, use and maintain.

• DevOps UI to further simplify and accelerate deployment and maintenance tasks

GET /fieldsConfiguration

The following endpoint returns the fields configuration needed by the frontend (JeMPI-UI) in order to properly display interactions data according to a specific implementation. This endpoint returns a JSON array. Below a sample of the response :

[
  {
    "fieldName": "type",
    "accessLevel": [],
    "fieldLabel": "Record Type",
    "scope": [
      "/patient/:uid",
      "/match-details",
      "/patient/:uid/linked-records"
    ],
    "groups": [
      "demographics",
      "linked_records"
    ],
    "fieldType": "String",
    "validation": {
      "regex":"/(Golden|Interaction)/",
      "required": true,
      "onErrorMessage": "This field cannot be empty"
    }
  },
  ...
]

For each field we have a set of attributes, as defined below :

The fields should be configured manually in the config json file : devops/linux/docker/data-config/config-reference-link-dp-api.json

There's two type of fields :

• Custom fields : Indexed by the key "fields", contains all the fields that are specific to the implementation. Examples : givenName, nationalId, ...

• System fields : Indexed by the key "systemFields", it contains all the fields that are readonly fields and do not change across the implementation. Example : uid, record type, score, ...

! IMPORTANT : The fieldName in config-reference-link-dp-api.json should be set in snake-case, but it's returned in camel-case by the API.

POST /notifications

Below is a sample of the body you are to send :

{
    "limit": 25,
    "offset":10 ,
    "startDate":"2024-04-15T23:59:59" ,
    "endDate": "2024-04-15T23:59:59",
    "states": ["OPEN"]
}

The following endpoint returns notifications. notifications are used to inform the user about potential interaction linking to golden records and are generated when a certain case is triggerd. the response contains parameters (count, skippedRecords ) that are useful for pagination.

Below are the necessary parameters to get the notifications list

Below is a sample of the response

{
  "count": 10,
  "skippedRecords": 773,
  "records": [
    {
      "score": 0.6156526,
      "candidates": [
        {
          "score": 0.6156526,
          "golden_id": "0x2a"
        }
      ],
      "golden_id": "0x2a",
      "names": "katanewa",
      "patient_id": "0x2b",
      "created": "2023-08-31",
      "id": "a448a79e-09c1-4e84-b7ed-0d0c3c4d3f61",
      "state": "New",
      "type": "THRESHOLD"
    },
    {
      "score": 1.0,
      "candidates": [
        {
          "score": 0.556004,
          "golden_id": "0xb"
        }
      ],
      "golden_id": "0xd",
      "names": "zuli,ezther",
      "patient_id": "0xc",
      "created": "2023-08-31",
      "id": "3b0f6dcf-428b-4c07-9b61-48220633569f",
      "state": "New",
      "type": "THRESHOLD"
    }
  ]
}

POST /interaction

The following endpoint returns an interaction given a uid is supplied. This endpoint returns a object.

Below is a sample of the body:

{
    "uid":"0x4"
}

Below is a sample of the response:

{
  "uid": "0x4",
  "sourceId": {
    "uid": "0x3",
    "facility": "MA2",
    "patient": "198702145001118"
  },
  "uniqueInteractionData": {
    "auxDateCreated": "2023-08-31T13:44:44.576006034",
    "auxId": "rec-0000000001--0",
    "auxClinicalData": "RANDOM DATA(645)"
  },
  "demographicData": {
    "givenName": "gift",
    "familyName": "zulu",
    "gender": "male",
    "dob": "19870214",
    "city": "lusaka",
    "phoneNumber": "0100563200",
    "nationalId": "198702145001118"
  }
}

POST /expandedGoldenRecord

Given a supplied uid, The following endpoint returns an expanded golden records=, meaning a golden record with interactions linked to it. This endpoint returns an object.

Below is a sample of the body you are to send :

{
    "gid":"0x2b20"
}

Below a sample of the response :

{
  "goldenRecord": {
    "uid": "0x2e",
    "sourceId": [
      {
        "uid": "0x31",
        "facility": "MA1",
        "patient": "199508125001143"
      }
    ],
    "uniqueGoldenRecordData": {
      "auxDateCreated": "2023-08-31T13:44:47.655570503",
      "auxAutoUpdateEnabled": true,
      "auxId": "rec-0000000005--4"
    },
    "demographicData": {
      "givenName": "katanekwa",
      "familyName": "simukungwe",
      "gender": "male",
      "dob": "1995o812",
      "city": "kitwe",
      "phoneNumber": "0112302557",
      "nationalId": "199508125001143"
    }
  },
  "interactionsWithScore": [
    {
      "interaction": {
        "uid": "0x32",
        "sourceId": {
          "uid": "0x31",
          "facility": "MA1",
          "patient": "199508125001143"
        },
        "uniqueInteractionData": {
          "auxDateCreated": "2023-08-31T13:44:44.653638164",
          "auxId": "rec-0000000005--6",
          "auxClinicalData": "RANDOM DATA(998)"
        },
        "demographicData": {
          "givenName": "katanekwa",
          "familyName": "simukungwe",
          "gender": "male",
          "dob": "19950812",
          "city": "kitwe",
          "phoneNumber": "0112302557",
          "nationalId": "199508125001143"
        }
      },
      "score": 1.0
    },
    {
      "interaction": {
        "uid": "0x33",
        "sourceId": {
          "uid": "0x31",
          "facility": "MA1",
          "patient": "199508125001143"
        },
        "uniqueInteractionData": {
          "auxDateCreated": "2023-08-31T13:44:44.657099612",
          "auxId": "rec-0000000005--7",
          "auxClinicalData": "RANDOM DATA(779)"
        },
        "demographicData": {
          "givenName": "katanekwa",
          "familyName": "simukungwe",
          "gender": "male",
          "dob": "",
          "city": "kitwie",
          "phoneNumber": "011230z557",
          "nationalId": "199508125001143"
        }
      },
      "score": 1.0
    }
  ]
}

POST /expandedGoldenRecords

The following endpoint will return a list of expanded golden records given a list of golden Ids (GIDS_LIST).

Below is a sample of the body you are to send :

{
    "uidList": [
        "0x4d03",
        "0x3a07",
        "0x2b20"
    ]
}

Below is a sample of the response:

[
  {
    "goldenRecord": {
      "uid": "0x4d03",
      "sourceId": [
        {
          "uid": "0x4d01",
          "facility": "MA3",
          "patient": "198304285002129"
        },
        {
          "uid": "0x4d08",
          "facility": "MA1",
          "patient": "198304285002129"
        },
        {
          "uid": "0x4d0a",
          "facility": "MA4",
          "patient": "198304285002129"
        }
      ],
      "uniqueGoldenRecordData": {
        "auxDateCreated": "2023-09-01T07:28:06.846286719",
        "auxAutoUpdateEnabled": true,
        "auxId": "rec-0000000997--0"
      },
      "demographicData": {
        "givenName": "davison",
        "familyName": "nyambe",
        "gender": "male",
        "dob": "19830428",
        "city": "mbala",
        "phoneNumber": "0332675512",
        "nationalId": "198304285002129"
      }
    },
    "interactionsWithScore": [
      {
        "interaction": {
          "uid": "0x4d02",
          "sourceId": {
            "uid": "0x4d01",
            "facility": "MA3",
            "patient": "198304285002129"
          },
          "uniqueInteractionData": {
            "auxDateCreated": "2023-09-01T07:26:04.152254576",
            "auxId": "rec-0000000997--0",
            "auxClinicalData": "RANDOM DATA(693)"
          },
          "demographicData": {
            "givenName": "davxison",
            "familyName": "nyambe",
            "gender": "",
            "dob": "19820428",
            "city": "mbala",
            "phoneNumber": "",
            "nationalId": "1983o4285002129"
          }
        },
        "score": 0.631448
      }
    ]
  }
]

GET /gidsAll

The following endpoint returns a list of the saved (created) Golden record Ids.

{
  "records": [
    "0x2715",
    "0x2718",
    "0x271b",
    "0x271d",
    "0x2726",
    "0x2729",
    "0x272c",
    "0x2733",
    "0x2737",
    "0x273a",
    "0x273e",
    "0x2748",
    "0x274f",
    "0x2752",
    "0x2754",
    "0x275c",
    "0x2761"
  ]
}

POST /gidsPaged

The following endpoint returns a list of gids paginated, given parameters OFFSET and a LENGTH.

Below is a sample of the body you are to send :

{
    "offset":0,
    "length":10
}

Below is a sample of the response:

{
  "goldenIds": [
    "0x2a",
    "0x27",
    "0x23",
    "0x1c",
    "0x19",
    "0x16",
    "0xd",
    "0xb",
    "0x8",
    "0x5"
  ]
}

POST /goldenRecordAuditTrail

The Following endpoint returns the audit trail for a given Golden Record with a Golden_Id GOLDEN_ID

Below is a sample of the body to send :

{
    "gid":"0x2b20"
}

Below a sample of the response :

{
  "entries": [
    {
      "inserted_at": "2023-09-04 08:37:02.000665",
      "created_at": "2023-09-04 08:37:02.000657",
      "interaction_id": "0x4c",
      "golden_id": "0x4d",
      "entry": "Interaction -> New GoldenRecord (1.000000)"
    },
    {
      "inserted_at": "2023-09-04 08:37:02.000732",
      "created_at": "2023-09-04 08:37:02.000725",
      "interaction_id": "0x4f",
      "golden_id": "0x4d",
      "entry": "Interaction -> Existing GoldenRecord (0.72150)"
    }
  ]
}

POST /interactionAuditTrail

The Following endpoint returns the audit trail for a given Interaction with Interaction Id INTERACTION_ID

Below a sample of the request body:

{
    "uid":"0x2b1f"
}

Below a sample of the response:

{
  "entries": [
    {
      "inserted_at": "2023-09-04 08:37:02.000665",
      "created_at": "2023-09-04 08:37:02.000657",
      "interaction_id": "0x4c",
      "golden_id": "0x4d",
      "entry": "Interaction -> New GoldenRecord (1.000000)"
    }
  ]
}

GET /countGoldenRecords

The following endpoint returns the golden record count available in the database

{
  "count": 1911
}

GET /countInteractions

The following endpoint returns the interaction count available in the database

{
  "count": 5035
}

GET /countRecords

The following endpoint returns the record count available in the database. bellow is a sample of the response

{
  "goldenRecords": 1911,
  "interactions": 5035
}

POST /notificationRequest

The following endpoint update the notification state given a notification Id and a state. Below a sample of the request:

{
  "notificationId": "53512711-aca5-4bfb-a239-51e438d74d36",
  "state": "Pending"
}

POST /search/(golden|patient)

The following endpoint is used for the simple search either for golden or interactions.

Below a sample of the request body :

{
  "parameters": [
    {
      "fieldName": "givenName",
      "value": "John",
      "distance": 0
    },
    {
      "fieldName": "familyName",
      "value": "Doe",
      "distance": 0
    },
    {
      "fieldName": "dob",
      "value": "19870423",
      "distance": 0
    },
    {
      "fieldName": "nationalId",
      "value": "198804042874913",
      "distance": 0
    }
  ],
  "sortBy": "givenName",
  "sortAsc": true,
  "offset": 0,
  "limit": 10
}

POST /search/golden

When the request is sent to the url /search/golden the response payload will contain the list of golden records along with the linked records and the result set total (useful for pagination) :

{
  "data": [
    {
      "goldenRecord": {
        "uid": "0x1019",
        "sourceId": [
          {
            "uid": "0x1017",
            "facility": "FA3",
            "patient": "198704115001082"
          }
        ],
        "uniqueGoldenRecordData": {
          "auxDateCreated": "2023-09-04T14:39:52.258643516",
          "auxAutoUpdateEnabled": true,
          "auxId": "rec-0000000425--0"
        },
        "demographicData": {
          "givenName": "jan",
          "familyName": "daka",
          "gender": "",
          "dob": "19879411",
          "city": "mongu",
          "phoneNumber": "",
          "nationalId": "898704115001082"
        }
      },
      "interactionsWithScore": [
        {
          "interaction": {
            "uid": "0x1018",
            "sourceId": {
              "uid": "0x1017",
              "facility": "FA3",
              "patient": "198704115001082"
            },
            "uniqueInteractionData": {
              "auxDateCreated": "2023-09-04T14:38:48.901033856",
              "auxId": "rec-0000000425--0",
              "auxClinicalData": "RANDOM DATA(290)"
            },
            "demographicData": {
              "givenName": "jan",
              "familyName": "daka",
              "gender": "",
              "dob": "19879411",
              "city": "mongu",
              "phoneNumber": "",
              "nationalId": "898704115001082"
            }
          },
          "score": 1.0
        }
      ]
    },
    {
      "goldenRecord": {
        "uid": "0x1220",
        "sourceId": [
          {
            "uid": "0x121e",
            "facility": "FA5",
            "patient": "198308125001080"
          }
        ],
        "uniqueGoldenRecordData": {
          "auxDateCreated": "2023-09-04T14:39:59.44475916",
          "auxAutoUpdateEnabled": true,
          "auxId": "rec-0000000480--2"
        },
        "demographicData": {
          "givenName": "jan",
          "familyName": "dakq",
          "gender": "",
          "dob": "",
          "city": "",
          "phoneNumber": "",
          "nationalId": "1983081q5001080"
        }
      },
      "interactionsWithScore": [
        {
          "interaction": {
            "uid": "0x121f",
            "sourceId": {
              "uid": "0x121e",
              "facility": "FA5",
              "patient": "198308125001080"
            },
            "uniqueInteractionData": {
              "auxDateCreated": "2023-09-04T14:38:49.755542027",
              "auxId": "rec-0000000480--2",
              "auxClinicalData": "RANDOM DATA(424)"
            },
            "demographicData": {
              "givenName": "jan",
              "familyName": "dakq",
              "gender": "",
              "dob": "",
              "city": "",
              "phoneNumber": "",
              "nationalId": "1983081q5001080"
            }
          },
          "score": 1.0
        }
      ]
    }
  ],
  "pagination": {
    "total": 7
  }
}

POST /search/patient

When the request is performed against the url /search/patient the response payload will contain the list of interactions along with the search result total count :

{
  "data": [
    {
      "uid": "0x1018",
      "sourceId": {
        "uid": "0x1017",
        "facility": "FA3",
        "patient": "198704115001082"
      },
      "uniqueInteractionData": {
        "auxDateCreated": "2023-09-04T14:38:48.901033856",
        "auxId": "rec-0000000425--0",
        "auxClinicalData": "RANDOM DATA(290)"
      },
      "demographicData": {
        "givenName": "jan",
        "familyName": "daka",
        "gender": "",
        "dob": "19879411",
        "city": "mongu",
        "phoneNumber": "",
        "nationalId": "898704115001082"
      }
    },
    {
      "uid": "0xaf4",
      "sourceId": {
        "uid": "0xaf3",
        "facility": "FA2",
        "patient": "199201285001099"
      },
      "uniqueInteractionData": {
        "auxDateCreated": "2023-09-04T14:38:46.01708765",
        "auxId": "rec-0000000297--0",
        "auxClinicalData": "RANDOM DATA(147)"
      },
      "demographicData": {
        "givenName": "jan",
        "familyName": "daia",
        "gender": "female",
        "dob": "",
        "city": "mwens3",
        "phoneNumber": "06841w1573",
        "nationalId": "199201285001099"
      }
    }
  ],
  "pagination": {
    "total": 18
  }
}

POST /customSearch/(golden|patient)

The following endpoint is used for the custom search either for golden or interactions.

Below a sample of the request body :

{
  "$or": [
    {
      "parameters": [
        {
          "fieldName": "givenName",
          "value": "John",
          "distance": 0
        },
        {
          "fieldName": "familyName",
          "value": "Doe",
          "distance": 3
        }
      ]
    },
    {
      "parameters": [
        {
          "fieldName": "nationalId",
          "value": "XXXXXXXXXXXXXX",
          "distance": 0
        }
      ]
    }
  ],
  "sortBy": "given_name",
  "sortAsc": true,
  "offset": 0,
  "limit": 10
}

The response payload is similar to the one returned by the simple search API endpoint.

POST /upload

The following endpoint is used to upload file into JeMPI. the file uploaded will be put into the async_reciever's storage under the /csv directory.

POST /calculateScores

The following endpoint is used to calculate the score between an interaction and a set of golden records

Below a sample of the request body

{
  "goldenIds": ["0x26bc"],
  "interactionId": "0x26bb"
}

POST /filterGids

The following endpoint returns a paginated Golden Ids list a request body illustrated in the example below

Note: this endpoint is similar to the search/(golden|patient)

{
  "parameters": [
    {
      "fieldName": "givenName",
      "value": "John",
      "distance": 0
    },
    {
      "fieldName": "familyName",
      "value": "Doe",
      "distance": 3
    }
  ],
  "sortBy": "givenName",
  "sortAsc": true,
  "offset": 0,
  "limit": 10
}

POST /crRegister

This endpoint is used for the patient registration process, in that it checks if the patient already exists, preventing duplicate patient registrations. This relies on the threshold provided in the request.

Below is a sample of the request body:

{
  "candidateThreshold": 0.9,
  "sourceId": {
    "facility": "fac1",
    "patient": "12345689"
  },
  "uniqueInteractionData": {
    "auxDateCreated": "2016-10-30T14:22:25.00Z",
    "auxClinicalData" : "REGISTER DATA",
    "auxId" : "1235",
    "auxIid" : "0x9"
  },
  "demographicData": {
    "givenName": "Sanket",
    "familyName": "",
    "gender": "male",
    "dob": "",
    "city": "Kolhapur",
    "nationalId": ""
  }
}

POST /crFind

This endpoint is used to find register patient golden records

Below is a sample of the request body:

{
  "operand": {
    "fn": "eq",
    "name": "givenName",
    "value": "Sanket"
  },
  "operands": [
    {
      "operator": "and",
      "operand": {
        "fn": "match",
        "distance": 1,
        "name": "familyName",
        "value": "Shevare"
      }
    }
  ]
}

POST /crCandidates

The following endpoint returns the list of candidate golden record given demographic data of a record and a threshold Below a sample of the request body

{
  "candidateThreshold": 0.9,
  "demographicData": {
    "givenName": "XX",
    "familyName": "YYY",
    "dob": "20000101",
    "gender": "female",
    "phoneNumber": "123456789",
    "city": "Cape Town",
    "nationalId": "123456789"
  }
}

Below a sample of the response body for this endpoint

{
  "goldenRecords": [
    {
      "goldenId": "0x9ed",
      "sourceId": [
        {
          "uid": "0x9e9"
        }
      ],
      "customUniqueGoldenRecordData": {
        "auxDateCreated": "2023-09-04T14:39:26.351509536",
        "auxAutoUpdateEnabled": true,
        "auxId": "rec-0000000272--1"
      },
      "demographicData": {
        "givenName": "alx",
        "familyName": "swna",
        "gender": "",
        "dob": "",
        "city": "ndolad",
        "phoneNumber": "",
        "nationalId": ""
      }
    },
    {
      "goldenId": "0x1220",
      "sourceId": [
        {
          "uid": "0x121e"
        }
      ],
      "customUniqueGoldenRecordData": {
        "auxDateCreated": "2023-09-04T14:39:59.44475916",
        "auxAutoUpdateEnabled": true,
        "auxId": "rec-0000000480--2"
      },
      "demographicData": {
        "givenName": "jan",
        "familyName": "dakq",
        "gender": "",
        "dob": "",
        "city": "",
        "phoneNumber": "",
        "nationalId": "1983081q5001080"
      }
    },
    {
      "goldenId": "0x1845",
      "sourceId": [
        {
          "uid": "0x1841"
        }
      ],
      "customUniqueGoldenRecordData": {
        "auxDateCreated": "2023-09-04T14:40:25.363902833",
        "auxAutoUpdateEnabled": true,
        "auxId": "rec-0000000643--2"
      },
      "demographicData": {
        "givenName": "james",
        "familyName": "bansa",
        "gender": "female",
        "dob": "",
        "city": "kansanvshi",
        "phoneNumber": "",
        "nationalId": "198602245001086"
      }
    }
  ]
}

POST /newlink

This endpoint is used to create a new golden record for an interaction, that the MPI identified as a possible match to another golden record

Below is a sample of the request body:

{
  "notificationId": "432fd5fe-e7db-4d3c-8a4a-fb23e8b9b45f",
  "notificationType": "MARGIN",
  "interactionId": "0xfb5",
  "currentGoldenId": "0xa5f",
  "resolutionState": "RELINKED_NEW",
  "currentCandidates": [
    "0xfaa"
  ],
  "newGoldenId": "",
  "score": 1
}

Below is a sample of the response:

{
  "goldenUID":"0x34ab",
  "interactionUID":"0xfb5",
  "sourceUID":"0x30",
  "score":2.0
}

POST /relink

This endpoint point is used to link an interaction, identified by the MPI as a possible link to a golden record

Below is a sample of the request body:

{
  "notificationId": "64c04a03-179f-4696-91ed-881dffc8f818",
  "notificationType": "MARGIN",
  "interactionId": "0x1f89",
  "currentGoldenId": "0xdda",
  "resolutionState": "RELINKED",
  "currentCandidates": [
    "0x1f7c"
  ],
  "newGoldenId": "0x1f7c",
  "score": 1
}

Below is a sample of the response:

{
  "goldenUID":"0x1f7c",
  "interactionUID":"0x1f89",
  "sourceUID":null,
  "score":1.0
}

POST /updateGoldenRecordFieldsForId

The following endpoint updates the fields of a golden record. This endpoint returns an object. Below a sample of the request :

{
  "goldenId":"0x2aeb",
  "fields": [
    {
      "name": "givenName",
      "value": "John"
    },
    {
      "name": "familyName",
      "value": "Doe"
    }
  ]
}

When at least one of the fields update is successful, the endpoint will return an array of the fields that have been successfully updated. If none of the fields has been updated, the endpoint will return a "400 Bad Request" response. The endpoint will return a "500 Internal Server Error" if the update failed for a different reason.

We use KeyCloak for identity management. This provide us with a OpenID Connect (an extension to OAuth 2.0) compliant identity service that we can use to authenticate users. Much like what Google and Github provide to login to other apps. Keycloak will provide:

• The login user experience, including signing in page

• 2FA

• Password reset features, account management

• The ability to manage user permissions centrally, across applications

• Our applications will just consume the resulting ID token that is produced to authenticate users and to check the roles that they are assigned.

We currently support the Auth Code Flow :

1. User access the JeMPI UI and clicks on "Sign-In with Keycloak".

2. User is redirected to Keycloak where he needs to submit his credentials.

3. User gets redirected back to the JeMPI UI along with the auth code parameters.

4. Auth code parameters are sent to the "POST /authenticate" JeMPI API endpoint.

5. JeMPI API sends the auth code to Keycloak along with the Client ID and Client Secret.

6. JeMPI gets token and verifies it, then parse the user infos (email, username, ...)

7. User is added to the Postgres Database if it's the first time he signs in.

8. JeMPI API creates a session and sends back the user object along with the session cookie.

9. User s redirected to the homepage.

Local development

Setup Local JeMPI

1. Clone the JeMPI git repository

git clone https://github.com/jembi/JeMPI.git && cd JeMPI/

1. Update local config to use Keycloak

export REACT_APP_JEMPI_BASE_API_PORT=50001
export REACT_APP_ENABLE_SSO="true"

1. Execute the local-deployment script

cd devops/linux/docker/deployment
./local-deployment.sh

1. Select Option 1: Deploy JeMPI (For Fresh Start)

2. Access : http://localhost:3000/login

1. Sign in with Keycloak user credentials

Overview

This Functionality provides detailed instructions on how to perform backup and restore operations using the JeMPI_BackupRestoreAPI.

This is a dedicated application for handling both backup and restore operations. The scripts included in this application cover:

• Backup Dgraph using API and dump JSON file.

• Backup Postgres using SQL-Dump

• Restore Dgraph using API using JSON file.

• Restore Postgres using SQL-Dump

Backup Operations

1. Dump sql files for Postgresql.

2. Get all GoldenIds

3. For each GoldenId: a. Retrieve the golden record b. Get the list of Golden Record Source Ids c. Get the list of interactions d. Write the data to file (JSON)

Prerequisites

• PostgreSQL Version: Ensure that PostgreSQL 15.5.0 is installed.

• The backup and restore operations are validated on this version. Verify the installation by running psql --version

Python Installation

• Make sure Python and the python-dotenv package are installed to manage environment variables.

• Verify the installation by running python3 -m dotenv --version

Using python-dotenv Load these variables using python-dotenv:

• from dotenv import load_dotenv

• import os

• load_dotenv(/path/to/your/.env)

• print(os.getenv(environment variable name))

• python3 test_dotenv.py (This should load the environment variables from your .env.local file)

Backup Dgraph and Postgres Process

• Backup Directory Dgraph: JeMPI/devops/linux/docker/docker_data/data/backups/dgraph

• Backup Directory Postgres: JeMPI/devops/linux/docker/docker_data/data/backups/postgres

• Deployment File: local-deployment.sh

cd JeMPI/devops/linux/docker/deployment
./local-deployment.sh
Select Option 5: Backup Postgres & Dgraph

• Backup Script Path: JeMPI/devops/linux/docker/backup_restore/dgraph-backup-api.sh

• Backup Script Logic Dgraph: JeMPI/devops/linux/docker/backup_restore/dgraph-backup-api.py

• Backup Script Logic Postgres: JeMPI/devops/linux/docker/backup_restore/postgres-backup.sh

The backup process creates a folder with a timestamp. Inside this folder, backups are created for each Dgraph and Postgres.

Manual Backup Process

• pg_dump -U -d <database_name> > /path/to/backup_file.sql

Verify if process was successful

• echo $? (This variable holds the exit status of the last command executed. An exit status of 0 indicates that the last command (pg_dump) completed successfully.)

• ls -lh

Restore Dgraph and Postgres Process

• Backup Directory: JeMPI/devops/linux/docker/docker_data/data/backups/dgraph

• Deployment File: local-deployment.sh

cd JeMPI/devops/linux/docker/deployment
./local-deployment.sh
Select Option 6: Restore Postgres & Dgraph

• It will prompt for confirmation (yes/no) and list the recent 5 backup folders.

• Enter the backup folder name - it will start the restoration from the selected backup file.

• Backup Directory Dgraph: JeMPI/devops/linux/docker/docker_data/data/backups/dgraph

• Backup Directory Postgres: JeMPI/devops/linux/docker/docker_data/data/backups/postgres

• Manual Backup Run Script: ./restore-dgraph-postgres.sh {{ Folder_Name }}

Configuration Settings The configuration settings screen enables the user to make edits to the default settings, the best fit the desired implementation of the MPI.

Common Properties This tab defines the demographic details for a patient that will be used for linking.

The user can do the following:

-Choose to select the close button to exit edit mode. -Choose to select the save icon button to save changes made and exit edit mode. -Edit the relevant fields and select the save button to save changes on the current tab.

Deterministic The deterministic tab is used to define the deterministic rules.

The deterministic tab has three sub tabs :

• Linking

• Validate

• Matching

Source view this view allows the user to do the following :

• View the displayed rules

• Click edit mode by clicking the edit icon button which opens up the design view

Design view this view allows the user to do the following :

• Select the operator values from a dropdown field, e.g., "And" and "Or"

• Select common field values from a dropdown field

• Select comparator function from a dropdown field, e.g., "Exact", "Low Fuzziness", etc.

• Add a second row of input fields by selecting the add icon button

• Save rule by selecting the add rule button

• Exit edit mode and cancel previous edits.

The blocking tabs have two sub tabs:

• Linking

• Matching

The blocking sub tabs have two different views :

Source view This view allows the user to do the following :

• View the displayed rules

• Click edit mode by clicking the edit icon button which opens up the design view

Design view This view must allow the user to do the following:

• Select the operator values from a dropdown field, e.g., "And" and "Or"

• Select common field values from a dropdown field.

• Select comparator function from a dropdown field, e.g., "Exact", "Low Fuzziness", etc.

• Add a second row of input fields by selecting the add icon button

• Save rule by selecting the add rule button

• Exit edit mode and cancel previous edits.

Rules on threshold slider

• Do not allow the link threshold (green circle): To be < the Minimum threshold review value To be > the Maximum threshold review value

Rules on Threshold For all threshold values that are entered, system allows for exponential notation e.g. 123E-3 which is the same as 0.012 System display default values

Nodes This section displays the following :

• Golden record node

• Interaction node

• Source ID

Golden record node shows properties unique to the golden record. Interaction node shows properties unique to the interaction. Source ID : The third node denoted e.g Source ID, shows unique common lists e.g

• Source ID list

• Biometric ID list

Section 1: Getting Started

Top Navigation

The navigation bar on the top of the application is always visible and accessible to the user. There are 4 main screens, i.e., Dashboard, Browse, Notifications, and Import.

Understanding the Top Navigation bar

Terms and Definitions to get started with understanding JeMPI

Blocking: Reducing the search space by grouping records by similar attributes into blocks.

Candidate records: A short list of Golden Records generated as a result of blocking. The records on this short list are referred to as candidate (golden) records, as they are potential candidates for linking.

Patient Interaction record: Stores demographic information of the patient e.g. name, surname, DOB, gender, address, etc. This information together with the unique source system ID is used to uniquely identify patients.

Patient ID: Unique identifier of the patient record assigned by the JeMPI upon entry.

Golden record (GR): A golden record is created for a patient if this is the first and/or only patient record to be stored in the database. This is the same as a master record. The golden record links records based on a match score, i.e., determines that 2 or more patient records belong to the same person.

Always has the most up-to-date information for a patient by consensus among the golden record's interactions.

Golden ID: Unique identifier of the Golden record.

Link Threshold (LTH): Predetermined values that allocate the match status of two records based on the probabilistic score generated by the comparison algorithm. M & U values: The matching (M) and unmatching (U) values derived for each record field.

• The m values can be expressed as data quality and calculated as the ratio of matching attributes given that they belong to the same record.

• The u values can be expressed as data uniqueness and calculated as the ratio of matching values given that they do not belong to the same record. -Because the status of matching records is unknown, m and u values are calculated using the Expectation Maximization algorithm.

Matching configuration: The basic matching configuration allows for the adjustment of an acceptable matching score between two or more records. Should the threshold be raised, only high matching scores will result in a confirmed match and thus is more stringent. Should the threshold be lowered, moderate matching scores will result in confirmed matches and thus be more lenient.

Relaxed Search: Relaxed searching is functionality where refined blocking is performed with different criteria (e.g., such as another condition with different fields) to increase the number of results. If the user is not happy with the results, this functionality may perform filtering instead of blocking. (e.g Find all males in village x)

Similarity Score: A similarity score is often the normalised expression of similarity between two strings, whereby 1 represents an exact match and 0 represents no similarity at all. Popular algorithms used to calculate similarity are Jaro-Winkler and Levenshtein (normalised).

Matching Score: Also referred to as a linking score, is the accumulated value attributed to show evidence for the positive match of two records.

Different metrics and values are applied to generate a matching score, and a threshold is chosen to assign positive matches for those records exceeding the threshold. A matching score can be normalised from 0 to 1.

Review Threshold (RTH): Determines if a record must be flagged for peer review and a notification must be sent.

GR Changed Patient record Threshold: When a Golden record field is edited and saved, the established links to the Golden Record are recomputed.

For each linked patient record where the similarity score falls below this threshold, a ‘for review’ event is queued for that patient record. Example: 5 Patient records are linked to a GR. GR fields are updated. After the update, 2 of the 5 patient records now fall below this threshold. For each of these 2 patient records, an event will be queued (Two events sent)

Important Identity and Unique numbers used in JeMPI

Golden Record ID: This is the unique identity number associated with each Golden Record. It is unique and cannot be edited. The golden records contain a set of all source ID’s from the linked patient records

Patient ID: For each patient record that enters JeMPI, the system will provide a unique ID for the patient record. This number is unique and cannot be edited.

Source System ID: The source system ID is the incoming patient identifier defined as the source system ID and the patient ID within that source.

Auxiliary ID: The auxiliary ID is a generated ID for Test Data developed to bootstrap and measure the accuracy of the JeMPI system. This ID is not relevant outside of testing.

STAN: System Trace Audit Number. A unique ID to trace messages through the JeMPI system, created by the client. The client defines the format of this STAN.

Section 2: Dashboard

The Dashboard screen has 3 tabs

• Confusion Matrix

• M & U values

• Import Process Status.

This tab is subdivided into 3 sections, starting with the right, the Confusion Matrix.

Confusion Matrix

Understanding the confusion matrix

The confusion matrix displays a tally of the true positives, false positives, true negatives and false negatives. This is used to calculate the precision and recall.

The f-score is a measure of a model’s accuracy on a dataset. It is a harmonic mean of precision and recall.

Beta F-scores

Records and notifications

Records

• Displays the total number of Golden records and total number of interactions. Notifications - Displays total number of notifications split by:

Open Notifications

• No. of New & Open notifications Closed notifications

• No. of Closed notifications

Note: The number of new and open notifications (basically notifications that are not closed) affects the accuracy of the F-score and depending on the % of notifications that have not been actioned.

Dashboard Tab 2: M & U values

This screen provides us with a view of the M & U values as per the last periodic update.

Tally Method

The Tally method computes M & U's per field, which is used for cross-checking against the M&U's computed by the EM algorithm. These M& U's are not used for probabilistic linking.

What happens when the score is within the notification threshold area?

When in the notification area, we are either above the notification TH or below the notification TH.

Above the threshold (using the incoming interaction and linked GR)

• Assume this is correct for 80% of the time - increment A or B by 0.8

• Assume this is incorrect for 20% of the time - increment C or D by 0.2

If admin confirms this assumption, then the system must adjust the tallies by adding the 0.2 to A or B and removing 0.2 from C or D If admin rejects this assumption, then the system must subtract 0.8 from A or B and add to C or D

Below the threshold (using the incoming interaction and the candidate GRs)

• Assume this is correct for 80% of the time - increment C or D by 0.8

• Assume this is incorrect for 20% of the time - increment A or B by 0.2 If admin confirms this assumption, then the system must adjust the tallies by adding the 0.2 to C or D and removing 0.2 from A or B If admin rejects this assumption, then system must subtract 0.8 from C or D and add to A or B

M and U values are calculated as follows:

• M = A/( A+B)

• U= C/(C+D)

Dashboard Tab 3: Import Process Status

This screen displays the progress of the processing of the file uploaded via the Import screen.

Section 3: Browse

Browse Patients

This screen displays a list of golden records, with the most recent golden record displayed on the top of the grid.

• Select the Browse option on the top navigation bar

• Screen is displayed with a list of current patient interactions. This is the default view.

• The options on this screen are:: a. Select one of the patient interaction (row) to view more details of the patient b. Filter the results to find specific patients and/or list of interactions

Filter by

Filter by option

1. Select the Filter by panel

2. System expands the panel and displays the various options to filter the results:

• Filter by start and end date - dates can be selected using the calendar picker

• Get interactions - returns the golden record and patient interactions for the golden record

• Filter by a single field or combination of the fields below: a. UID, First Name, Last Name, Gender, Date of birth, City, Phone No. b. For each of the fields selected, the search can be further extended by selecting a type per field i.e. (i) Exact - returns results that exactly match the value entered (ii) Levenshtein 1 - returns results with low fuzziness and a distance parameter = 1 (iii) Levenshtein 2 - returns results with medium fuzziness and a distance parameter = 2 (iv)Levenshtein 3 - returns results with high fuzziness and a distance parameter = 3 (default)

1. Enter the search criteria value for one or more fields that you want to search on

2. Select the FILTER button to view the results a. If no results are found, the system displays a message informing the user that no results are available.

3. Select the CANCEL button to clear the entered search criteria and repeat steps if required.

Filter by (Get interactions)

When the Get interactions toggle is switched on, the system displays the Golden record (GR) (row highlighted in yellow) and the linked patient interactions. All patient interactions that belong to the GR are displayed under the GR. To view the details of a patient, select the relevant row. In order to view the details of a patient, select the relevant row. System navigates to a detailed view of the selected patient’s interactions.

View Details of Patient Interaction

This is a detailed view of a patient. The first row (highlighted in yellow) is the Golden record. The rows below are the patient interactions. In this example below, the patient has 1 interaction.

How does this work?

When a patient interaction is loaded for the first time, and there is no matching record, a golden record is created using the patient interaction details. Thereafter, every matching patient interaction is linked to the golden record. The golden record is updated based on the following rules:

1. If a golden record has missing values and the 2nd interaction comes in with a populated value, the system will update the null value in the GR to match the field in the 2nd interaction.

2. Thereafter, if there are 2 or more interactions with a different field value to the GR field value, then the majority rule applies, in that the field in the GR will be updated as per the majority. This update is configurable and can be disabled

On the Patient interaction screen, the user can do the following:

• View the details of the Golden record and its linked patient interactions together with the audit trail

• Edit the Golden record (with permissions)

• Relink the patient interaction

View Patient Interactions and Audit Trail

The golden record and the interactions are clickable.

• When the Golden record is selected, the full audit trail for the patient is displayed, i.e., all the events that occurred on each interaction are displayed.

• When the Interaction is selected, the audit trail displays the event for that interaction only (refer to screenshot below).

Editing a Golden Record

The user also has the option to update the applicable GR fields where edits are allowed. No edits will be allowed on any system generated fields, e.g., Golden ID. The fields that are editable are configurable.

How does this work?

After updating the GR field values, on save, the system does the following:

• re-computes scores for all automatically linked patient interactions

• updates the similarity score to indicate that the record has been manually updated (link score = 3.0)

• disables the Master auto-update fields flag to prevent auto-updates

• checks the new GR changed Patient record Threshold(TH) and if the scores for any of the linked patient interaction records fall below this TH, then sends a notification for Admin user to review.

1. Select the record

2. Select the edit option - the GR row becomes editable.

3. Enter or edit the field value as required

4. Select the save option

5. System displays a successfully saved message.

Relink a patient interaction

Relinking a patient interaction means that the interaction is not correctly linked to a Golden record and the Admin user wants to relink the interaction to an existing Golden record or create a new Golden record. There are 2 ways that a patient interaction can be relinked:

1. From viewing an interaction - when the user views the interaction, the user may choose to relink the interaction (i.e. no notification received)

2. From a notification - a notification is received informing the user that some action must be taken. The user can choose to relink the interaction to another golden record or create a new golden record to link to.

Relink from Patient Interactions screen

1. Select the Relink option

2. System displays the Review Linked Patient screen (below)