Package can be stood up individually using the instant package init -n <package_name> command, or they can be included in your own recipes. This can be accomplished by that includes the necessary packages and any custom configuration packages.

This component consists of two services:

• Interoperability Layer - OpenHIM for routing the events

• Mongo DB for storing the transactions

It provides an interface for:

1. Checking the transactions logs

2. Configuring the channels to route the events

3. User authentication logs

4. Service logs

5. Rerun the transactions tasks

6. Reprocess mediator launching

OpenHIM is based on two 3 main services, openhim-core as a backend, openhim-console as a frontend and mongo as a database.

It is a mandatory component in the stack and the entry point for all incoming requests from the external systems.

Accessing the services

OpenHIM

• Console: http://127.0.0.1:9000

• Username: root@openhim.org

• Password: instant101

Testing the Interoperability Component

As part of the Interoperability Layer setup we also do some initial config import for connecting the services together.

• OpenHIM: Import a channel configuration that routes requests to the Data Store - HAPI FHIR service

This config importer will import channels and configuration according to the file openhim-import.json in the folder <path to project packages>/interoperability-layer-openhim/importer/volume.

The monitoring package sets up services to monitor the entire deployed stack. This includes the state of the servers involved in the docker swarm, the docker containers themselves and particular applications such as Kafka. It also captures the logs from the various services.

This monitoring package uses:

• Grafana: for dashboards

• Prometheus: for recording metrics

• Cadvisor: for reading docker container metrics

• Loki: for storing logs

• Node Exporter: for monitoring host machine metrics like CPU, memory etc

To use the monitoring services, include the monitoring package id to your list of package ids when standing up the platform.

Adding application specific metrics

The monitoring service utilises service discovery to discover new metric endpoints to scrape.

To use custom metrics for an application, first configure that application to provide a Prometheus compatible metrics endpoint. Then, let the monitoring service know about it by configuring specific docker service labels that tell the monitoring service to add a new endpoint to scrape. E.g. see lines 8-9:

  kafka-minion:
    image: quay.io/cloudhut/kminion:master
    hostname: kafka-minion
    environment:
      KAFKA_BROKERS: kafka:9092
    deploy:
      labels:
        - prometheus-job-service=kafka
        - prometheus-address=kafka-minion:8080

prometheus-job-service lets Prometheus know to enable monitoring for this container and prometheus-address gives the endpoint that Prometheus can access the metrics on. By default this is assumed to be at the path /metrics by Prometheus.

By using the prometheus-job-service label prometheus will only create a single target for your application even if it is replicated via service config in docker swarm. If you would like to monitor each replica separately (i.e. if metrics are only captured for that replica and not shared to some central location in the application cluster) you can instead used the prometheus-job-task label and Prometheus will create a target for each replica.

A full list od supported labels are listed below:

• prometheus-job-service - indicates this service should be monitored

• prometheus-job-task - indicates each task in the replicated service should be monitored separately

• prometheus-address - the service address Prometheus can scrape metrics from, can only be used with prometheus-job-service

• prometheus-scheme - the scheme to use when scaping a task or service (e.g. http or https), defaults to http

• prometheus-metrics-path - the path to the metrics endpoint on the target (defaults to /metrics)

• prometheus-port - the port of the metrics endpoint. Only usable with prometheus-job-task, defaults to all exposed ports for the container if no label is present

All services must also be on the prometheus_public network to be able to be seen by Prometheus for metrics scraping.

Adding additional dashboards

To add additional dashboards simply use docker configs to add new Grafana dashboard json files into this directory in the Grafana container: /etc/grafana/provisioning/dashboards/

That directory will be scanned periodically and new dashboards will automatically be added to Grafana.

Grafana dashboard json file may be exported directly from the Grafana when saving dashboards or you may lookup the many existing dashboard in the Grafana marketplace.

Kafka-mapper-consumer

A Kafka processor that will consume messages from Kafka topics. This messages will be mapped according to the mapping defined in the file called fhir-mapping.json.

This flattened data will be then sent to Clickhouse DB to be stored.

Each topic has its own table mapping, plugin and filter and one topic may be mapped in different ways.

An example of fhir-mapping.json can be found in the package.

Each new message with new ID will be inserted as a new row in the table defined in the mapping. An update of the message will result on update in Clickhouse DB accordingly. Link to GitHub repo: https://github.com/jembi/kafka-mapper-consumer.

Logstash provides a data transformation pipeline for analytics data. In the platform it is responsible for transforming FHIR messages into a flattened object that can be inserted into Elasticsearch.

Input

Logstash allows for different types of input to read the data: Kafka, HTTP ports, files, etc.

Filters

With a set of filters and plugins, the data can be transformed, filtered, and conditioned.

This allows the creation of a structured and flattened object out of many nested and long resources.

Accessing the different fields will be much easier and we will get rid of the unused data.

Output

To save the data, Logstash provides a set of outputs such as: Elasticsearch, S3, files, etc.

Adding pipelines and configs

To add Logstash config files, you can add files into the <path to project packages>/data-mapper-logstash/pipeline.

Developing the Logstash configs locally

When seeking to make changes to the Logstash configs without having to repeatedly start and stop the service, one can set the LOGSTASH_DEV_MOUNT env var in your .env file to true to attach the service's config files to those on your local machine.

Cluster

When attaching Logstash to an Elasticsearch cluster ensure you use the ES_HOSTS environment variable. eg. ES_HOSTS="analytics-datastore-elastic-search-1:9200","analytics-datastore-elastic-search-2:9200","analytics-datastore-elastic-search-3:9200" and reference it in your logstash configs eg. hosts => [$ES_HOSTS]

Notes

• With LOGSTASH_DEV_MOUNT=true, you have to set the LOGSTASH_PACKAGE_PATH variable with the absolute path to package containing your Logstash config files, i.e., LOGSTASH_PACKAGE_PATH=/home/user/Documents/Projects/platform/data-mapper-logstash.

• WARNING: do not edit the pipeline files from within the Logstash container, or the group ID and user ID will change, and subsequently will result in file permission errors on your local file system.

Ofelia - Job Scheduler

Job Docs

The platform uses image: mcuadros/ofelia:v0.3.6 which has the following limitations:

• Ofelia does not support config.ini files when run in docker mode (which enables scheduling jobs with docker labels) thus we need to always use the config.ini file for creating jobs.

• Ofelia does not support attaching to a running instance of a service.

• Ofelia does not support job-run (which allows you to launch a job with a specified image name) labels on non-ofelia services (ie. you may not specify a job of type job-run within the nginx package as ofelia will not pick it up)

• Ofelia only initializes jobs when it stands up and does not listen for new containers with new labels to update it's schedules, thus Ofelia needs to be re-up'd every time a change is made to a job that is configured on another service's label.

Example of a job config

An example of job config in the file config.example.ini existing in the folder <path to project packages>/job-scheduler-ofelia/.

[job-run "renew-certs"]
schedule = @every 1440h ;60 days
image = jembi/swarm-nginx-renewal:v1.0.0
volume = renew-certbot-conf:/instant
volume = /var/run/docker.sock:/var/run/docker.sock:ro
environment = RENEWAL_EMAIL=${RENEWAL_EMAIL}
environment = STAGING=${STAGING}
environment = DOMAIN_NAME=${DOMAIN_NAME}
environment = SUBDOMAINS=${SUBDOMAINS}
environment = REVERSE_PROXY_STACK_NAME=${REVERSE_PROXY_STACK_NAME}
delete = true

The Ofelia service does not make use of any environment variables. However, when specifying jobs in the config.ini file(s) we can pass any environment variable in.

Example:

[job-run "mongo-backup"]
schedule= @daily
image= mongo:4.2
network= mongo_backup
volume= /backups:/tmp/backups
command= sh -c 'mongodump --uri=${OPENHIM_MONGO_URL} --gzip --archive=/tmp/backups/mongodump_$(date +%s).gz'
delete= true

Launching

Launching this package executes the following two steps:

• Running Clickhouse service

• Running config importer to run the initial SQL script

Initializing ClickHouse

The config importer will be launched to run a NodeJS script after ClickHouse has started.

It will run SQL queries to initialize the tables and the schema, and can also include initial seed data if required.

The config importer looks for two files clickhouseTables.js and clickhouseConfig.js found in <path to project packages>/analytics-datastore-clickhouse/importer/config.

For specific implementation, this folder can be overridden.

Launching

Launching this package follows different steps:

• [Cluster mode] Creating certificates and configuring the nodes

• Running Elasticsearch

• Setting Elasticsearch passwords

• Importing Elasticsearch index

Importing

To initialize the index mapping in Elasticsearch, a helper container is launched to import a config file to Elasticsearch. The config importer looks for a field named fhir-enrich-report.json in <path to project packages>/analytics-datastore-elastic-search/importer.

The file fhir-enrich-report.json will contain the mapping of the index fhir-enrich-reports.

Elasticsearch will create a dynamic mapping for the incoming data if we don't specify one, this dynamic mapping may cause issues when we start sending the data as it doesn't necessarily conform 100% to the data types that we're expecting when querying the data out of Elasticsearch again.

Therefore, the mapping should be initialized in Elasticsearch using the config importer.

The file fhir-enrich-report.json is just an example, the name and the mapping can be overridden.

Running in Dev Mode

When running in DEV mode, Elasticsearch is reachable at:

http://127.0.0.1:9201/

Elasticsearch Backups

For detailed steps about creating backups see: Snapshot filesystem repository docs.

Elasticsearch offers the functionality to save a backup in different ways, for further understanding, you can use this link: Register a snapshot repository docs.

Elasticsearch Restore

To see how to restore snapshots in Elasticsearch: Snapshot Restore docs.

Pre-Deploy Configuration

If running in clustered mode, take note that each machine has to have the following vm.max_map_count setting:

sysctl -w vm.max_map_count=262144

This package consists of four services:

• Postgres Main DB

• Postgres Audit DB

• SanteMPI Web UI -

• SanteMPI API -

Note

The environment variable SANTEMPI_REPMGR_PARTNER_NODES will differ from cluster and single mode.

Default value for SANTEMPI_MAIN_CONNECTION_STRING:

server=santempi-psql-1;port=5432; database=santedb; user id=santedb; password=SanteDB123; pooling=true; MinPoolSize=5; MaxPoolSize=15; Timeout=60;

Default value for SANTEMPI_AUDIT_CONNECTION_STRING:

server=santempi-psql-1;port=5432; database=auditdb; user id=santedb; password=SanteDB123; pooling=true; MinPoolSize=5; MaxPoolSize=15; Timeout=60;

Accessing the Service

Jsreport - http://127.0.0.1:5488/

Scripts/Templates Development

When seeking to make changes to the Jsreport scripts/templates without having to repeatedly start and stop the service, one can set the JS_REPORT_DEV_MOUNT environment variable in your .env file to true to attach the service's content files to those on your local machine.

Export & Import

After editing the templates in Jsreport, you will need to save these changes, it is advised to export a file containing all the changes named export.jsrexport and put it into the folder <path to project packages>/dashboard-visualiser-jsreport/importer.

The config importer of Jsreport will import the export.jsrexport and then all the templates, assets, and scripts will be loaded in Jsreport.

Accessing the Service

Kibana - http://127.0.0.1:5601/

Importing Saved Objects

The config importer will import the file kibana-export.ndjson that exists in the folder <path to project packages>/dashboard-visualiser-kibana/importer.

The saved objects that will be imported are the index patterns and dashboards. If you made any changes to these objects please don't forget to export them and save the file kibana-export.ndjson under the folder specified above.

Version upgrade process (with rollback capability)

By default if you simply update the image that the superset service uses to a later version, when the container is scheduled it will automatically run a database migration and the version of superset will be upgraded. The problem, however, is that if there is an issue with this newer version you cannot rollback the upgrade since the database migration that ran will cause the older version to throw an error and the container will no longer start. As such it is recommended to first create a postgres dump of the superset postgres database before attempting to upgrade superset's version.

1. Exec into the postgres container as the root user (otherwise you will get write permission issues)

1. Run the pg_dump command on the superset database. The database name is stored in SUPERSET_POSTGRESQL_DATABASE and defaults to superset

1. Copy that dumpped sql script outside the container

1. Update the superset version (either through a platform deploy or with a docker command on the server directly -- docker service update superset_dashboard-visualiser-superset --image apache/superset:tag)

Rolling back upgrade

In the event that something goes wrong you'll need to rollback the database changes too, i.e.: run the superset_backup.sql script we created before upgrading the superset version

1. Copy the superset_backup.sql script into the container

1. Exec into the postgres container

1. Run the sql script (where -d superset is the database name stored in SUPERSET_POSTGRESQL_DATABASE)

Accessing the Service

Superset - http://127.0.0.1:8089/

Using the superset_config.py file

The Superset package is configured to contain a superset_config.py file, which Superset looks for, and subsequently activates the contained feature flags. For more information on the allowed feature flags, visit https://github.com/apache/superset/blob/master/RESOURCES/FEATURE_FLAGS.md.

Importing & Exporting Assets

The config importer written in JS will import the file superset-export.zip that exists in the folder <path to project packages>/dashboard-visualiser-superset/importer/config. The assets that will be imported to Superset are the following:

• The link to the Clickhouse database

• The dataset saved from Clickhouse DB

• The dashboards

• The charts

If you made any changes to these objects please don't forget to export and save the file as superset-export.zip under the folder specified above. NB! It is not possible to export all these objects from the Superset UI, you can check the Postman collection: CARES DISI CDR -> Superset export assets and you will find two requests. To do the export, three steps are required:

1. Run the Get Token Superset request to get the token (please make sure that you are using the correct request URL). An example of a response from Superset that will be displayed: { "access_token": "eyJ0eXAiOiJKV1...." }

2. Copy the access token and put it into the second request Export superset assets in the Authorization section.

3. Run the second request Export superset assets . You can save the response into a file called superset-export.zip under the folder specified above.

Your changes should then be saved.

Components

The message-bus-kafka package consists of a few components, those being Kafka, Kafdrop, and Kminion.

Kafka

The core stream-processing element of the message-bus-kafka package.

Kafdrop

Kafdrop is a web user-interface for viewing Kafka topics and browsing consumer-groups.

Kminion

A prometheus exporter for Kafka.

Kafka Topics Configuration

Using a config importer, Kafka's topics are imported to Kafka. The topics are specified using the KAFKA_TOPICS environment variable, and must be of syntax:

topic or topic:partition:replicationFactor

Using topics 2xx, 3xx, and metrics (partition=3, replicationFactor=1) as an example, we would declare:

KAFKA_TOPICS=2xx,3xx,metrics:3:1

where topics are separated by commas.

Accessing Kafdrop

Kafdrop - http://127.0.0.1:9013/

The HAPI FHIR service will be used for two mandatory functionalities:

• A validator of FHIR messages

• A storage of FHIR message

A validator

Incoming messages from an EMR or Postman bundles are not always well structured and it may be missing required elements or be malformed.

HAPI FHIR will use a FHIR IG to validate these messages.

It will reject any invalid resources and it will return errors according to the IG.

HAPI FHIR is the first check to make sure the data injected in the rest of the system conforms to the requirements.

A storage

Backed by a PostgreSQL database, all the validated incoming messages will be stored.

This will allow HAPI FHIR to check for correct links and references between the resources, as well as another storage for backups in case the data is lost.

Instant OpenHIE FHIR Data Store Component

This component consists of two services:

• Postgres

• HAPI FHIR Server - HAPI FHIR

Accessing the services

HAPI FHIR

This service is accessible for testing via:

http://127.0.0.1:3447

In a publicly accessible deployment this port should not be exposed. The OpenHIM should be used to access HAPI-FHIR.

Testing the HAPI FHIR Component

For testing this component we will be making use of curl for sending our request, but any client could be used to achieve the same result.

Execute the command below

curl http://127.0.0.1:3447/fhir/Patient

The kafka unbundler will consume resources of topix 2xx from Kafka, split them according to their resource type and send them again to Kafka to new topics.

Each resource type has its own topic.

Link for github repo: https://github.com/jembi/kafka-unbundler-consumer.

A helper for Kafka message bus service, It sends data to the HAPI FHIR datastore and then to the Kafka message bus based on the response from HAPI FHIR.

More particularly:

1. It receives messages from OpenHIM

2. It sends the data to the HAPI FHIR server and waits for the response

3. It gets the response. According to the response status, it will send the message to the topic that corresponds to that status (2xx, 4xx, 5xx, ... )

4. It will send back the response from HAPI FHIR to OpenHIM as well

Nginx Reverse Proxy

This package can be used to secure all of the data transfered to and from services using SSL encryption and also to generate SSL certificates as well.

Instead of configuring each package separately, we're using this package that will hold all of the Nginx configuration.

It will generate Staging or Production certificates from Let's Encrypt to ensure a secure connection (in case we require SSL to be enabled).

It is responsible for routing network traffic to the correct service.

Structure of Reverse Proxy Nginx package

The current package contains the following:

• config: A folder that contains the general Nginx config for secure and insecure mode.

• package-conf-insecure: A folder that contains all the insecure configs related to the services that need outside access.

• package-conf-secure: A folder that contains all the secure configs related to the services that need outside access.

A job using Ofelia exists to renew the certificates automatically based on the certificate renewal period.