Backup and Restore
Enable backup and restore for Postgres and Dgraph datastore's
Overview
This Functionality provides detailed instructions on how to perform backup and restore operations using the JeMPI_BackupRestoreAPI.
This is a dedicated application for handling both backup and restore operations. The scripts included in this application cover:
Backup Dgraph using API and dump JSON file.
Backup Postgres using SQL-Dump
Restore Dgraph using API using JSON file.
Restore Postgres using SQL-Dump
Backup Operations
Dump sql files for Postgresql.
Get all GoldenIds
For each GoldenId: a. Retrieve the golden record b. Get the list of Golden Record Source Ids c. Get the list of interactions d. Write the data to file (JSON)
Prerequisites
PostgreSQL Version: Ensure that PostgreSQL 15.5.0 is installed.
The backup and restore operations are validated on this version. Verify the installation by running psql --version
Python Installation
Make sure Python and the python-dotenv package are installed to manage environment variables.
Verify the installation by running python3 -m dotenv --version
Using python-dotenv Load these variables using python-dotenv:
from dotenv import load_dotenv
import os
load_dotenv(/path/to/your/.env)
print(os.getenv(environment variable name))
python3 test_dotenv.py (This should load the environment variables from your .env.local file)
Backup Dgraph and Postgres Process
Backup Directory Dgraph: JeMPI/devops/linux/docker/docker_data/data/backups/dgraph
Backup Directory Postgres: JeMPI/devops/linux/docker/docker_data/data/backups/postgres
Deployment File: local-deployment.sh
Backup Script Path: JeMPI/devops/linux/docker/backup_restore/dgraph-backup-api.sh
Backup Script Logic Dgraph: JeMPI/devops/linux/docker/backup_restore/dgraph-backup-api.py
Backup Script Logic Postgres: JeMPI/devops/linux/docker/backup_restore/postgres-backup.sh
The backup process creates a folder with a timestamp. Inside this folder, backups are created for each Dgraph and Postgres.
Manual Backup Process
pg_dump -U -d <database_name> > /path/to/backup_file.sql
Verify if process was successful
echo $? (This variable holds the exit status of the last command executed. An exit status of 0 indicates that the last command (pg_dump) completed successfully.)
ls -lh
Restore Dgraph and Postgres Process
Backup Directory: JeMPI/devops/linux/docker/docker_data/data/backups/dgraph
Deployment File: local-deployment.sh
It will prompt for confirmation (yes/no) and list the recent 5 backup folders.
Enter the backup folder name - it will start the restoration from the selected backup file.
Backup Directory Dgraph: JeMPI/devops/linux/docker/docker_data/data/backups/dgraph
Backup Directory Postgres: JeMPI/devops/linux/docker/docker_data/data/backups/postgres
Manual Backup Run Script: ./restore-dgraph-postgres.sh {{ Folder_Name }}
Last updated