AWS Deployment using EFS

Overview

A shared file system is always required to create OpenSearch snapshots. To register the snapshot repository using OpenSearch, it is necessary to mount the same shared filesystem to the exact location on all master and data nodes. Register the location in the path.repo setting on all master and data nodes.

Setting up the backup configuration

• Create an EFS file system, please refer sample steps here

• Let’s create a folder structure /mnt/automate_backups/ on all the Frontend and backend nodes, then we have to mount EFS to all the vm’s manually. To do that please refer this

Configuration in OpenSearch Node

• Mount the EFS on all OpenSearch Node. For example you mount the EFS to folder structure /mnt/automate_backups/

• Create an opensearch sub-directory and set permissions as mention below (all the opensearch nodes).

  sudo mkdir -p /mnt/automate_backups/opensearch
  sudo chown hab:hab /mnt/automate_backups/opensearch/
  

Configuration for OpenSearch Node from Provision host

Configure the OpenSearch path.repo attribute.

• Create a toml file (os_config.toml) and add below template

  [path]
  repo = "/mnt/automate_backups/opensearch"
  

• Patch the config os_config.toml from bastion to the opensearch cluster.

  chef-automate config patch --opensearch os_config.toml
  

• Above command will restart the opensearch cluster.

Healthcheck commands

• Following command can be run in the OpenSearch node

  hab svc status (check whether OpenSearch service is up or not)
  
  curl -k -X GET "<https://localhost:9200/_cat/indices/*?v=true&s=index&pretty>" -u admin:admin (Another way to check is to check whether all the indices are green or not)
  
  # Watch for a message about OpenSearch going from RED to GREEN
  `journalctl -u hab-sup -f | grep 'automate-ha-opensearch'
  

Configuration for Automate node from Bastion host

• Mount the EFS to all the Frontend node manually. For example you mount the EFS to folder structure /mnt/automate_backups

• Create an automate.toml file on the bastion host using the following command:

  touch automate.toml
  

• Add the following configuration to automate.toml on the bastion host:

  [global.v1.external.opensearch.backup]
  enable = true
  location = "fs"
  
  [global.v1.external.opensearch.backup.fs]
  # The `path.repo` setting you've configured on your OpenSearch nodes must be a parent directory of the setting you configure here:
  path = "/mnt/automate_backups/opensearch"
  
  [global.v1.backups.filesystem]
  path = "/mnt/automate_backups/backups"
  

• Patch the config using below command.

  ./chef-automate config patch --frontend automate.toml
  

Backup and Restore commands

Backup

• Run the backup command from bastion as shown below to create a backup:

  chef-automate backup create
  

Restoring the EFS Backed-up Data

To restore backed-up data of the Chef Automate High Availability (HA) using External File System (EFS), follow the steps given below:

• Check the status of all Chef Automate and Chef Infra Server front-end nodes by executing the chef-automate status command.

• Execute the restore command from bastionchef-automate backup restore <BACKUP-ID> -b /mnt/automate_backups/backups --airgap-bundle </path/to/bundle>.

Troubleshooting

Try these steps if Chef Automate returns an error while restoring data.

1. Check the Chef Automate status.

   chef-automate status
   

2. Check the status of your Habitat service on the Automate node.

   hab svc status
   

3. If the deployment services are not healthy, reload them.

   hab svc load chef/deployment-service
   

Now check the status of the Automate node and then try running the restore command from the bastion host.