On-Prem Deployment using Filesystem

Overview

A shared file system is always required to create OpenSearch snapshots. To register the snapshot repository using OpenSearch, it is necessary to mount the same shared filesystem to the exact location on all master and data nodes. Register the location (or one of its parent directories) in the path.repo setting on all master and data nodes.

Setting up the backup configuration

Configuration in OpenSearch Node

• Mount the shared file system to the base mount path which is mentioned in backup_mount on all OpenSearch and Frontend servers.

Apply the following steps on all of the OpenSearch server node

• Create an OpenSearch sub-directory and set permissions (only if the network mount is correctly mounted).

  sudo mkdir /mnt/automate_backups/opensearch
  sudo chown hab:hab /mnt/automate_backups/opensearch/
  

Configuration for OpenSearch Node from Bastion Host

Configure the OpenSearch path.repo setting by following the steps given below:

• Create a .toml (say os_config.toml) file in the Bastion host and copy the following template with the path to the repo.

    [path]
    # Replace /mnt/automate_backups with the backup_mount config found on the Bastion host in /hab/a2_deploy_workspace/a2ha.rb
    repo = "/mnt/automate_backups/opensearch"
  

• Following command will add the configuration to the OpenSearch node.

    chef-automate config patch --opensearch <PATH TO OS_CONFIG.TOML>
  

Healthcheck commands

• Following command can be run in the OpenSearch node

  chef-automate status
  

• Following command can be run in the OpenSearch node

  hab svc status (check whether OpenSearch service is up or not)
  
  curl -k -X GET "<https://localhost:9200/_cat/indices/*?v=true&s=index&pretty>" -u admin:admin (Another way to check is to check whether all the indices are green or not)
  
  # Watch for a message about OpenSearch going from RED to GREEN
  journalctl -u hab-sup -f | grep 'automate-ha-opensearch'
  

Configuration for Automate Node from Provision Host

• Configure Automate to handle External OpenSearch Backups.

• Create an automate.toml file on the provisioning server using the following command:

  touch automate.toml
  

  Add the following configuration to automate.toml on the provisioning host:

  [global.v1.external.opensearch.backup]
  enable = true
  location = "fs"
  
  [global.v1.external.opensearch.backup.fs]
  # The `path.repo` setting you've configured on your OpenSearch nodes must be a parent directory of the setting you configure here:
  path = "/mnt/automate_backups/opensearch"
  
  [global.v1.backups.filesystem]
  path = "/mnt/automate_backups/backups"
  

• Patch the automate.toml config to trigger the deployment from the provision host.

  chef-automate config patch --fe automate.toml
  

Backup and Restore

Backup

To create the backup, by running the backup command from bastion. The backup command is as shown below:

chef-automate backup create

Restore

To restore backed-up data of the Chef Automate High Availability (HA) using External File System (EFS), follow the steps given below:

• Check the status of Automate HA Cluster from the bastion nodes by executing the chef-automate status command.

• Execute the restore command from bastionchef-automate backup restore <BACKUP-ID> --yes -b /mnt/automate_backups/backups --airgap-bundle </path/to/bundle>.

Troubleshooting

Try these steps if Chef Automate returns an error while restoring data.

1. Check the Chef Automate status.

   chef-automate status
   

2. Check the status of your Habitat service on the Automate node.

   hab svc status
   

3. If the deployment services are not healthy, reload them.

   hab svc load chef/deployment-service
   

Now check the status of the Automate node and then try running the restore command from the bastion host.