Splitting an existing PEM server v10.2

You can split an existing PEM installation into two installations. Splitting is useful if your original PEM instance has become overloaded and you want to split the load over two servers. It can also be useful if you're restructuring your database estate and want to take a segmented approach to monitoring.

You can split an existing PEM instance into three or more instances by splitting each new instance. You can also set up a new PEM instance and move a portion of the monitored servers and agents to the new instance.

Install PEM on the new host

1. Install the PEM server package on the new host.

2. Configure it by running the PEM configuration script.

Copy all data from your existing PEM instance

On the new PEM host:

1. Stop the Postgres and PEM agent services:

   → WrapCopy

   systemctl stop postgresql-16
   systemctl stop pemagent

2. Delete the Postgres data directory:

   → WrapCopy

   rm -rf /var/lib/pgsql/16/data/*

3. Clone the the existing PEM data in your new instance:

   → WrapCopy

   su - postgres
   pg_basebackup -h <OLD_PEM_IP> -p <OLD_PEM_SERVER_PORT> -U postgres \
   -D /var/lib/pgsql/16/data -Fp -Xs -P

Update the pg_hba.conf file on the new PEM instance

At this point, the pg_hba.conf file on the new PEM instance is a copy of the one on the existing PEM instance. If any entries reference the external IP of the existing PEM server (for example, host ... <old_ip> ...), modify them to use the new PEM server IP address or localhost/127.0.0.1, as appropriate.

You can now start the Postgres and PEM agent services. (You might need to restart them again later.)

```shell
systemctl start postgresql-16
systemctl start pemagent
```

Configure SSL on the new server and reconfigure PEM agents

How you configure SSL depends on how you want to manage SSL certificates. From the four approaches, select the one that meets your needs.

All of these scenarios are viable if you're using unique agent users or a single common agent user.

Approach 1 – Reuse certificates from the existing PEM instance

Use this approach if you want to use the same server certificates for both PEM instances. This approach requires no changes to any SSL settings or files.

1. Copy the PEM agent certificates from the old PEM instance to the new PEM instance. (By default, these are located in /root/.pem on Linux and C:\Users\\AppData\Roaming\pem on Windows.)

2. Restart Postgres and the PEM agent services on the new system.

3. For all agents you want to migrate to the new PEM instance, update the pemhost parameter in the agent.cfg file of all agents to the address of the new PEM instance.

4. Restart the affected PEM agents.

Approach 2 – Use different server certificates but the same CA

Use this approach if you want to have different server certificates on the two PEM servers and want to allow both PEM servers to act as a single certificate authority.

Using this approach, agent certificates issued by either PEM server will be signed identically and trusted by both servers. This is convenient as it avoids the need to replace any agent certificates while still ensuring the servers have distinct identities.

1. Back up the existing server.crt and server.key located in the Postgres data directory.

2. Generate a new server.key and CSR for the new PEM server.

   → WrapCopy

   openssl genrsa -out server.key 4096
   openssl req -new -key server.key -out server.csr -subj '/C=your_country/ST=your_state/L=your_locality/O=your_org/CN=common_name_for_server'

3. Sign the CSR, and generate a server certificate with your existing CA certificate.

   → WrapCopy

   openssl x509 -req -days 365 -in server.csr -CA ca_certificate.crt -CAkey ca_key.key -CAcreateserial -out server.crt

4. Change the owner and the permissions on the generated server.crt file.

   → WrapCopy

   chown postgres /var/lib/pgsql/16/data/server.crt
   chmod 600 /var/lib/pgsql/16/data/server.crt

5. Restart Postgres and PEM agent services on the new system.

6. For all agents you want to migrate to the new PEM instance, update the pemhost parameter in the agent.cfg file of all agents to the address of the new PEM instance.

7. Restart the affected PEM agents.

Approach 3 – Replace all server and agent certificates

Use this approach if you want to have completely distinct server certificates and certificate authorities for the two servers. Using this approach, a given agent can authenticate only with its "parent" PEM server. The two PEM instances will have no SSL configuration in common.

1. Follow the instructions for regenerating the server SSL certificates to regenerate the following files:

   • server.crt
   • server.key
   • root.crt
   • root.crl
   • ca_certificate.crt
   • ca_key.key

2. Restart Postgres and PEM agent services on the new system.

3. For all agents you want to migrate to the new PEM instance, update the pemhost parameter in the agent.cfg file of all agents to the address of the new PEM instance.

4. Regenerate agent certificates for each agent user and copy the new certificates and keys into place. You must also regenerate the certificate and key and for the agent on the PEM server.

5. Restart the affected PEM agents.

Approach 4 – Use only certificates from an external CA

Use this approach if you exclusively use SSL certificates provided or signed by an external certificate authority.

Because your existing agent certificates were issued by a trusted authority that's included in the Postgres server CA cert, both PEM servers can validate all your agent certificates, meaning all agents are already able to connect to either PEM server. You need to request a new server certificate only for the new PEM server.

1. Obtain a new server certificate and key from your provider, supplying the details of the new PEM instance.

2. Replace server.crt and server.key (or whatever files are currently specified by ssl_server_cert and ssl_server_key) with your new certificate and key respectively.

3. Restart Postgres.

Disable disconnected agents on each PEM instance

You now have two PEM instances, with your existing agents split between them. However, both instances contain records for all agents. Clean up the PEM instances to reduce wasted resources and false alerts by marking the agents that aren't connected to each instance as inactive.

In the PEM web application, delete the agent and its monitored servers. In Object Explorer, right-click each one and select Remove.

Alternatively, you can perform this action through SQL. Execute the following on the pem database:

UPDATE pem.agent SET active = false WHERE id in (list_of_ids);
UPDATE pem.server SET active = false WHERE id in (list_of_ids);

Validate your PEM instances

Open the PEM monitoring dashboard of the new instance.

• Confirm that only the required servers and agents are visible.
• Verify that these servers/agents show an active status and are sending metrics.
• Check the agent logs for any connection or authentication errors.