Troubleshooting RabbitMQ

RabbitMQ Service Disk Full

Should the situation arise where the RabbitMQ service cannot be started or fails shortly after startup, please check the RabbitMQ log file, which is located in the following folder path:

<Prognosis_Home>\Server\Configuration\Messaging\RabbitMQ\log\rabbit@<hostname>.log

=ERROR REPORT==== 2-Mar-2018::14:31:45 ===
** Generic server <0.9892.0> terminating
** Last message in was {notify_down,<0.9889.0>}
** When Server state == {q,
{amqqueue,
{resource,<<"/">>,queue,

When the RabbitMQ service is not running, the following items will be affected:

  • All messages (data) sent via RabbitMQ will be lost since the problem occurred.

  • All messages stored persistently by RabbitMQ will be lost, these are RabbitMQ messages that have not been delivered before the RabbitMQ service shutdown. This data is lost because RabbitMQ's persistent store is corrupted.

  • In the case of Skype for Business, any messages that have not been delivered and saved into the Postgres database will be lost.

  • It will not be possible to use Skype for Business to query Conversations or Session Details.

Correction Procedure

The following manual steps can be used to remove all persistent messages and set RabbitMQ into an empty state, while still keeping all existing configurations.

Go to the following folder path:

<Prognosis_Home>\Server\Configuration\Messaging\RabbitMQ\db\rabbit@<Hostname>-mnesia

Where <Hostname> is the name of the server.

Delete the following three folders:

  • queues
  • msg_store_transient
  • msg_store_persistent

Restart the PrognosisRabbitMQ service.

RabbitMQ Certificates Expired

It is possible that the self-signed certificates created by the installer for RabbitMQ service may expire, which will lead to TLS issues when the Prognosis Server attempts to use RabbitMQ.

If a certificate has expired, check for the following error in the RabbitMQ log file:

<Prognosis_Home>\Server\Configuration\microservices\logs\aggregation-service.log

connection failed - amqps://localhost:567 Certificate has expired

The following steps should be followed to create RabbitMQ certificates for use on all affected servers.

On the Global Management Server, start a 'Command Prompt' (with Administrator privileges)

Change to the following folder: <Prognosis_Home>\Server\Configuration\utils\rabbitmq

Run the command:

C:\Prognosis\Server\Configuration\utils\rabbitmq> generatecerts.bat <Prognosis_Home>

This will generate the following files in the folder: <Prognosis_Home>\Server\Configuration\Messaging\RabbitMQ\keys:

  • ca_certificate.pem (type=ssl_ca)
  • client_key.pem (type=ssl_private)
  • client_certificate.pem (type=ssl_public)
  • server_key.pem (type=ssl_serv_private)
  • server_certificate.pem (type=ssl_serv_public)

Continue to Installing RabbitMQ Certificates below.

Installing RabbitMQ Certificates

The following steps should be followed to re-issue either custom or generated certificates on all affected servers.

Start the Web Application and go to the Administration tool, select the Global Management Server from the Server Navigation panel.

In the Configuration section, click on the MESSAGING Configuration

Update the ADD KEY 'content=' values to contain the text from the files created in Step 3. Use the' type=' mapping above to ensure the correct content is placed in the corresponding KEY, this is an example of the resulting ADD KEY entries:

ADD KEY( name=prognosis, type=ssl_ca, content= 
-----BEGIN CERTIFICATE----- 
MIICzDCCAbSgAwIBAgIJALZa/gckBGsKMA0GCSqGSIb3DQEBCwUAMBYxFDASBgNVBAMMC1Byb2dub3Npc0NBMB4XDTE4MDkyNzEyMzY1NVoXDTIzMDkyNjEyMzY1NVowFjEUMBIGA1UEAwwLUHJvZ25vc2lzQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCp9eUeA/p9d8GW8w+ed/qxmKV4yNxKR4kzcpAExqqfxrwbnol04brC3IHUkPsF7CM9uaHV/jn+NKClP7U7NosDRQ0N/s5kLXW3LjZox58IpwNQyrL7F5z8FVzjafde4uctNAkq9dMxOzolgryL4lgP7EIPEgM3Z97MdZifauaAm6dLDJDvH6RYbHIrt70RRXA/xMSph8vthNnK5+dJ+8FjNaOQLrez8U0qvdh+rL/MOfLMpS6ocqS3oHxKVQ+P102nSpkULCvyCAo9eFwfo/Frr2fZ3daby8qoF/VcwgGUSg9gs3BlcmFQ5Mp/+TnFZBod4aj0hZyVaCng0azWhffTAgMBAAGjHTAbMAwGA1UdEwQFMAMBAf8wCwYDVR0PBAQDAgEGMA0GCSqGSIb3DQEBCwUAA4IBAQB9y/uXlXi6JDm8ZMiGkCDYtnTa9MIB+Y/CDc+MS0MTY1FPV+T7ueRM9xWfPMtpwvz1YINR2tPbGHet6opVAgiPiT0t/1eUgTQHxoQGYIcrKLUZjDKHC23UTnWlJvLwSdGF2YlYO0vK0Pm9XLQ5M4SMztm56O+mmf+sk3XZUCuSiOrv+m7k+ntINifpQjiBBRpVUVhGZLIBl3Mkx9SKia5esYO/CI/6yDDdLpJVbupeUdNkpXDw5a/zmdXhZ45HdQl+cNYyl7cCdTb3IZ2y4PdB5MBh8vrM1VdexOkeBOVXAwdB79UpLEqld3QoeDwqvTZwFoNzlVr3ltI6ajpOyiiI 
-----END CERTIFICATE-----) 
ADD KEY( name=prognosis, type=ssl_public, content= 
-----BEGIN CERTIFICATE----- 
MIIC7zCCAdegAwIBAgIBAjANBgkqhkiG9w0BAQsFADAWMRQwEgYDVQQDDAtQcm9nbm9zaXNDQTAgFw0xODA5MjcxMjM2NTdaGA8yMDIyMDEwMTAwMDAwMFowLTESMBAGA1UEAwwJVk0tQ0stVFNUMRcwFQYDVQQKDA5SYWJiaXRNUUNsaWVudDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL8tbAXPeDwp1f/OM3KsHJ1Qsv2wVauYzLt/BEN/E3wCjoJ+6DeHTqp2xvkxjAxo9WKxk2E87ewvneWW5Eh48Sl6hjGJdciHj1WnUCpokeVAQYJY5qkkYPFN/M8vfGomgCUNpG4dDttQz/CMNyi3UQi9BZbMUt0KwqXLH6xO+OHXJxXTa3uPX7pD357wXuR7DJq4QLYOwIbWweWydXHDTLhJ5mRYhRFUAzd/a98eyxvFntNg+m5y+v9jbImTj6I5kKSYTITxjwo9mQrM90i5J0ynmOvkGEyOxWCtoe2qYL6110RNjQzRPl6wyev1tpFyoEunlWKBogRlTSJPlO4IVnUCAwEAAaMvMC0wCQYDVR0TBAIwADALBgNVHQ8EBAMCB4AwEwYDVR0lBAwwCgYIKwYBBQUHAwIwDQYJKoZIhvcNAQELBQADggEBAJjoWC+gBl/kxkNcuZURcnv8029uCLijaqC280oRl4Fo3HIyMqJsKaBOks5zvLd0tqGW2o9KazoTQbG3Y3sllpY7Tme8QmEBE9mNynSjbEgJ+v+LcikiiQOlEpEfSLGhNmTJ/5HYas0wzJTbBKKERYdx+4WxXBfqjUfg90OjUXTLGyRICoqlydFE1aViBJMW+8Zm+7oCCz3+1F9iW25zM9QTG2hVxIFBTOG6l7Oq+sa2TuhgqGKwZedjNptjRbxGxAa41eaTl+/1m/otrF2Gitg0WZvRTd0Roxh78aCRRQPzGKV5+qvw/elXSH3W7yrNQ3MVYsd7KhotCbK2f8PgZGY= 
-----END CERTIFICATE-----) 
ADD KEY( name=prognosis, type=ssl_private, content= 
-----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEAvy1sBc94PCnV/84zcqwcnVCy/bBVq5jMu38EQ38TfAKOgn7oN4dOqnbG+TGMDGj1YrGTYTzt7C+d5ZbkSHjxKXqGMYl1yIePVadQKmiR5UBBgljmqSRg8U38zy98aiaAJQ2kbh0O21DP8Iw3KLdRCL0FlsxS3QrCpcsfrE744dcnFdNre49fukPfnvBe5HsMmrhAtg7AhtbB5bJ1ccNMuEnmZFiFEVQDN39r3x7LG8We02D6bnL6/2NsiZOPojmQpJhMhPGPCj2ZCsz3SLknTKeY6+QYTI7FYK2h7apgvrXXRE2NDNE+XrDJ6/W2kXKgS6eVYoGiBGVNIk+U7ghWdQIDAQABAoIBADWfOeNPU/wcB+geIBQT7lq5ndrGMOFWlGYqtDlyVTa9SNeVXPDmRMxPUHPoz81TSzCJejsV6e4LVRBmiFVDhvEJasZONfku/KcbrMKYXgOdSEKVSO08ROyp4TUDrrPfZKX/85TnFZmySwC/c2E6OcJnsj+fw8Y4FGyIUrPz2p8NSxHC+mPM/3mbpj8YOP1psj3HIK2utKOyC6xGNf/FjT8DmPzi8rQsMeQnHN0cVQBBY9amL7tiFLi0zvZUEEDxXyjte8MmCWc6nou2QoZkvPCXZTgzek/zWu1vPnxRP21cuWxlHmCAijyzlgZ1cbQcx3BjFSCYsRCKUjEqv99lWqkCgYEA9t0SFffUan9XbQL0At148vdFzeDhi3uS3mF5tRgaJkEYDlX0jSuzSqG26rmuJ7M7+ZKvFF7feVeXN/R7iOwbISb4TQ1bjQcxH/gfzagpM3eW2GMK2Rh/wC6bh1Lk6weN3ZeO5C7GTW0KYCz+AxOaAiKK4PYF7l4TZ2LKjA2YnjcCgYEAxkC/muLp5qBnz+haZCNFEHtnGxx9XVeaBSGJ98dTWQoU2T9GJKRFeP23xz/+0/n0Wa+iaCt1wfU1aXznIrCjElhSBt4IiHhd3Q/S+/BJ8D+grENRJfnn0YRDdg2dRWGk8Tg6CgZq1HXmcBFrvGIKtFDLve0cyEWYBZ6Ve0Sw+rMCgYEA3z0Beu43n+ryI3l1MhYXpxfnfwsqy1j/cqyRRCu8kfVxjJ08XmEzbH0bDd7F187koId+XhyBWJjV/drPz/07d1pnVj6LGNh1GyTllbPJNmqh9aJzD+lwtVkwekqDCus05juFE2IdH61IobpuJlgWgHG+RzsUF//o6ihTZB1zczcCgYEAm1YkSyG8j33jmV2/ZXLVLILkWLPL7pUzEw4iYkiFn+BsUYRoW1FzNcfmdXJJIqJYuOUK5YAX9m9A+HRrTh0EyNz2+HaETRmywgEiGVE/PcU9CXx9Wroiz2FI0nj8X16QtIiPwUch8XKnqeucdLCFHp+n5BqjULBm2xBUo49A9E0CgYATCyRWMumidMOhMbTDJXMB3Tuhke1ptAS3+xwYSG0i9z5fxKigmNcTO4G/RqsgbG10RbvrBwpZWz97B05PDabXtZH2v+Yo0KwJZRJFmK/wsiNOHFrptetRvsEEIrLZpjIZroqF+z4kRzI6vdYdKuaeda2l201BsNVlcZd+TjYMBA==     
-----END RSA PRIVATE KEY-----) 
ADD KEY( name=prognosis, type=ssl_serv_public, content= 
-----BEGIN CERTIFICATE----- 
MIIC7zCCAdegAwIBAgIBATANBgkqhkiG9w0BAQsFADAWMRQwEgYDVQQDDAtQcm9nbm9zaXNDQTAgFw0xODA5MjcxMjM2NTZaGA8yMDIyMDEwMTAwMDAwMFowLTESMBAGA1UEAwwJVk0tQ0stVFNUMRcwFQYDVQQKDA5SYWJiaXRNUVNlcnZlcjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALgQecCSZ5JkbQkZ+zcfc7hJv3IbJEHOwL4sOYks/FfZxyuuZieDSzL+St7Ww2r8yGPj91Z0DYH/o+ouxn9/vhZPdD/DAD+sSyb99QhTR+To69n9JG21BAZnux9WDhOxfsLp/fqGY3IOz/dBzrRnPAFPfeCbqhT5dhZdvv5KBxOAhXVtbP6MIgKUPuHLXk7a5g9tXIPBfJkRhrhYWUdBSWnePGObOUhvIVwNZIkIuE+1RfnB6GX7BvlED35NnDhqLppLgcrDl1M4QONPePjDkj0rMaTrA0Dd6B8FaE0N6XpI9tfHIGjmxpE8u8R9yFKXWLMQuy7vXXSoolMzSQERXFcCAwEAAaMvMC0wCQYDVR0TBAIwADALBgNVHQ8EBAMCBSAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDQYJKoZIhvcNAQELBQADggEBAHg/hleeScXGSL4mYqnVtTxMK7FkeSkXhh0wF0Z3hGmkKahjnCFD6OHHPTQGS7aur3CbeH5Jk8PvtmuFzY/oX02AtqdOG2lffHej5/9pBEwNI19gYits5L9zE1OwUncyAYUM2eDTQw0B6F6hvnkN9xGs8m3/FaiWUEMG74anzkfgDlPX9/Ia3MbAZlzz0c0WQcG+wZlLlqoF4xsfPyNVbt5MzqT/p1/dHJ8BinHe4wkLfT3FoYrQ29mQfb6UCCVknls5KpSOQ3wqWOqolR+AZuscV+oxWDJV3P7E2vCqBLv1k97lFp+6Cl1Y1QxK5jS+3B+/RuYqd4iSqxFI15V54f8= 
-----END CERTIFICATE-----) 
ADD KEY( name=prognosis, type=ssl_serv_private, content= 
-----BEGIN RSA PRIVATE KEY----- MIIEowIBAAKCAQEAuBB5wJJnkmRtCRn7Nx9zuEm/chskQc7Aviw5iSz8V9nHK65mJ4NLMv5K3tbDavzIY+P3VnQNgf+j6i7Gf3++Fk90P8MAP6xLJv31CFNH5Ojr2f0kbbUEBme7H1YOE7F+wun9+oZjcg7P90HOtGc8AU994JuqFPl2Fl2+/koHE4CFdW1s/owiApQ+4cteTtrmD21cg8F8mRGGuFhZR0FJad48Y5s5SG8hXA1kiQi4T7VF+cHoZfsG+UQPfk2cOGoumkuBysOXUzhA4094+MOSPSsxpOsDQN3oHwVoTQ3pekj218cgaObGkTy7xH3IUpdYsxC7Lu9ddKiiUzNJARFcVwIDAQABAoIBAH+uBhb8N9ZHzY7zs9yf97DNj3zCsRrpJbCFkv/B8eWqzstRNdGQ9NgRp7P+fB457XPFTextonUb9QNcoRsObLijMvuL9B83EAajQZcd1hBHNlvHir6DQ6CB0j7BfhBRHqJsmF5xXIhhGAfnYVCpD+nEz4e/03KrPSv0qalRhqkSX8R4+m56EOF0/kw/G6WDEbdqeGQTDl5A9NALywoD237WdkfvRccLr/0D6hhPbtmxrIDpVuLOUHfdnMcEwcK8cCmjW6riJmqJGuO5nRfAWOu+9ArBihZ2N1pd+3IVl4p9rdb7Kh/76WO6iTPwOCWTA8HFIuoZYk9La2bVQUij+fECgYEA3igzsB2pMHeMFLF68GI1qF3fRxnSaV5SOMHHi6FYmVK6MCkjElXzDGdx752DLzAA54mFYc7MOrYMXeTyVJ6Pg3Z1S0HB35IOorBIhA9RPIgGpXWO5jXSzemmTNg7t4CICaQNDrStXvAmQsY+VU80k7NlXKHPtcq2gfL77MnkVD8CgYEA1Bq3EPvHP00oL2hR3x/7KRppnIgJKAYVHMZMFdXkmjQhZoLd14ulJPYG4Hr04tbj47VwaxwG/rp2oUsNPPX8MQ4dQpn67MS38yWdz9tGqquta7YlG9f7ysy1SFYtBeCQY682oXZKxmMyGBPYxPzensrPV92etwzTWRj2Am03kekCgYBRjF5U4r5tbSS9n9yqCtL5h+PjHKZxSU9TSEawibyVI5n2/dLC80vxnLNlKuOx2aQrWf7dRWUQlCz79oPB7iZuMnih9HRvVhULQTyNRXYjiycta3bWKKHGNmN2R9W76XGGYHLumAIgHHVfA5h0Bkwdf7fQhhFSASR3lTNG22F2TwKBgDXvSSfnbkYZf6N2eoMrZbmQ/tQP/4wO5+V2Qzi2ODhCE1A27OR3nRnl+d+xRH4+OJrZs2l5KgoyLiJbrFUZE/UBas1ZzGXSrBgHM7b8Ql754tJl8c+/v0ZVga+QQkihTvYaQuER7HfAF6fpvBBzYMSuO8VTFJ3lhAS/BSee0CepAoGBANKCPohK9wHTJcLEomicckzTr7e1YuNvijv2BtWG+8Ho11q+unzI+dN7WKFVZOiD8uNSmYg2L2YPjIDsGy+W+ITRZsQdBWLQqLnR+lyBILW3YgBNzeXVxZy+zAkcSZxJC/IxgNmfMrVeKt1UXtnHZYlGFzBdk8cGj7sgPWOgU452 
----END RSA PRIVATE KEY-----)

Click the Start On RabbitMQ Nodegroup button, this will apply the new certificates on all the servers which use RabbitMQ service.

On all servers in the RabbitMQ Nodegroup, the following processes and services will need to be restarted:

  • irMSMGS.EXE
  • irSFBCSM.EXE
  • irALERTIMP.EXE
  • PrognosisRabbitMQ Service
Provide feedback on this article