add the ability to create internal queues as quorum type instead of hard-coded classic

[Michalis-Apostolou]

Need

We need to upgrade the rmq cluster to version 4+. in this version High Availability (HA) policy with mirror queues has been deprecated and completely removed https://www.rabbitmq.com/docs/3.13/ha . So we need to replace all classic queue definitions with quorum

Problem

Rabbit-queue creates internal queues to implement scheduled publishing for a message

1. there is a queue created on app connect {prefix}_delay_reply
2. The publisher for a delayed messaged for XXX amount creates the queue {prefix}_delay_{XXX}

Example:

Since the library does not define the type of the queue , the are created to what is default on the virtual host.

If can't/don't want to change the default queue type of the virtual host , you should have the ability to define if these queues should be created as quorum or not.

Solution

By utilizing a new option available in the package scheduledPublishQueuesAsQuorum on rabbit config, this will passed around during the creating of the queues.
In order to avoid conflicts between already created classic queues and newly created quorum queues , the naming convention changes to

1. {prefix}_delay_quorum_reply
2. {prefix}_delay_quorum_{XXX}

[iliasbibas]

I think we need to look a bit deeper on how quorum queues handle dead lettering, a policy might be needed if we need the messages to end up in a DLQ instead of silently dropped.

[Michalis-Apostolou]

I think we need to look a bit deeper on how quorum queues handle dead lettering, a policy might be needed if we need the messages to end up in a DLQ instead of silently dropped.

@iliasbibas Are you referring to enable the at-least-once guarantee ? https://www.rabbitmq.com/docs/quorum-queues#activating-at-least-once-dead-lettering. to override the default at-most-once ?

[iliasbibas]

I think we need to look a bit deeper on how quorum queues handle dead lettering, a policy might be needed if we need the messages to end up in a DLQ instead of silently dropped.

@iliasbibas Are you referring to enable the at-least-once guarantee ? https://www.rabbitmq.com/docs/quorum-queues#activating-at-least-once-dead-lettering. to override the default at-most-once ?

Yes, for the delay queues I believe at-least-once and overflow: reject-publish should be set so that the delayed job end up in the delay_reply queue (I think that the delay_reply is already set as the target for these messages).

[iliasbibas]

Btw @Michalis-Apostolou , the _quorum_ addition in the queue name should perhaps be optional right? Otherwise it break compatibility, ie will doubly create queues even if an existing quorum queue without the name addition exists.

[DimitrK]

💭 hold the naming order of quorum until SREs verify what regexps are using for alerts on delay queues (cc @a-margaritis )

[ikarachristos]

Checked Datadog: no monitor names delay queues. The only ones that cover them are SRE's per-queue length catch-all (6080758) and the unacked-rate anomaly monitor (112880512), and both pick up <app>_delay_quorum_* automatically. @a-margaritis, can you confirm there's nothing on the CloudAMQP side (alarms or policies) that matches _delay_ by name? If not, I think the naming hold can be lifted.

[Michalis-Apostolou]

Btw @Michalis-Apostolou , the _quorum_ addition in the queue name should perhaps be optional right? Otherwise it break compatibility, ie will doubly create queues even if an existing quorum queue without the name addition exists.

it was intentional , because if there was already existing a classic queue with the same name. and you opt-in to create the same queue with different config (i.e. quorum queue type) it will conflict ungracefully and will cause to disconnect.

Then there is the issue on how to resolve such conflict. you can't update a queue config. you have to destroy and recreate the queue. During the recreation of the queue in a high traffic topic may lead to message loss

[iliasbibas]

Btw @Michalis-Apostolou , the _quorum_ addition in the queue name should perhaps be optional right? Otherwise it break compatibility, ie will doubly create queues even if an existing quorum queue without the name addition exists.

it was intentional , because if there was already existing a classic queue with the same name. and you opt-in to create the same queue with different config (i.e. quorum queue type) it will conflict ungracefully and will cause to disconnect.

Then there is the issue on how to resolve such conflict. you can't update a queue config. you have to destroy and recreate the queue. During the recreation of the queue in a high traffic topic may lead to message loss

I understand that, I'm saying that in a cluster that already has a quorum queue example_queue (e.g. vhost default) , creating a new example_quorum_queue by default will not be correct.
Also an introduction of new naming conventions will require updating the existing policies, most of which rely on regex.

[ikarachristos]

@iliasbibas revisited the naming now that #81 is merged. #81 only tolerates a queue type or priority mismatch, and only when you opt in. Quorum delay queues also declare x-overflow: reject-publish, which isn't tolerated, so reusing the name would make publishWithDelay fail until each queue is converted. Converting a delay queue in place also re-publishes its messages, which restarts their delays. New names let old and new pods run side by side, and the old queues empty out before we delete them.

Your duplicate case only applies where <app>_delay_<ms> is already quorum. rabbit-queue declares delay queues without a type, so that only happens on a vhost whose default queue type is quorum, and there you can leave the option off.

On policies: none of the policies on the sourcing brokers target delay queues by name. The HA catch-all covers the new names, and quorum queues ignore its ha-* keys.