Message Queues

• Suppose you have bunch of requests coming in….

• Now our server, db, etc is having issues keeping up with all of these requests

• So what can we do? Well we could horizontally / vertically scale our servers, but thats not always cost effective.

• Similarly, if we don’t need to respond to these queries ASAP, we can store them in a message queue.

• Basically this works as que-ing up requests for it to be handled asynchronously (on your own time)

• These events/requests are sent to a new service, such as Message Queue which can hold it off in a line until your server has capacity to deal with it, or deal with it on your own time.

  

• Essentially we have DECOUPLED these events… (everything that is stored in a Queue, is actually stored in a DISK, so it’s Durable!) now these events can be processed in our own time, so its de-coupled. Each system can act as it’s own entity (app, app server)

• NOTE that this only works for events that doesn’t need realtime answer / quick response right away!

• So How Does Message Queuing work? There are 2 main models

  1. Push/Pull model
     1. Pull-Based Model: In this scenario, the application is responsible for monitoring the message queue for any new messages. If new messages are present and the app server has the capacity, it "pulls" from the queue. This approach can be more efficient in terms of managing the server-side load. However, if the queue is empty, it may introduce latency.
        1. Once a message is sent, the server can Ack the queue to let it know that the message has been recieved!
        2. This way we don’t lose messages
     2. Push-Based Model: In this case, the queue takes on the responsibility of pushing messages to the server. But, this strategy might overload the server if the rate of incoming messages is excessively high.

  https://imagedelivery.net/CLfkmk9Wzy8_9HRyug4EVA/bf047eb2-5a84-4d65-e857-2363d38e1f00/public

  The visual above demonstrates the pull architecture.

  https://imagedelivery.net/CLfkmk9Wzy8_9HRyug4EVA/02015d11-e071-4a6e-d953-7e1d66430700/public

  The visual above demonstrates the push architecture.

  When the message queue dispatches a message to the application server, the consumer or the server sends an acknowledgement after successfully processing a message (similar to what we discussed in the networking section).

  If the queue does not receive an acknowledgement for a message within a specific time frame, it can infer that the message was not processed, prompting the queue to resend it. This approach, akin to what we discussed in the networking chapter, ensures message delivery, even in the event of temporary server issues.

• We can also have multiple apps feeding into a queue, and multiple applications that are reading from the queue (many to many relationship) → PUBSUB

Pub/Sub

• The above mentioned is the second model, called PubSub (Publishers / Subscribers)
• Here, the publishers are ones who are sending requests, and subscribers are the ones that are receiving and storing those requests
• This is where the queue really comes into play, as we are able to have custom logic on the queue itself… They are designed to store, route, and deliver messages between different parts of a distributed system.
• To ROUTE the messages, messages are categorized into topics. Producers (publishers) send messages to specific topics rather than directly to queues. Then the Message queue can route the certain topic to CERTAIN subcribers (doesn’t have to be all of them)

Let's take an example using Apache Kafka, a popular distributed streaming platform:

• Topics in Kafka:
  • Messages are published to specific topics, such as "Orders," "Payments," or "CustomerUpdates."
• Consumer Groups:
  • Consumers subscribe to one or more topics based on their responsibilities. For instance, there might be a consumer group for processing orders and another for handling payments.