Amazon SQS is a fundamental service for building decoupled and scalable cloud applications.
Amazon SQS is a message queuing service. It is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.
SQS facilitates decoupling and buffering workloads for asynchronous processing. It allows consumers to pull messages when ready, preventing single points of failure and enabling applications to handle unpredictable and intermittent traffic without downtime.
SQS is used for building new applications and continuously processing messages with unpredictable and intermittent traffic, prioritizing cost-effectiveness and avoiding downtime.
SQS offers different queue types tailored for various messaging requirements, primarily differing in message ordering and delivery guarantees.
SQS Standard Queue
Standard queues offer high throughput and best-effort ordering. They provide at-least-once delivery, meaning a message might be delivered more than once.
throughput:
High
ordering_guarantee:
Best-effort
delivery_guarantee:
At-least-once
Use Cases:
- General-purpose message queuing where exact message order is not critical.
SQS FIFO Queue
FIFO (First-In, First-Out) queues guarantee strict ordering and exactly-once processing. When paired with AWS Lambda, they provide a serverless, low-operational-overhead solution for real-time, ordered processing.
ordering_guarantee:
Strict (First-In, First-Out)
delivery_guarantee:
Exactly-once
deduplication:
Yes
throughput:
Lower than Standard queues
Use Cases:
- Processing ordered event data where order must be maintained.
- Scenarios where message ordering and deduplication are critical.
Understanding these operational details is crucial for efficient and cost-effective SQS implementation.
Messages can be retained in SQS queues for a configurable period of up to 14 days.
Technical Specs: Up to 14 days
The message visibility timeout is the length of time that a message is hidden from other consuming components after a consuming component retrieves it from the queue. This prevents multiple consumers from processing the same message.
SQS supports both short-polling and long-polling. Long-polling is preferred over short-polling (0 seconds) to avoid unnecessary charges from empty pulls, by waiting for messages to arrive.
SQS queues are not bidirectional; consumers cannot be producers.
Dead-Letter Queues (DLQs) are used for saving failed messages from both SQS and SNS. They store messages that a source queue or topic couldn't process successfully after a specified number of retries.
Access to SQS queues is controlled using IAM permission policies and Queue Access Policies, which define who can perform actions like publishing, sending, and receiving messages.
SQS integrates seamlessly with various AWS services to build robust, event-driven architectures.
SQS messages can be processed asynchronously by AWS Lambda functions, which can automatically scale to meet demand. This is a common pattern for processing data from SQS with minimal operational overhead.
S3 events, such as object creation or removal, can trigger workflows by sending notifications to SQS queues. This allows for asynchronous processing of S3 object events by backend applications.
Elastic Beanstalk worker environments can host applications that read messages from an SQS queue. The worker environment's autoscaling group can scale based on the queue depth, ensuring efficient processing of messages.
When designing with SQS, several factors influence the choice of queue type and operational configuration.
SQS inherently supports asynchronous processing, allowing producers to send messages without waiting for consumers to process them. This decouples components and improves system responsiveness.
Technical Specs: SQS is asynchronous
SQS utilizes a pull-based messaging model where consumers retrieve messages from the queue when they are ready to process them. This gives consumers control over the rate of message consumption.
Technical Specs: SQS is pull-based
The need for strict message order and protection against duplication is a critical factor in deciding between SQS Standard and SQS FIFO queues.
The expected application load and throughput requirements should be considered, as SQS FIFO queues have lower throughput compared to Standard queues.
AWS offers multiple messaging and streaming services, each with distinct characteristics for different use cases.
Unlike SQS, Amazon SNS (Simple Notification Service) is a push-based messaging service used for fanout architectures and alerts. SNS does not guarantee message ordering when distributing messages to subscribers.
SQS is a simple messaging service with low configuration requirements and is not designed for real-time delivery. Kinesis, on the other hand, is a more complex platform for big data applications, providing real-time communication for decoupling and streaming data.
While ElastiCache can be used for messaging queues, SQS is the dedicated fully managed service for this purpose.
Glossary
Amazon SQS (Simple Queue Service)
A fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.
SQS FIFO Queue
A type of SQS queue that guarantees strict message ordering and exactly-once processing.
SQS Standard Queue
A type of SQS queue that offers high throughput and best-effort message ordering, providing at-least-once delivery.
Message Visibility Timeout
The period during which a message is hidden from other consumers after it has been retrieved from the SQS queue, preventing duplicate processing.
Long-polling
An SQS polling mechanism where the consumer waits for messages to arrive in the queue, reducing empty responses and unnecessary charges.
Short-polling
An SQS polling mechanism where the queue is immediately checked for messages, potentially returning empty responses and incurring more requests.
Dead-Letter Queue (DLQ)
A queue that receives messages that a source queue or topic (SQS or SNS) could not successfully deliver or process after a maximum number of retries.