SQS & Kafka Messaging Quiz

SQS & Kafka Messaging Quiz

Test your knowledge on distributed messaging architecture, queues, and event streams.

1. 1. What is the primary purpose of a message queue in a distributed system?

   • To store data permanently in a relational database
   • To decouple producers and consumers so they can operate independently
   • To synchronize HTTP requests between microservices
   • To provide a shared in-memory cache for services

2. 2. In the producer/consumer model, what is the role of a producer?

   • It reads and processes messages from the queue
   • It manages the queue infrastructure and routing rules
   • It sends messages to the queue for downstream processing
   • It monitors queue depth and triggers auto-scaling

3. 3. What is Amazon SQS?

   • A fully managed relational database service from AWS
   • A fully managed message queuing service from AWS
   • An open-source distributed streaming platform
   • A serverless compute engine for batch jobs

4. 4. What is the key difference between an SQS Standard queue and an SQS FIFO queue?

   • Standard queues guarantee strict message ordering; FIFO queues do not
   • FIFO queues guarantee ordering and exactly-once processing; Standard queues offer best-effort ordering and at-least-once delivery
   • Standard queues support encryption; FIFO queues do not
   • FIFO queues are cheaper but have no throughput limits

5. 5. What does 'at-least-once delivery' mean in a messaging system?

   • A message is delivered exactly one time with no duplicates
   • A message may be delivered more than once, so consumers must handle duplicates
   • A message is delivered only if the consumer is online
   • A message is never delivered more than once per consumer group

6. 6. What is a Dead Letter Queue (DLQ)?

   • A queue that deletes messages automatically after 1 minute
   • A secondary queue where messages are sent after failing processing a configured number of times
   • A high-priority queue for urgent system alerts
   • A queue that stores only metadata about processed messages

7. 7. What is Apache Kafka?

   • A relational database optimized for time-series data
   • A distributed event streaming platform designed for high-throughput, fault-tolerant messaging
   • An AWS-managed in-memory cache service
   • A REST API gateway for microservice communication

8. 8. How does a Kafka topic differ from a traditional message queue?

   • A topic deletes messages immediately after they are read by one consumer
   • A topic retains messages on a log so multiple independent consumer groups can read the same data
   • A topic only supports a single producer at a time
   • A topic guarantees delivery within 1 second

9. 9. What is a Kafka consumer group?

   • A set of brokers that replicate partition data
   • A collection of consumers that jointly consume a topic, each reading from a distinct subset of partitions
   • An admin group that manages topic creation and deletion
   • A group of producers that share the same client ID

10. 10. Why is strict global message ordering difficult to guarantee at scale in distributed messaging systems?

    • Network latency and parallel processing across multiple nodes make a single global sequence hard to enforce without becoming a bottleneck
    • Ordering requires a relational database, which messaging systems cannot use
    • TCP connections inherently scramble packet order
    • Message brokers deliberately randomize order to improve performance

11. 11. What is the SQS visibility timeout and why does it matter?

    • The maximum time a message can stay in the queue before being deleted automatically
    • The period during which a received message is hidden from other consumers, preventing duplicate processing
    • The delay before a newly sent message becomes visible to consumers
    • The time window allowed for a consumer to authenticate before reading a message

12. 12. What is the advantage of SQS long polling over short polling?

    • Long polling compresses messages to reduce bandwidth usage
    • Long polling holds the connection open up to 20 seconds, reducing empty responses and lowering cost
    • Long polling guarantees messages arrive in FIFO order
    • Long polling allows consumers to push acknowledgements back to SQS in real time

13. 13. What are the default and maximum message retention periods for an SQS queue?

    • Default 1 day, maximum 7 days
    • Default 4 days, maximum 14 days
    • Default 7 days, maximum 30 days
    • Default 1 hour, maximum 24 hours

14. 14. How do Kafka partitions enable parallelism in message consumption?

    • Each partition runs on a separate AWS region, so consumers process data geographically
    • Each consumer in a group is assigned one or more partitions, allowing multiple consumers to read different partitions simultaneously
    • Partitions compress messages so each consumer processes twice as many per second
    • Partitions allow producers to write faster by batching messages into chunks

15. 15. What is the difference between a committed offset and the latest offset in Kafka?

    • The committed offset is the next message to produce; the latest offset is the last message produced
    • The committed offset is the last position the consumer group confirmed processing; the latest offset is the last message written to the partition
    • The committed offset is stored on disk; the latest offset is in memory only
    • They are synonymous terms used interchangeably in Kafka documentation

16. 16. What triggers a Kafka consumer group rebalance?

    • A topic's replication factor is increased by an administrator
    • A consumer joins or leaves the group, or a session timeout expires, causing partitions to be reassigned among active consumers
    • A producer sends a message larger than the configured batch size
    • The ZooKeeper leader election completes a new round

17. 17. What is the purpose of a Kafka replication factor, and what does ISR mean?

    • Replication factor sets message TTL; ISR stands for In-Sync Retention
    • Replication factor controls how many broker copies exist for each partition; ISR (In-Sync Replicas) is the set of replicas fully caught up with the leader
    • Replication factor determines the number of consumer groups; ISR tracks their lag
    • Replication factor sets the compression level; ISR is the index of stored records

18. 18. Which Kafka configuration achieves exactly-once semantics end-to-end?

    • Setting acks=0 on the producer and enabling auto-commit on the consumer
    • Enabling idempotent producers (enable.idempotence=true) combined with transactional APIs on producers and isolation.level=read_committed on consumers
    • Using a FIFO topic with a single partition and a single consumer
    • Disabling batching and setting max.in.flight.requests.per.connection=1

19. 19. What is the primary difference between Amazon SQS, Amazon SNS, and Amazon EventBridge?

    • SQS is a queue for point-to-point messaging; SNS is a pub/sub fanout to multiple subscribers; EventBridge is an event bus with routing rules and schema registry
    • SQS is for streaming data; SNS is for batch jobs; EventBridge is for IoT devices only
    • All three are identical services with different pricing models
    • SNS stores messages for 14 days like SQS; EventBridge does not retain events

20. 20. What is Kafka log compaction and when should you use it?

    • A background process that compresses log files using gzip to save disk space
    • A retention policy that keeps only the latest value per key, discarding older records with the same key
    • A Kafka Streams operation that merges two topics into one sorted topic
    • A process that deletes all messages older than the configured retention period

21. 21. What are the three files that make up a Kafka log segment and what does each store?

    • .log stores raw message bytes; .index maps offsets to byte positions in .log; .timeindex maps timestamps to offsets
    • .log stores offsets; .index stores message bytes; .timeindex stores partition metadata
    • .log stores consumer group state; .index stores producer IDs; .timeindex stores replication lag
    • .log stores compressed batches; .index stores schema IDs; .timeindex stores TTL expiry times

22. 22. What is a hot partition in Kafka and how does key-based partitioning cause it?

    • A hot partition is one that receives too many replicas; key-based routing distributes replicas unevenly
    • A hot partition receives disproportionately more messages than others because a high-cardinality key (e.g., a popular user ID) consistently hashes to the same partition
    • A hot partition is temporarily offline; key-based routing avoids it until it recovers
    • A hot partition stores compacted data; round-robin partitioning avoids compaction

23. 23. How does back-pressure manifest in a Kafka consumer chain and what is a common mitigation strategy?

    • Back-pressure appears as producer throttling; the mitigation is to increase the number of topics
    • Back-pressure appears as growing consumer lag when processing is slower than the production rate; mitigations include adding consumers up to the partition count, increasing partitions, or rate-limiting the producer
    • Back-pressure appears as broker disk saturation; the mitigation is to enable log compaction
    • Back-pressure appears as ZooKeeper timeouts; the mitigation is to upgrade to KRaft mode

24. 24. How does Kafka Streams differ from Apache Flink and Apache Spark Structured Streaming for stateful stream processing?

    • Kafka Streams is a batch framework; Flink and Spark are pure streaming engines
    • Kafka Streams is a client library that runs inside your application without a separate cluster; Flink and Spark are standalone distributed processing frameworks with their own resource management
    • Kafka Streams requires Kubernetes; Flink and Spark run on bare metal only
    • Kafka Streams does not support stateful operations; Flink and Spark do

25. 25. What is the throughput limit of an SQS FIFO queue and how do message group IDs affect it?

    • FIFO queues support unlimited throughput; message group IDs have no effect on throughput
    • FIFO queues support up to 300 messages/second (or 3,000 with batching); messages in the same group ID are processed sequentially, so more distinct group IDs enable greater parallelism
    • FIFO queues support 10,000 messages/second; each group ID adds 1,000 messages/second of capacity
    • FIFO queues support up to 1,000 messages/second regardless of group IDs

26. 26. Why is a schema registry important when using Avro or Protobuf with Kafka, and what problem does it solve?

    • A schema registry encrypts messages in transit; without it, Avro payloads are sent in plain text
    • A schema registry centralizes schema versioning so producers and consumers can evolve schemas independently while maintaining backward or forward compatibility, and stores only the schema ID in the message payload
    • A schema registry is a Kafka broker plugin that validates message size limits
    • A schema registry replaces ZooKeeper for cluster metadata management in newer Kafka versions

27. 27. What metrics and tools are commonly used to monitor Kafka consumer lag in production?

    • Consumer lag can only be monitored via Kafka's built-in web UI dashboard
    • Consumer lag is monitored using kafka-consumer-groups.sh, Kafka's JMX metrics (records-lag-max), Prometheus with the kafka-exporter or Confluent's JMX exporter, and platforms like Datadog or Grafana
    • Consumer lag is visible only in AWS CloudWatch when using Amazon MSK
    • Consumer lag is tracked by reading the __consumer_offsets topic directly from the application code

28. 28. What is the transactional outbox pattern and how does it solve dual-write problems between a database and a message broker?

    • The outbox pattern stores messages in an SQS FIFO queue before writing to the database, ensuring the database only updates if the queue accepts the message
    • The outbox pattern writes both the domain state change and the outgoing message to the same database transaction; a separate relay process then reads the outbox table and publishes to the broker, ensuring atomicity without distributed transactions
    • The outbox pattern uses two-phase commit across the database and Kafka broker to guarantee simultaneous writes
    • The outbox pattern buffers messages in application memory and flushes them to Kafka after the database transaction commits

29. 29. What is Kafka Connect and what are its two main connector types?

    • Kafka Connect is a client library for building custom consumer groups; its two types are polling and push connectors
    • Kafka Connect is a framework for scalably streaming data between Kafka and external systems; Source connectors ingest data into Kafka, and Sink connectors export data from Kafka to external targets
    • Kafka Connect is a monitoring tool; its connector types are metrics and logs connectors
    • Kafka Connect is a schema registry extension; its types are Avro connectors and Protobuf connectors

30. 30. How does cross-region replication differ between SQS-based fan-out and Kafka MirrorMaker 2?

    • SQS fan-out and MirrorMaker 2 are functionally identical; the choice depends only on cost
    • SQS fan-out uses SNS to push messages to queues in multiple regions simultaneously (active-active), while MirrorMaker 2 asynchronously replicates Kafka topics and consumer group offsets across clusters, supporting both active-active and active-passive topologies
    • SQS fan-out replicates messages using Kafka Connect; MirrorMaker 2 uses AWS Data Sync
    • MirrorMaker 2 works only with exactly-once semantics; SQS fan-out works only with at-most-once semantics