SQS lambda trigger not working as expected
The scenario is this: When my process start I send around 3k messages to a queue in SQS. From there I have a lambda that I want to pick 500 of those messages and process them, then take the next 500 messages and so on until the queue is empty. Since the lambda that processes the messages has to talk to a very slow API I can’t use concurrent lambda executions, otherwise I will reach the limits on that API.
The problem that I’m seeing is that sometimes the lambda starts picking 500 messages but in the next execution picks less than 200 and the next execution it picks even less and so on, instead of picking 500 everytime.
Maybe I misunderstanding something in the settings. How can I achieve this behaviour?
These are my settings:
SQS Messages Enter Visibility Timeout but Lambda did not Receive Them
I have a single concurrency Lambda that I want to trigger from SQS messages, I also have defined BatchSize to one so only one message is received by the Lambda. However, I can see that my SQS is sending more than one message as I can see from the Messages Sent metric and the worst thing is I have a function in my Lambda to delete certain messages in the queue however all the messages are unavailable and immediately enter the visibility timeout.