Skip to content

Conversation

@shaked-lokits
Copy link

Issue number: closes #5362

Summary

Add support for processing Kafka and MSK events with partial batch failure reporting, leveraging the new ReportBatchItemFailures functionality that AWS recently enabled for Kafka event sources.

Changes

  • Add EventType.Kafka to batch processor for unified MSK/self-managed Kafka support
  • Implement _collect_kafka_failures() for composite identifiers (topic-partition + offset)
  • Add _get_records_from_event() helper to flatten Kafka's nested record structure
  • Add KafkaItemIdentifier TypedDict for type safety
  • Add 14 functional tests covering sync/async processing, partial failures, and edge cases
  • Update batch processing documentation with Kafka section and SAM template
  • Update Kafka utility documentation with cross-reference to batch processing

User experience

Before: Users had to manually implement partial batch failure handling for Kafka/MSK events, including constructing the correct response format with composite identifiers.

After: Users can use the familiar batch processing API:

from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, process_partial_response

processor = BatchProcessor(event_type=EventType.Kafka)

def record_handler(record):
    # Process record
    return record.value

def lambda_handler(event, context):
    return process_partial_response(event=event, record_handler=record_handler, processor=processor, context=context)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@shaked-lokits shaked-lokits requested a review from a team as a code owner January 12, 2026 16:13
@boring-cyborg
Copy link

boring-cyborg bot commented Jan 12, 2026

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@boring-cyborg boring-cyborg bot added documentation Improvements or additions to documentation tests labels Jan 12, 2026
@pull-request-size pull-request-size bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jan 12, 2026
@sonarqubecloud
Copy link

@codecov
Copy link

codecov bot commented Jan 19, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.73%. Comparing base (8761fe7) to head (38538e6).

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #7941   +/-   ##
========================================
  Coverage    96.72%   96.73%           
========================================
  Files          278      278           
  Lines        13626    13642   +16     
  Branches      1083     1085    +2     
========================================
+ Hits         13180    13196   +16     
  Misses         327      327           
  Partials       119      119           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: add msk to batch processing

2 participants