https://aws.amazon.com/blogs/aws/new-create-point-to-point-integrations-between-event-producers-and-consumers-with-amazon-eventbridge-pipes/

目前客户使用越来越多云上的服务来搭建事件型驱动应用,但需要了解它们之间的通讯技术,给开发人员带来额外的不便

EventBridge Pipes是点对点集成的解决方案,让生产者和消息者进行通讯,不再需要写额外的胶水代码

img

这个pipe包括源端和目标端,中间有可选的过滤和enrich功能,它不仅简单、省钱、而且可靠,有失败重试功能

使用EventBridge Pipes有以下步骤:

  1. 选择一个源端来发送消息,包括Amazon DynamoDB , Amazon Kinesis Data Streams , Amazon SQS , Amazon Managed Streaming for Apache Kafka , Amazon MQ ( ActiveMQ 和 RabbitMQ).
  2. (可选)定义一个事件过滤器,只匹配特定的事件。被过滤掉的事件就不收费了,帮助用户节省成本
  3. (可选)使用内置的转换器来对事件进行加工,或者使用AWS Lambda , AWS Step Functions , Amazon API Gateway , EventBridge API Destinations 来进行更丰富的加工
  4. 选择目标端,包括 Amazon Step Functions, Kinesis Data Streams, AWS Lambda或三方API

EventBridge Pipes减少了学习服务的成本、避免写服务之间的集成代码,并保证了可靠性和一致性

img

接下来我们看一个例子,使用EventBridge Pipes将SQS与Step Functions来集成

下面图是创建好的SQS和Step Functions,我们需要将SQS队列中的事件传入到Step Functions来处理:

Existing Amazon SQS queue and AWS Step Functions state machine

进入 Amazon EventBridge Pipes页面,选择 Create pipe

img

在这个页面,可以配置一个pipe并设置IAM权限,进入 Pipe settings 页面:

Navigate to Pipe Settings

Permissions部分,为这个pipe创建一个新的IAM role或使用已有的role。为了改善开发者体验,EventBridge Pipes会计算出所需的最小权限,不需要手工添加。这里选择创建新的IAM Role:

Setting IAM Permission for pipe

接下来进入Build pipe部分,在这里可以看到EventBridge Pipes支持的事件源:

Then, I go back to the Build pipe section. On this page, I can see the available event sources supported by EventBridge Pipes.

List of available services as the event source

选择SQS,如果需要做批处理,可以在Additional settings里定义批的大小和窗口。

Select SQS Queue as event source

进入下一步,进行事件过滤,这一步是可选的,如果事件被过滤了则不会收费:

Event filtering in Amazon EventBridge Pipes

例如只处理RETURN_PURCHASE大于100的事件:

{
   "event-type":"RETURN_PURCHASE",
   "value":100
}

在event pattern部分定义以下内容:

{
   "event-type": ["RETURN_PURCHASE"],
   "value": [{
      "numeric": [">=", 100]
   }]
}

可以点击test pattern进行测试,进行验证:

Defining and testing an event pattern for filtering

Enrichment 部分,可以使用lambda或三方API对事件进行加工,也可以使用自带的 Enrichment Input Transformer进行加工:

img

最后一步是定义target来处理接收到的事件:

Defining target destination service

这里选择AWS支持的目标端:

img

I select my existing AWS Step Functions state machine, named pipes-statemachine.

img

In addition, I can also use Target Input Transformer by referring to the Transforming Amazon EventBridge target input documentation. For my case, I need to define a high priority for events going into this target. To do that, I define a sample custom event in Sample events/Event Payload and add the priority: HIGH in the Transformer section. Then in the Output section, I can see the final event to be passed to the target destination service. Then, I select Create pipe.

img

In less than a minute, my pipe was successfully created.

Pipe successfully created

To test this pipe, I need to put an event into the Amaon SQS queue.

Sending a message into Amazon SQS Queue

To check if my event is successfully processed by Step Functions, I can look into my state machine in Step Functions. On this page, I see my event is successfully processed.

img

I can also go to Amazon CloudWatch Logs to get more detailed logs.

img

Things to Know Event Sources – At launch, Amazon EventBridge Pipes supports the following services as event sources: Amazon DynamoDB, Amazon Kinesis, Amazon Managed Streaming for Apache Kafka (Amazon MSK) alongside self-managed Apache Kafka, Amazon SQS (standard and FIFO), and Amazon MQ (both for ActiveMQ and RabbitMQ).

Event Targets – Amazon EventBridge Pipes supports 15 Amazon EventBridge targets, including AWS Lambda, Amazon API Gateway, Amazon SNS, Amazon SQS, and AWS Step Functions. To deliver events to any HTTPS endpoint, developers can use API destinations as the target.

Event Ordering – EventBridge Pipes maintains the ordering of events received from an event sources that support ordering when sending those events to a destination service.

Programmatic Access – You can also interact with Amazon EventBridge Pipes and create a pipe using AWS Command Line Interface (CLI) , AWS CloudFormation , and AWS Cloud Development Kit (AWS CDK) .

Independent Usage – EventBridge Pipes can be used separately from Amazon EventBridge bus and Amazon EventBridge Scheduler. This flexibility helps developers to define source events from supported AWS and self-managed services as event sources without Amazon EventBridge Event Bus.

Availability – Amazon EventBridge Pipes is now generally available in all AWS commercial Regions, with the exception of Asia Pacific (Hyderabad) and Europe (Zurich).

Visit the Amazon EventBridge Pipes page to learn more about this feature and understand the pricing. You can also visit the documentation page to learn more about how to get started.

Happy building!