注意

本文档适用于 Ceph 开发版本。

Bucket Notifications 

Nautilus 版本中的新功能。

Squid 版本中的更改：可以使用 notification_v2 特性启用新的“v2”格式主题和通知元数据。zone 特性。 zone feature. Enabling this feature after an upgrade from an older version will trigger migration of the existing Topic and Notification metadata. In a greenfield deployment, the new format will be used. The new format allows for the data to be synced between zones in the zonegroup.

桶通知提供了一种机制，在桶上发生特定事件时将信息从 radosgw 发送出去。通知可以发送到 HTTP 端点、AMQP0.9.1 端点和 Kafka 端点。

用户可以创建主题。主题实体由其名称定义，并且是“按租户”。用户只能将其主题（通过通知配置）与其拥有的桶相关联。

必须创建通知实体才能为特定桶发送事件通知。可以创建通知实体用于事件类型的一个子集，或者用于所有“删除”和“创建”事件类型（这是默认值）。通知还可以根据（1）键、（2）附加到对象的元数据属性或（3）对象标签的前缀和后缀的匹配来过滤事件。也可以在这些上使用正则表达式匹配来创建过滤器。通知和主题之间存在多对多关系。一个主题可以接收多个通知，一个通知可以被发送到多个主题。

已定义 REST API，以提供桶通知机制的配置和控制接口。

S3 桶通知兼容性

Note

要启用桶通知 API，配置参数 rgw_enable_apis 应该包含：“notifications”。rgw_enable_apis configuration parameter should contain: “notifications”.

通知可靠性 

通知可以同步或异步发送。本节描述了同步和异步通知的延迟和可靠性。

同步通知 

通知可以同步发送，作为触发它们的操作的一部分。在这种模式下，操作只有在通知发送到主题配置的端点后才被确认（acked）。这意味着通知的往返时间（将通知发送到主题端点的时间加上接收确认的时间）被添加到操作本身的延迟中。

Note

即使通知出错、无法交付或超时，原始触发操作也被认为成功。

异步通知 

通知可以异步发送。它们被提交到持久存储，然后异步发送到主题配置的端点。只有当触发操作成功时，通知才会被提交到持久存储。503 Service Unavailable, which tells the client that it may retry later.

Note

如果通知出错、无法交付或超时，它将重试，直到成功确认。

提示

为了尽量减少异步通知增加的延迟，我们建议将“log”池放置在快速媒体上。

通过 CLI 管理主题 

运行以下命令以获取与租户关联的所有主题的配置：

radosgw-admin topic list [--tenant={tenant}]  [--uid={user}]

运行以下命令以获取特定主题的配置：

radosgw-admin topic get --topic={topic-name} [--tenant={tenant}]

运行以下命令以删除主题：

radosgw-admin topic rm --topic={topic-name} [--tenant={tenant}]

运行以下命令以获取持久主题统计信息（即保留、条目和大小）：

radosgw-admin topic stats --topic={topic-name} [--tenant={tenant}]

运行以下命令以以 JSON 格式转储持久主题的所有待处理的桶通知：

radosgw-admin topic dump --topic={topic-name} [--tenant={tenant}] [--max-entries={max-entries}]

通知性能统计 

pubsub_event_triggered事件计数器，至少有一个主题与之关联
pubsub_event_lost事件计数器，曾经有主题与之关联，但未推送到任何端点
pubsub_push_ok所有通知的事件计数器，成功推送到其端点
pubsub_push_fail所有通知的事件计数器，未能推送到其端点
pubsub_push_pending推送到端点但尚未确认或否认的事件的计量值

Note

pubsub_event_triggered和pubsub_event_lost每个通知上的每个事件都会增加，但pubsub_push_ok和pubsub_push_fail每个通知上的每个推送操作都会增加。

桶通知 REST API 

主题 

Note

在所有主题操作中，参数都是 URL 编码的，并使用此内容类型在消息正文中使用：application/x-www-form-urlencoded.

创建主题 

这会创建一个新主题。提供推送端点参数，稍后在创建通知时将使用这些参数。将生成响应。成功的响应包括主题的ARN（“Amazon 资源名称”，一个用于引用主题的唯一标识符）。

提示

主题上已经关联的任何通知都必须重新创建，以便主题可以更新。

POST

Action=CreateTopic
&Name=<topic-name>
[&Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=<exchange>]
[&Attributes.entry.2.key=amqp-ack-level&Attributes.entry.2.value=none|broker|routable]
[&Attributes.entry.3.key=verify-ssl&Attributes.entry.3.value=true|false]
[&Attributes.entry.4.key=kafka-ack-level&Attributes.entry.4.value=none|broker]
[&Attributes.entry.5.key=use-ssl&Attributes.entry.5.value=true|false]
[&Attributes.entry.6.key=ca-location&Attributes.entry.6.value=<file path>]
[&Attributes.entry.7.key=OpaqueData&Attributes.entry.7.value=<opaque data>]
[&Attributes.entry.8.key=push-endpoint&Attributes.entry.8.value=<endpoint>]
[&Attributes.entry.9.key=persistent&Attributes.entry.9.value=true|false]
[&Attributes.entry.10.key=cloudevents&Attributes.entry.10.value=true|false]
[&Attributes.entry.11.key=mechanism&Attributes.entry.11.value=<mechanism>]
[&Attributes.entry.12.key=time_to_live&Attributes.entry.12.value=<seconds to live>]
[&Attributes.entry.13.key=max_retries&Attributes.entry.13.value=<retries number>]
[&Attributes.entry.14.key=retry_sleep_duration&Attributes.entry.14.value=<sleep seconds>]
[&Attributes.entry.15.key=Policy&Attributes.entry.15.value=<policy-JSON-string>]
[&Attributes.entry.16.key=user-name&Attributes.entry.16.value=<user-name-string>]
[&Attributes.entry.17.key=password&Attributes.entry.17.value=<password-string>]
[&Attributes.entry.18.key=kafka-brokers&Attributes.entry.18.value=<kafka-broker-list>]

请求参数：

push-endpoint: 这是发送推送通知的端点的 URI。
OpaqueData: 透明数据在主题配置中设置，并添加到所有由主题触发的通知中。
persistent: 这表示发送到此端点的通知是否持久（=异步）或不持久。（默认为“false”）。
time_to_live: 这将限制保留通知的时间（以秒为单位）。rgw_topic_persistency_time_to_live。
max_retries: 这将限制在过期之前最大重试次数。rgw_topic_persistency_max_retries。
retry_sleep_duration: 这将控制重试通知的频率。rgw_topic_persistency_sleep_duration。

Policy: 这将控制除主题所有者之外谁可以访问主题。

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": ["arn:aws:iam::usfolks:user/fred:subuser"]},
    "Action": ["sns:GetTopicAttributes","sns:Publish"],
    "Resource": ["arn:aws:sns:default::mytopic"],
  }]
}

目前，我们只支持以下操作：

HTTP 端点

URI:http[s]://<fqdn>[:<port]

port: 这对于 HTTP 默认为 80，对于 HTTPS 默认为 443。

verify-ssl: 这表示客户端是否验证服务器证书。（默认为“true”）。

cloudevents: 这表示 HTTP 标头是否应包含根据S3 CloudEvents Spec中的属性。（默认为“false”）。

AMQP0.9.1 端点

URI:amqp[s]://[<user>:<password>@]<fqdn>[:<port>][/<vhost>]

user/password: 这对于未加密连接默认为“guest/guest”。

user/password: 只有在使用 HTTPS 时才必须提供。否则，主题创建请求将被拒绝。

port: 对于未加密连接默认为 5672，对于 SSL 加密连接默认为 5671。

vhost: 默认为“/”。

verify-ssl: 这表示客户端是否验证服务器证书。（默认为“true”）。

如果ca-location如果提供了此值并使用安全连接，则将使用指定的 CA 来验证代理。将不使用默认 CA。

amqp-exchange: 交换必须存在，并且必须能够根据主题路由消息。此参数是必需的。

amqp-ack-level: 不需要端到端确认。消息可能在交付到最终目的地之前在代理中持久存在。存在三种确认方法：

“none”：如果消息发送到代理，则将其视为“已交付”。

“broker”：如果代理确认消息，则将其视为“已交付”（默认）。

“routable”：如果代理可以路由到消费者，则将其视为“已交付”。

提示

主题名称（见创建主题）用于 AMQP 主题（主题交换的“路由键”）。

Kafka 端点

URI:kafka://[<user>:<password>@]<fqdn>[:<port]

use-ssl: 如果此值设置为“true”，则使用安全连接连接到代理。（默认为“false”）。

ca-location: 如果提供了此值并使用安全连接，则将使用指定的 CA 而不是默认 CA 来验证代理。

user/password: 应该在使用 HTTPS 时提供。否则，需要配置参数rgw_allow_notification_secrets_in_cleartext设备必须是true以创建主题。

user/password: 应该与use-ssl一起提供。否则，代理凭据将通过不安全的传输发送。

mechanism: 可以与 user/password 一起提供（默认：PLAIN）。支持的 SASL 机制是：

user-name: 连接到 Kafka 代理时使用的用户名。如果提供了此参数和 URI user，则此参数将覆盖 URI user。
对于此参数与 user/password 相同的安全注意事项。

password: 连接到 Kafka 代理时使用的密码。如果提供了此参数和 URI password，则此参数将覆盖 URI password。
对于此参数与 user/password 相同的安全注意事项。

PLAIN

SCRAM-SHA-256

SCRAM-SHA-512

GSSAPI

OAUTHBEARER

port: 默认为 9092。

kafka-ack-level: 不需要端到端确认。消息可能在交付到最终目的地之前在代理中持久存在。存在两种确认方法：

“none”：如果消息发送到代理，则将其视为“已交付”。

“broker”：如果代理确认消息，则将其视为“已交付”。（这是默认值。）

kafka-brokers: A command-separated list of host:port of kafka brokers. These brokers (may contain a broker which is defined in kafka uri) will be added to kafka uri to support sending notifcations to a kafka cluster.

Note

The key-value pair of a specific parameter need not reside in the same line as the parameter, and need not appear in any specific order, but it must use the same index.
Attribute indexing need not be sequential and need not start from any specific value.
AWS Create Topic provides a detailed explanation of the endpoint attributes format. In our case, however, different keys and values are used.

The response has the following format:

<CreateTopicResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/">
    <CreateTopicResult>
        <TopicArn></TopicArn>
    </CreateTopicResult>
    <ResponseMetadata>
        <RequestId></RequestId>
    </ResponseMetadata>
</CreateTopicResponse>

The topic ARN in the response has the following format:

arn:aws:sns:<zone-group>:<tenant>:<topic>

获取主题属性 

This returns information about a specific topic. This includes push-endpoint information, if provided.

POST

Action=GetTopicAttributes
&TopicArn=<topic-arn>

The response has the following format:

<GetTopicAttributesResponse>
    <GetTopicAttributesResult>
        <Attributes>
            <entry>
                <key>User</key>
                <value></value>
            </entry>
            <entry>
                <key>Name</key>
                <value></value>
            </entry>
            <entry>
                <key>EndPoint</key>
                <value></value>
            </entry>
            <entry>
                <key>TopicArn</key>
                <value></value>
            </entry>
            <entry>
                <key>OpaqueData</key>
                <value></value>
            </entry>
        </Attributes>
    </GetTopicAttributesResult>
    <ResponseMetadata>
        <RequestId></RequestId>
    </ResponseMetadata>
</GetTopicAttributesResponse>

User: the name of the user that created the topic.
Name: the name of the topic.
EndPoint: The JSON-formatted endpoint parameters, including:
- EndpointAddress: The push-endpoint URL.
- EndpointArgs: The push-endpoint args.
- EndpointTopic: The topic name to be sent to the endpoint (can be different than the above topic name).
- HasStoredSecret: This is “true” if the endpoint URL contains user/password information. In this case, the request must be made over HTTPS. The “topic get” request will otherwise be rejected.
- Persistent: This is “true” if the topic is persistent.
- TimeToLive: This will limit the time (in seconds) to retain the notifications.
- MaxRetries: This will limit the max retries before expiring notifications.
- RetrySleepDuration: This will control the frequency of retrying the notifications.
TopicArn: topic ARN.
OpaqueData: The opaque data set on the topic.
Policy: Any access permission set on the topic.

获取主题信息 

This returns information about a specific topic. This includes push-endpoint information, if provided. Note that this API is now deprecated in favor of the AWS compliant GetTopicAttributesAPI 为集群设置默认值。

POST

Action=GetTopic
&TopicArn=<topic-arn>

The response has the following format:

<GetTopicResponse>
    <GetTopicResult>
        <Topic>
            <User></User>
            <Name></Name>
            <EndPoint>
                <EndpointAddress></EndpointAddress>
                <EndpointArgs></EndpointArgs>
                <EndpointTopic></EndpointTopic>
                <HasStoredSecret></HasStoredSecret>
                <Persistent></Persistent>
            </EndPoint>
            <TopicArn></TopicArn>
            <OpaqueData></OpaqueData>
        </Topic>
    </GetTopicResult>
    <ResponseMetadata>
        <RequestId></RequestId>
    </ResponseMetadata>
</GetTopicResponse>

User: The name of the user that created the topic.
Name: The name of the topic.
EndpointAddress: The push-endpoint URL.
EndpointArgs: The push-endpoint args.
EndpointTopic: The topic name to be sent to the endpoint (which can be different than the above topic name).
HasStoredSecret: This is “true” if the endpoint URL contains user/password information. In this case, the request must be made over HTTPS. The “topic get” request will otherwise be rejected.
Persistent: “true” if topic is persistent.
TopicArn: topic ARN.
OpaqueData: the opaque data set on the topic.
Policy: Any access permission set on the topic.

删除主题 

POST

Action=DeleteTopic
&TopicArn=<topic-arn>

This deletes the specified topic.

Note

Deleting an unknown notification (for example, double delete) is not considered an error.
Deleting a topic does not automatically delete all notifications associated with it.

The response has the following format:

<DeleteTopicResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/">
    <ResponseMetadata>
        <RequestId></RequestId>
    </ResponseMetadata>
</DeleteTopicResponse>

列出主题 

List all topics associated with a tenant.

POST

Action=ListTopics

The response has the following format:

<ListTopicsResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/">
    <ListTopicsResult>
        <Topics>
            <member>
                <User></User>
                <Name></Name>
                <EndPoint>
                    <EndpointAddress></EndpointAddress>
                    <EndpointArgs></EndpointArgs>
                    <EndpointTopic></EndpointTopic>
                </EndPoint>
                <TopicArn></TopicArn>
                <OpaqueData></OpaqueData>
            </member>
        </Topics>
    </ListTopicsResult>
    <ResponseMetadata>
        <RequestId></RequestId>
    </ResponseMetadata>
</ListTopicsResponse>

If the endpoint URL contains user/password information in any part of the topic, the request must be made over HTTPS. The “topic list” request will otherwise be rejected.

设置主题属性 

POST

Action=SetTopicAttributes
&TopicArn=<topic-arn>&AttributeName=<attribute-name>&AttributeValue=<attribute-value>

This allows to set/modify existing attributes on the specified topic.

Note

The AttributeName passed will either be updated or created (if not exist) with AttributeValue passed.
Any unsupported AttributeName passed will result in error 400.

The response has the following format:

<SetTopicAttributesResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/">
    <ResponseMetadata>
        <RequestId></RequestId>
    </ResponseMetadata>
</SetTopicAttributesResponse>

Valid AttributeName that can be passed:

push-endpoint: 这是发送推送通知的端点的 URI。

OpaqueData: 透明数据在主题配置中设置，并添加到所有由主题触发的通知中。

persistent: 这表示发送到此端点的通知是否持久（=异步）或不持久。（默认为“false”）。

time_to_live: This will limit the time (in seconds) to retain the notifications.

max_retries: This will limit the max retries before expiring notifications.

retry_sleep_duration: This will control the frequency of retrying the notifications.

Policy: This will control who can access the topic other than owner of the topic.

verify-ssl: This indicates whether the server certificates must be validated by the client. This is “true” by default.

use-ssl: If this is set to “true”, a secure connection is used to connect to the broker. This is “false” by default.

cloudevents: 这表示 HTTP 标头是否应包含根据S3 CloudEvents Spec.

amqp-exchange: The exchanges must exist and must be able to route messages based on topics.

amqp-ack-level: No end2end acknowledgement is required. Messages may persist in the broker before being delivered to their final destinations.

ca-location: 如果提供了此值并使用安全连接，则将使用指定的 CA 而不是默认 CA 来验证代理。

mechanism: 可以与 user/password 一起提供（默认：PLAIN).

kafka-ack-level: No end2end acknowledgement is required. Messages may persist in the broker before being delivered to their final destinations.

kafka-brokers: Set endpoint with broker(s) as a comma-separated list of host or host:port (default port 9092).

通知 

Detailed under: Bucket Operations.

Note

“Abort Multipart Upload” request does not emit a notification
Both “Initiate Multipart Upload” and “POST Object” requests will emit an s3:ObjectCreated:Post notification

事件 

Events are in JSON format (regardless of the actual endpoint), and are S3-compatible. For example:

{"Records":[
    {
        "eventVersion":"2.1",
        "eventSource":"ceph:s3",
        "awsRegion":"zonegroup1",
        "eventTime":"2019-11-22T13:47:35.124724Z",
        "eventName":"ObjectCreated:Put",
        "userIdentity":{
            "principalId":"tester"
        },
        "requestParameters":{
            "sourceIPAddress":""
        },
        "responseElements":{
            "x-amz-request-id":"503a4c37-85eb-47cd-8681-2817e80b4281.5330.903595",
            "x-amz-id-2":"14d2-zone1-zonegroup1"
        },
        "s3":{
            "s3SchemaVersion":"1.0",
            "configurationId":"mynotif1",
            "bucket":{
                "name":"mybucket1",
                "ownerIdentity":{
                    "principalId":"tester"
                },
                "arn":"arn:aws:s3:zonegroup1::mybucket1",
                "id":"503a4c37-85eb-47cd-8681-2817e80b4281.5332.38"
            },
            "object":{
                "key":"myimage1.jpg",
                "size":"1024",
                "eTag":"37b51d194a7513e45b56f6524f2d51f2",
                "versionId":"",
                "sequencer": "F7E6D75DC742D108",
                "metadata":[],
                "tags":[]
            }
        },
        "eventId":"",
        "opaqueData":"me@example.com"
    }
]}

awsRegion: The zonegroup.
eventTime: The timestamp, indicating when the event was triggered.
eventName: For the list of supported events see: S3 Notification Compatibility. Note that eventName values do not start with the s3:前缀。
userIdentity.principalId: The user that triggered the change.
requestParameters.sourceIPAddress: not supported
responseElements.x-amz-request-id: The request ID of the original change.
responseElements.x_amz_id_2: The RGW on which the change was made.
s3.configurationId: The notification ID that created the event.
s3.bucket.name: The name of the bucket.
s3.bucket.ownerIdentity.principalId: The owner of the bucket.
s3.bucket.arn: The ARN of the bucket.
s3.bucket.id: The ID of the bucket. (This is an extension to the S3 notification API.)
s3.object.key: The object key.
s3.object.size: The object size.
s3.object.eTag: The object etag.
s3.object.versionId: The object version, if the bucket is versioned. When a copy is made, it includes the version of the target object. When a delete marker is created, it includes the version of the delete marker.
s3.object.sequencer: The monotonically-increasing identifier of the “change per object” (hexadecimal format).
s3.object.metadata: Any metadata set on the object that is sent as x-amz-meta- (that is, any metadata set on the object that is sent as an extension to the S3 notification API).
s3.object.tags: Any tags set on the object. (This is an extension to the S3 notification API.)
s3.eventId: The unique ID of the event, which could be used for acking. (This is an extension to the S3 notification API.)
s3.opaqueData: This means that “opaque data” is set in the topic configuration and is added to all notifications triggered by the topic. (This is an extension to the S3 notification API.)

由 Ceph 基金会带给您

Ceph 文档是一个社区资源，由非盈利的 Ceph 基金会资助和托管Ceph Foundation. 如果您想支持这一点和我们的其他工作，请考虑加入现在加入.