agent-claw: automated task changes
This commit is contained in:
daniel
346
cdk/node_modules/aws-cdk-lib/aws-kinesis/README.md
generated
vendored
Normal file
346
cdk/node_modules/aws-cdk-lib/aws-kinesis/README.md
generated
vendored
Normal file
@@ -0,0 +1,346 @@
|
||||
# Amazon Kinesis Construct Library
|
||||
|
||||
|
||||
[Amazon Kinesis](https://docs.aws.amazon.com/streams/latest/dev/introduction.html) provides collection and processing of large
|
||||
[streams](https://aws.amazon.com/streaming-data/) of data records in real time. Kinesis data streams can be used for rapid and continuous data
|
||||
intake and aggregation.
|
||||
|
||||
## Table Of Contents
|
||||
|
||||
- [Streams](#streams)
|
||||
- [Encryption](#encryption)
|
||||
- [Import](#import)
|
||||
- [Permission Grants](#permission-grants)
|
||||
- [Read Permissions](#read-permissions)
|
||||
- [Write Permissions](#write-permissions)
|
||||
- [Custom Permissions](#custom-permissions)
|
||||
- [Metrics](#metrics)
|
||||
- [Shard-level Metrics](#shard-level-metrics)
|
||||
- [Stream Consumers](#stream-consumers)
|
||||
- [Read Permissions](#read-permissions-1)
|
||||
- [Resource Policy](#resource-policy)
|
||||
|
||||
|
||||
## Streams
|
||||
|
||||
Amazon Kinesis Data Streams ingests a large amount of data in real time, durably stores the data, and makes the data available for consumption.
|
||||
|
||||
Using the CDK, a new Kinesis stream can be created as part of the stack using the construct's constructor. You may specify the `streamName` to give
|
||||
your own identifier to the stream. If not, CloudFormation will generate a name.
|
||||
|
||||
```ts
|
||||
new kinesis.Stream(this, 'MyFirstStream', {
|
||||
streamName: 'my-awesome-stream',
|
||||
});
|
||||
```
|
||||
|
||||
You can also specify properties such as `shardCount` to indicate how many shards the stream should choose and a `retentionPeriod`
|
||||
to specify how long the data in the shards should remain accessible.
|
||||
Read more at [Creating and Managing Streams](https://docs.aws.amazon.com/streams/latest/dev/working-with-streams.html)
|
||||
|
||||
```ts
|
||||
new kinesis.Stream(this, 'MyFirstStream', {
|
||||
streamName: 'my-awesome-stream',
|
||||
shardCount: 3,
|
||||
retentionPeriod: Duration.hours(48),
|
||||
});
|
||||
```
|
||||
|
||||
### Encryption
|
||||
|
||||
[Stream encryption](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kinesis-stream-streamencryption.html) enables
|
||||
server-side encryption using an AWS KMS key for a specified stream.
|
||||
|
||||
Encryption is enabled by default on your stream with the master key owned by Kinesis Data Streams in regions where it is supported.
|
||||
|
||||
```ts
|
||||
new kinesis.Stream(this, 'MyEncryptedStream');
|
||||
```
|
||||
|
||||
You can enable encryption on your stream with a user-managed key by specifying the `encryption` property.
|
||||
A KMS key will be created for you and associated with the stream.
|
||||
|
||||
```ts
|
||||
new kinesis.Stream(this, 'MyEncryptedStream', {
|
||||
encryption: kinesis.StreamEncryption.KMS,
|
||||
});
|
||||
```
|
||||
|
||||
You can also supply your own external KMS key to use for stream encryption by specifying the `encryptionKey` property.
|
||||
|
||||
```ts
|
||||
const key = new kms.Key(this, 'MyKey');
|
||||
|
||||
new kinesis.Stream(this, 'MyEncryptedStream', {
|
||||
encryption: kinesis.StreamEncryption.KMS,
|
||||
encryptionKey: key,
|
||||
});
|
||||
```
|
||||
|
||||
### Import
|
||||
|
||||
Any Kinesis stream that has been created outside the stack can be imported into your CDK app.
|
||||
|
||||
Streams can be imported by their ARN via the `Stream.fromStreamArn()` API
|
||||
|
||||
```ts
|
||||
const importedStream = kinesis.Stream.fromStreamArn(this, 'ImportedStream',
|
||||
'arn:aws:kinesis:us-east-2:123456789012:stream/f3j09j2230j',
|
||||
);
|
||||
```
|
||||
|
||||
Encrypted Streams can also be imported by their attributes via the `Stream.fromStreamAttributes()` API
|
||||
|
||||
```ts
|
||||
const importedStream = kinesis.Stream.fromStreamAttributes(this, 'ImportedEncryptedStream', {
|
||||
streamArn: 'arn:aws:kinesis:us-east-2:123456789012:stream/f3j09j2230j',
|
||||
encryptionKey: kms.Key.fromKeyArn(this, 'key',
|
||||
'arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012',
|
||||
),
|
||||
});
|
||||
```
|
||||
|
||||
### Permission Grants
|
||||
|
||||
IAM roles, users or groups which need to be able to work with Amazon Kinesis streams at runtime should be granted IAM permissions.
|
||||
|
||||
Any object that implements the `IGrantable` interface (has an associated principal) can be granted permissions by calling:
|
||||
|
||||
- `grantRead(principal)` - grants the principal read access
|
||||
- `grantWrite(principal)` - grants the principal write permissions to a Stream
|
||||
- `grantReadWrite(principal)` - grants principal read and write permissions
|
||||
|
||||
#### Read Permissions
|
||||
|
||||
Grant `read` access to a stream by calling the `grantRead()` API.
|
||||
If the stream has an encryption key, read permissions will also be granted to the key.
|
||||
|
||||
```ts
|
||||
const lambdaRole = new iam.Role(this, 'Role', {
|
||||
assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'),
|
||||
description: 'Example role...',
|
||||
});
|
||||
|
||||
const stream = new kinesis.Stream(this, 'MyEncryptedStream', {
|
||||
encryption: kinesis.StreamEncryption.KMS,
|
||||
});
|
||||
|
||||
// give lambda permissions to read stream
|
||||
stream.grantRead(lambdaRole);
|
||||
```
|
||||
|
||||
The following read permissions are provided to a service principal by the `grantRead()` API:
|
||||
|
||||
- `kinesis:DescribeStreamSummary`
|
||||
- `kinesis:GetRecords`
|
||||
- `kinesis:GetShardIterator`
|
||||
- `kinesis:ListShards`
|
||||
- `kinesis:SubscribeToShard`
|
||||
|
||||
#### Write Permissions
|
||||
|
||||
Grant `write` permissions to a stream is provided by calling the `grantWrite()` API.
|
||||
If the stream has an encryption key, write permissions will also be granted to the key.
|
||||
|
||||
```ts
|
||||
const lambdaRole = new iam.Role(this, 'Role', {
|
||||
assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'),
|
||||
description: 'Example role...',
|
||||
});
|
||||
|
||||
const stream = new kinesis.Stream(this, 'MyEncryptedStream', {
|
||||
encryption: kinesis.StreamEncryption.KMS,
|
||||
});
|
||||
|
||||
// give lambda permissions to write to stream
|
||||
stream.grantWrite(lambdaRole);
|
||||
```
|
||||
|
||||
The following write permissions are provided to a service principal by the `grantWrite()` API:
|
||||
|
||||
- `kinesis:ListShards`
|
||||
- `kinesis:PutRecord`
|
||||
- `kinesis:PutRecords`
|
||||
|
||||
#### Custom Permissions
|
||||
|
||||
You can add any set of permissions to a stream by calling the `grant()` API.
|
||||
|
||||
```ts
|
||||
const user = new iam.User(this, 'MyUser');
|
||||
|
||||
const stream = new kinesis.Stream(this, 'MyStream');
|
||||
|
||||
// give my user permissions to list shards
|
||||
stream.grant(user, 'kinesis:ListShards');
|
||||
```
|
||||
|
||||
### Metrics
|
||||
|
||||
You can use common metrics from your stream to create alarms and/or dashboards. The `stream.metric('MetricName')` method creates a metric with the stream namespace and dimension. You can also use pre-define methods like `stream.metricGetRecordsSuccess()`. To find out more about Kinesis metrics check [Monitoring the Amazon Kinesis Data Streams Service with Amazon CloudWatch](https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-cloudwatch.html).
|
||||
|
||||
```ts
|
||||
const stream = new kinesis.Stream(this, 'MyStream');
|
||||
|
||||
// Using base metric method passing the metric name
|
||||
stream.metric('GetRecords.Success');
|
||||
|
||||
// using pre-defined metric method
|
||||
stream.metricGetRecordsSuccess();
|
||||
|
||||
// using pre-defined and overriding the statistic
|
||||
stream.metricGetRecordsSuccess({ statistic: 'Maximum' });
|
||||
```
|
||||
|
||||
#### Shard-level Metrics
|
||||
|
||||
You can enable enhanced shard-level metrics for your Kinesis stream to get detailed monitoring of individual shards. Shard-level metrics provide more granular insights into the performance and health of your stream.
|
||||
|
||||
```ts
|
||||
const stream = new kinesis.Stream(this, 'MyStream', {
|
||||
shardLevelMetrics: [kinesis.ShardLevelMetrics.ALL],
|
||||
});
|
||||
```
|
||||
|
||||
You can also specify individual metrics that you want to monitor:
|
||||
|
||||
```ts
|
||||
const stream = new kinesis.Stream(this, 'MyStream', {
|
||||
shardLevelMetrics: [
|
||||
kinesis.ShardLevelMetrics.INCOMING_BYTES,
|
||||
kinesis.ShardLevelMetrics.INCOMING_RECORDS,
|
||||
kinesis.ShardLevelMetrics.ITERATOR_AGE_MILLISECONDS,
|
||||
kinesis.ShardLevelMetrics.OUTGOING_BYTES,
|
||||
kinesis.ShardLevelMetrics.OUTGOING_RECORDS,
|
||||
kinesis.ShardLevelMetrics.READ_PROVISIONED_THROUGHPUT_EXCEEDED,
|
||||
kinesis.ShardLevelMetrics.WRITE_PROVISIONED_THROUGHPUT_EXCEEDED,
|
||||
],
|
||||
});
|
||||
```
|
||||
|
||||
Available shard-level metrics include:
|
||||
|
||||
- `INCOMING_BYTES` - The number of bytes successfully put to the shard
|
||||
- `INCOMING_RECORDS` - The number of records successfully put to the shard
|
||||
- `ITERATOR_AGE_MILLISECONDS` - The age of the last record in all GetRecords calls made against a shard
|
||||
- `OUTGOING_BYTES` - The number of bytes retrieved from the shard
|
||||
- `OUTGOING_RECORDS` - The number of records retrieved from the shard
|
||||
- `READ_PROVISIONED_THROUGHPUT_EXCEEDED` - The number of GetRecords calls throttled for the shard
|
||||
- `WRITE_PROVISIONED_THROUGHPUT_EXCEEDED` - The number of records rejected due to throttling for the shard
|
||||
- `ALL` - All available metrics
|
||||
|
||||
Note: You cannot specify `ALL` together with other individual metrics. If you want all metrics, use `ALL` alone.
|
||||
|
||||
For more information about shard-level metrics, see [Monitoring the Amazon Kinesis Data Streams Service with Amazon CloudWatch](https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-cloudwatch.html#kinesis-metrics-shard).
|
||||
|
||||
## Stream Consumers
|
||||
|
||||
Creating stream consumers allow consumers to receive data from the stream using enhanced fan-out at a rate of up to 2 MiB per second for every shard.
|
||||
This rate is unaffected by the total number of consumers that read from the same stream.
|
||||
|
||||
For more information, see [Develop enhanced fan-out consumers with dedicated throughput](https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html).
|
||||
|
||||
To create and associate a stream consumer with a stream
|
||||
|
||||
```ts
|
||||
const stream = new kinesis.Stream(this, 'MyStream');
|
||||
|
||||
const streamConsumer = new kinesis.StreamConsumer(this, 'MyStreamConsumer', {
|
||||
streamConsumerName: 'MyStreamConsumer',
|
||||
stream,
|
||||
});
|
||||
```
|
||||
|
||||
#### Read Permissions
|
||||
|
||||
Grant `read` access to a stream consumer, and the stream it is registered with, by calling the `grantRead()` API.
|
||||
|
||||
```ts
|
||||
const lambdaRole = new iam.Role(this, 'Role', {
|
||||
assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'),
|
||||
description: 'Example role...',
|
||||
});
|
||||
|
||||
const stream = new kinesis.Stream(this, 'MyEncryptedStream', {
|
||||
encryption: kinesis.StreamEncryption.KMS,
|
||||
});
|
||||
const streamConsumer = new kinesis.StreamConsumer(this, 'MyStreamConsumer', {
|
||||
streamConsumerName: 'MyStreamConsumer',
|
||||
stream,
|
||||
});
|
||||
|
||||
// give lambda permissions to read stream via the stream consumer
|
||||
streamConsumer.grantRead(lambdaRole);
|
||||
```
|
||||
|
||||
In addition to stream's permissions, the following permissions are provided to a service principal by the `grantRead()` API:
|
||||
|
||||
- `kinesis:DescribeStreamConsumer`,
|
||||
- `kinesis:SubscribeToShard`,
|
||||
|
||||
## Resource Policy
|
||||
|
||||
You can create a resource policy for a data stream or a stream consumer.
|
||||
For more information, see [Controlling access to Amazon Kinesis Data Streams resources using IAM](https://docs.aws.amazon.com/streams/latest/dev/controlling-access.html).
|
||||
|
||||
A resource policy is automatically created when `addToResourcePolicy` is called, if one doesn't already exist.
|
||||
|
||||
Using `addToResourcePolicy` is the simplest way to add a resource policy:
|
||||
|
||||
```ts
|
||||
const stream = new kinesis.Stream(this, 'MyStream');
|
||||
const streamConsumer = new kinesis.StreamConsumer(this, 'MyStreamConsumer', {
|
||||
streamConsumerName: 'MyStreamConsumer',
|
||||
stream,
|
||||
});
|
||||
|
||||
// create a stream resource policy via addToResourcePolicy method
|
||||
stream.addToResourcePolicy(new iam.PolicyStatement({
|
||||
resources: [stream.streamArn],
|
||||
actions: ['kinesis:GetRecords'],
|
||||
principals: [new iam.AnyPrincipal()],
|
||||
}));
|
||||
|
||||
// create a stream consumer resource policy via addToResourcePolicy method
|
||||
streamConsumer.addToResourcePolicy(new iam.PolicyStatement({
|
||||
resources: [stream.streamArn],
|
||||
actions: ['kinesis:DescribeStreamConsumer'],
|
||||
principals: [new iam.AnyPrincipal()],
|
||||
}));
|
||||
```
|
||||
|
||||
You can create a resource manually by using `ResourcePolicy`.
|
||||
Also, you can set a custom policy document to `ResourcePolicy`.
|
||||
If not, a blank policy document will be set.
|
||||
|
||||
```ts
|
||||
const stream = new kinesis.Stream(this, 'MyStream');
|
||||
const streamConsumer = new kinesis.StreamConsumer(this, 'MyStreamConsumer', {
|
||||
streamConsumerName: 'MyStreamConsumer',
|
||||
stream,
|
||||
});
|
||||
|
||||
// create a custom policy document
|
||||
const policyDocument = new iam.PolicyDocument({
|
||||
assignSids: true,
|
||||
statements: [
|
||||
new iam.PolicyStatement({
|
||||
actions: ['kinesis:GetRecords'],
|
||||
resources: [stream.streamArn],
|
||||
principals: [new iam.AnyPrincipal()],
|
||||
}),
|
||||
],
|
||||
});
|
||||
|
||||
// create a stream resource policy manually
|
||||
new kinesis.ResourcePolicy(this, 'ResourcePolicy', {
|
||||
stream,
|
||||
policyDocument,
|
||||
});
|
||||
|
||||
// create a stream consumer resource policy manually
|
||||
new kinesis.ResourcePolicy(this, 'ResourcePolicy', {
|
||||
streamConsumer,
|
||||
policyDocument,
|
||||
});
|
||||
```
|
||||
Reference in New Issue
Block a user