282 lines
11 KiB
Markdown
282 lines
11 KiB
Markdown
# Amazon FSx Construct Library
|
||
|
||
[Amazon FSx](https://docs.aws.amazon.com/fsx/?id=docs_gateway) provides fully managed third-party file systems with the
|
||
native compatibility and feature sets for workloads such as Microsoft Windows–based storage, high-performance computing,
|
||
machine learning, and electronic design automation.
|
||
|
||
Amazon FSx supports two file system types: [Lustre](https://docs.aws.amazon.com/fsx/latest/LustreGuide/index.html) and
|
||
[Windows](https://docs.aws.amazon.com/fsx/latest/WindowsGuide/index.html) File Server.
|
||
|
||
## FSx for Lustre
|
||
|
||
Amazon FSx for Lustre makes it easy and cost-effective to launch and run the popular, high-performance Lustre file
|
||
system. You use Lustre for workloads where speed matters, such as machine learning, high performance computing (HPC),
|
||
video processing, and financial modeling.
|
||
|
||
The open-source Lustre file system is designed for applications that require fast storage—where you want your storage
|
||
to keep up with your compute. Lustre was built to solve the problem of quickly and cheaply processing the world's
|
||
ever-growing datasets. It's a widely used file system designed for the fastest computers in the world. It provides
|
||
submillisecond latencies, up to hundreds of GBps of throughput, and up to millions of IOPS. For more information on
|
||
Lustre, see the [Lustre website](http://lustre.org/).
|
||
|
||
As a fully managed service, Amazon FSx makes it easier for you to use Lustre for workloads where storage speed matters.
|
||
Amazon FSx for Lustre eliminates the traditional complexity of setting up and managing Lustre file systems, enabling
|
||
you to spin up and run a battle-tested high-performance file system in minutes. It also provides multiple deployment
|
||
options so you can optimize cost for your needs.
|
||
|
||
Amazon FSx for Lustre is POSIX-compliant, so you can use your current Linux-based applications without having to make
|
||
any changes. Amazon FSx for Lustre provides a native file system interface and works as any file system does with your
|
||
Linux operating system. It also provides read-after-write consistency and supports file locking.
|
||
|
||
### Installation
|
||
|
||
Import to your project:
|
||
|
||
```ts nofixture
|
||
import * as fsx from 'aws-cdk-lib/aws-fsx';
|
||
```
|
||
|
||
### Basic Usage
|
||
|
||
Setup required properties and create:
|
||
|
||
```ts
|
||
declare const vpc: ec2.Vpc;
|
||
|
||
const fileSystem = new fsx.LustreFileSystem(this, 'FsxLustreFileSystem', {
|
||
lustreConfiguration: { deploymentType: fsx.LustreDeploymentType.SCRATCH_2 },
|
||
storageCapacityGiB: 1200,
|
||
vpc,
|
||
vpcSubnet: vpc.privateSubnets[0],
|
||
});
|
||
```
|
||
|
||
### File System Type Version
|
||
|
||
You can set [the Lustre version for the file system](https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-resource-fsx-filesystem.html#cfn-fsx-filesystem-filesystemtypeversion). To do this, use the `fileSystemTypeVersion` property:
|
||
|
||
```ts
|
||
declare const vpc: ec2.Vpc;
|
||
|
||
const fileSystem = new fsx.LustreFileSystem(this, 'FsxLustreFileSystem', {
|
||
lustreConfiguration: { deploymentType: fsx.LustreDeploymentType.SCRATCH_2 },
|
||
storageCapacityGiB: 1200,
|
||
vpc,
|
||
vpcSubnet: vpc.privateSubnets[0],
|
||
fileSystemTypeVersion: fsx.FileSystemTypeVersion.V_2_15,
|
||
});
|
||
```
|
||
|
||
**Note**: The `fileSystemTypeVersion` has a restrictions on the values that can be set based on the `deploymentType`.
|
||
|
||
- `V_2_10` is supported by the Scratch and `PERSISTENT_1` deployment types.
|
||
- `V_2_12` is supported by all Lustre deployment types.
|
||
- `V_2_15` is supported by all Lustre deployment types and is recommended for all new file systems.
|
||
|
||
**Note**: The default value of `fileSystemTypeVersion` is `V_2_10` except for `PERSISTENT_2` deployment type where the default value is `V_2_12`.
|
||
|
||
### Connecting
|
||
|
||
To control who can access the file system, use the `.connections` attribute. FSx has a fixed default port, so you don't
|
||
need to specify the port. This example allows an EC2 instance to connect to a file system:
|
||
|
||
```ts
|
||
declare const fileSystem: fsx.LustreFileSystem;
|
||
declare const instance: ec2.Instance;
|
||
|
||
fileSystem.connections.allowDefaultPortFrom(instance);
|
||
```
|
||
|
||
### Mounting
|
||
|
||
The LustreFileSystem Construct exposes both the DNS name of the file system as well as its mount name, which can be
|
||
used to mount the file system on an EC2 instance. The following example shows how to bring up a file system and EC2
|
||
instance, and then use User Data to mount the file system on the instance at start-up:
|
||
|
||
```ts
|
||
import * as iam from 'aws-cdk-lib/aws-iam';
|
||
|
||
declare const vpc: ec2.Vpc;
|
||
const lustreConfiguration = {
|
||
deploymentType: fsx.LustreDeploymentType.SCRATCH_2,
|
||
};
|
||
|
||
const fs = new fsx.LustreFileSystem(this, 'FsxLustreFileSystem', {
|
||
lustreConfiguration,
|
||
storageCapacityGiB: 1200,
|
||
vpc,
|
||
vpcSubnet: vpc.privateSubnets[0],
|
||
});
|
||
|
||
const inst = new ec2.Instance(this, 'inst', {
|
||
instanceType: ec2.InstanceType.of(ec2.InstanceClass.T2, ec2.InstanceSize.LARGE),
|
||
machineImage: new ec2.AmazonLinuxImage({
|
||
generation: ec2.AmazonLinuxGeneration.AMAZON_LINUX_2,
|
||
}),
|
||
vpc,
|
||
vpcSubnets: {
|
||
subnetType: ec2.SubnetType.PUBLIC,
|
||
},
|
||
});
|
||
fs.connections.allowDefaultPortFrom(inst);
|
||
|
||
// Need to give the instance access to read information about FSx to determine the file system's mount name.
|
||
inst.role.addManagedPolicy(iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonFSxReadOnlyAccess'));
|
||
|
||
const mountPath = '/mnt/fsx';
|
||
const dnsName = fs.dnsName;
|
||
const mountName = fs.mountName;
|
||
|
||
inst.userData.addCommands(
|
||
'set -eux',
|
||
'yum update -y',
|
||
'amazon-linux-extras install -y lustre2.10',
|
||
// Set up the directory to mount the file system to and change the owner to the AL2 default ec2-user.
|
||
`mkdir -p ${mountPath}`,
|
||
`chmod 777 ${mountPath}`,
|
||
`chown ec2-user:ec2-user ${mountPath}`,
|
||
// Set the file system up to mount automatically on start up and mount it.
|
||
`echo "${dnsName}@tcp:/${mountName} ${mountPath} lustre defaults,noatime,flock,_netdev 0 0" >> /etc/fstab`,
|
||
'mount -a',
|
||
);
|
||
```
|
||
|
||
### Importing an existing Lustre filesystem
|
||
|
||
An FSx for Lustre file system can be imported with `fromLustreFileSystemAttributes(this, id, attributes)`. The
|
||
following example lays out how you could import the SecurityGroup a file system belongs to, use that to import the file
|
||
system, and then also import the VPC the file system is in and add an EC2 instance to it, giving it access to the file
|
||
system.
|
||
|
||
```ts
|
||
const sg = ec2.SecurityGroup.fromSecurityGroupId(this, 'FsxSecurityGroup', '{SECURITY-GROUP-ID}');
|
||
const fs = fsx.LustreFileSystem.fromLustreFileSystemAttributes(this, 'FsxLustreFileSystem', {
|
||
dnsName: '{FILE-SYSTEM-DNS-NAME}',
|
||
fileSystemId: '{FILE-SYSTEM-ID}',
|
||
securityGroup: sg,
|
||
});
|
||
|
||
const vpc = ec2.Vpc.fromVpcAttributes(this, 'Vpc', {
|
||
availabilityZones: ['us-west-2a', 'us-west-2b'],
|
||
publicSubnetIds: ['{US-WEST-2A-SUBNET-ID}', '{US-WEST-2B-SUBNET-ID}'],
|
||
vpcId: '{VPC-ID}',
|
||
});
|
||
|
||
const inst = new ec2.Instance(this, 'inst', {
|
||
instanceType: ec2.InstanceType.of(ec2.InstanceClass.T2, ec2.InstanceSize.LARGE),
|
||
machineImage: new ec2.AmazonLinuxImage({
|
||
generation: ec2.AmazonLinuxGeneration.AMAZON_LINUX_2,
|
||
}),
|
||
vpc,
|
||
vpcSubnets: {
|
||
subnetType: ec2.SubnetType.PUBLIC,
|
||
},
|
||
});
|
||
|
||
fs.connections.allowDefaultPortFrom(inst);
|
||
```
|
||
|
||
### Lustre Data Repository Association support
|
||
|
||
The LustreFilesystem Construct supports one [Data Repository Association](https://docs.aws.amazon.com/fsx/latest/LustreGuide/fsx-data-repositories.html) (DRA) to an S3 bucket. This allows Lustre hierarchical storage management to S3 buckets, which in turn makes it possible to use S3 as a permanent backing store, and use FSx for Lustre as a temporary high performance cache.
|
||
|
||
Note: CloudFormation does not currently support for `PERSISTENT_2` filesystems, and so neither does CDK.
|
||
|
||
The following example illustrates setting up a DRA to an S3 bucket, including automated metadata import whenever a file is changed, created or deleted in the S3 bucket:
|
||
|
||
```ts
|
||
import { aws_s3 as s3 } from 'aws-cdk-lib';
|
||
|
||
declare const vpc: ec2.Vpc;
|
||
declare const bucket: s3.Bucket;
|
||
|
||
const lustreConfiguration = {
|
||
deploymentType: fsx.LustreDeploymentType.SCRATCH_2,
|
||
exportPath: bucket.s3UrlForObject(),
|
||
importPath: bucket.s3UrlForObject(),
|
||
autoImportPolicy: fsx.LustreAutoImportPolicy.NEW_CHANGED_DELETED,
|
||
};
|
||
|
||
const fs = new fsx.LustreFileSystem(this, "FsxLustreFileSystem", {
|
||
vpc: vpc,
|
||
vpcSubnet: vpc.privateSubnets[0],
|
||
storageCapacityGiB: 1200,
|
||
lustreConfiguration,
|
||
});
|
||
```
|
||
|
||
### Compression
|
||
|
||
By default, transparent compression of data within FSx for Lustre is switched off. To enable it, add the following to your `lustreConfiguration`:
|
||
|
||
```ts
|
||
const lustreConfiguration = {
|
||
// ...
|
||
dataCompressionType: fsx.LustreDataCompressionType.LZ4,
|
||
// ...
|
||
}
|
||
```
|
||
|
||
When you turn data compression on for an existing file system, only newly written files are compressed. Existing files are not compressed. For more information, see [Compressing previously written files](https://docs.aws.amazon.com/fsx/latest/LustreGuide/data-compression.html#migrate-compression).
|
||
|
||
### Backups
|
||
|
||
You can take daily automatic backups by setting `automaticBackupRetention` to a non-zero day in the `lustreConfiguration`.
|
||
|
||
Additionally, you can set the backup window by specifying the `dailyAutomaticBackupStartTime`.
|
||
|
||
```ts
|
||
import * as cdk from 'aws-cdk-lib';
|
||
|
||
const lustreConfiguration = {
|
||
// ...
|
||
automaticBackupRetention: cdk.Duration.days(3), // backup retention
|
||
copyTagsToBackups: true, // if true, tags are copied to backups
|
||
dailyAutomaticBackupStartTime: new fsx.DailyAutomaticBackupStartTime({ hour: 11, minute: 30 }), // backup window
|
||
// ...
|
||
}
|
||
```
|
||
|
||
For more information, see [Working with backups
|
||
](https://docs.aws.amazon.com/fsx/latest/LustreGuide/using-backups-fsx.html).
|
||
|
||
### Storage Type
|
||
|
||
By default, FSx for Lustre uses SSD storage. To use HDD storage, specify `storageType`:
|
||
|
||
```ts
|
||
declare const vpc: ec2.Vpc;
|
||
|
||
const fileSystem = new fsx.LustreFileSystem(this, 'FsxLustreFileSystem', {
|
||
lustreConfiguration: { deploymentType: fsx.LustreDeploymentType.PERSISTENT_1 },
|
||
storageCapacityGiB: 1200,
|
||
vpc,
|
||
vpcSubnet: vpc.privateSubnets[0],
|
||
storageType: fsx.StorageType.HDD,
|
||
});
|
||
```
|
||
|
||
**Note:** The HDD storage type is only supported for `PERSISTENT_1` deployment types.
|
||
|
||
To improve the performance of frequently accessed files by caching up to 20% of the total storage capacity of the file system, set `driveCacheType` to `READ`:
|
||
|
||
```ts
|
||
declare const vpc: ec2.Vpc;
|
||
|
||
const fileSystem = new fsx.LustreFileSystem(this, 'FsxLustreFileSystem', {
|
||
lustreConfiguration: {
|
||
deploymentType: fsx.LustreDeploymentType.PERSISTENT_1,
|
||
driveCacheType: fsx.DriveCacheType.READ,
|
||
},
|
||
storageCapacityGiB: 1200,
|
||
vpc,
|
||
vpcSubnet: vpc.privateSubnets[0],
|
||
storageType: fsx.StorageType.HDD,
|
||
});
|
||
```
|
||
|
||
## FSx for Windows File Server
|
||
|
||
The L2 construct for the FSx for Windows File Server has not yet been implemented. To instantiate an FSx for Windows
|
||
file system, the L1 constructs can be used as defined by CloudFormation.
|