文档

mc batch generate

MinIO 版本中已更改: RELEASE.2022-10-08T20-11-00Z 或更高版本

语法

mc batch generate 命令会为指定的作业类型创建一个基本的 YAML 格式的模板文件。

MinIO 创建文件后,请在您首选的文本编辑器工具中打开它以进一步自定义。您可以在每个批处理文件中定义一个作业任务定义。

有关您可以生成的受支持作业,请参见 作业类型

以下命令会为 myminio 别名的 mybucket 存储桶创建一个基本的 YAML 文件,用于复制作业。

mc batch generate myminio replicate

该命令具有以下语法

mc [GLOBALFLAGS] batch generate \
                       ALIAS   \
                       JOBTYPE
  • 方括号 [] 表示可选参数。

  • 共享一行参数是相互依赖的。

  • 使用管道 | 运算符分隔的参数是相互排斥的。

在终端/shell 中运行命令之前,请将示例复制到文本编辑器中并根据需要进行修改。

参数

ALIAS
必需

用于生成 YAML 模板文件的 别名。指定的 alias 不会限制您可以在其中使用生成的文件的部署。

例如

mc batch generate myminio replicate
JOBTYPE
必需

要为其生成 YAML 文档的作业类型。

支持以下值

全局标志

此命令支持任何 全局标志

示例

为复制作业类型生成 yaml 文件

以下命令生成复制类型批处理作业的 YAML 蓝图,并将文件命名为 replicate,扩展名为 .yaml

mc batch generate alias replicate > replicate.yaml
  • alias 替换为要用于生成 yaml 文件的 alias

  • replicate 替换为要为其生成 yaml 文件的作业类型。

    :mc:mc batch 支持 replicatekeyrotate 作业类型。

S3 兼容性

mc 命令行工具旨在与 AWS S3 API 兼容,并已在 MinIO 和 AWS S3 上进行了测试,以确保预期的功能和行为。

MinIO 不保证其他与 S3 兼容的服务,因为它们的 S3 API 实现未知,因此不受支持。虽然 mc 命令可能按文档工作,但任何此类使用均需自行承担风险。

作业类型

mc batch 当前支持以下作业任务类型

  • replicate

    在两个 MinIO 部署之间复制对象。提供类似于 存储桶复制 的功能,但作为批处理作业而不是持续扫描功能。

  • keyrotate

    MinIO 版本新增功能: RELEASE.2023-04-07T05-28-58Z

    旋转 MinIO 部署中静止对象上的 sse-s3 或 sse-kms 密钥。

  • expire

    MinIO 版本新增功能: RELEASE.2023-12-02T10-51-33Z

    使用类似于 自动对象过期 的语义,使对象过期。

replicate

您可以使用以下示例配置作为构建您自己的自定义复制批处理作业的起点

replicate:
  apiVersion: v1
  # source of the objects to be replicated
  source:
    type: TYPE # valid values are "s3" or "minio"
    bucket: BUCKET
    prefix: PREFIX # 'PREFIX' is optional
    # If your source is the 'local' alias specified to 'mc batch start', then the 'endpoint' and 'credentials' fields are optional and can be omitted
    # Either the 'source' or 'remote' *must* be the "local" deployment
    endpoint: "http[s]://HOSTNAME:PORT" 
    # path: "on|off|auto" # "on" enables path-style bucket lookup. "off" enables virtual host (DNS)-style bucket lookup. Defaults to "auto"
    credentials:
      accessKey: ACCESS-KEY # Required
      secretKey: SECRET-KEY # Required
    # sessionToken: SESSION-TOKEN # Optional only available when rotating credentials are used
    snowball: # automatically activated if the source is local
      disable: false # optionally turn-off snowball archive transfer
      batch: 100 # upto this many objects per archive
      inmemory: true # indicates if the archive must be staged locally or in-memory
      compress: false # S2/Snappy compressed archive
      smallerThan: 5MiB # create archive for all objects smaller than 5MiB
      skipErrs: false # skips any source side read() errors

  # target where the objects must be replicated
  target:
    type: TYPE # valid values are "s3" or "minio"
    bucket: BUCKET
    prefix: PREFIX # 'PREFIX' is optional
    # If your source is the 'local' alias specified to 'mc batch start', then the 'endpoint' and 'credentials' fields are optional and can be omitted

    # Either the 'source' or 'remote' *must* be the "local" deployment
    endpoint: "http[s]://HOSTNAME:PORT"
    # path: "on|off|auto" # "on" enables path-style bucket lookup. "off" enables virtual host (DNS)-style bucket lookup. Defaults to "auto"
    credentials:
      accessKey: ACCESS-KEY
      secretKey: SECRET-KEY
    # sessionToken: SESSION-TOKEN # Optional only available when rotating credentials are used

  # NOTE: All flags are optional
  # - filtering criteria only applies for all source objects match the criteria
  # - configurable notification endpoints
  # - configurable retries for the job (each retry skips successfully previously replaced objects)
  flags:
    filter:
      newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
      olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
      createdAfter: "date" # match objects created after "date"
      createdBefore: "date" # match objects created before "date"

      ## NOTE: tags are not supported when "source" is remote.
      # tags:
      #   - key: "name"
      #     value: "pick*" # match objects with tag 'name', with all values starting with 'pick'

      # metadata:
      #   - key: "content-type"
      #     value: "image/*" # match objects with 'content-type', with all values starting with 'image/'

    notify:
      endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
      token: "Bearer xxxxx" # optional authentication token for the notification endpoint

    retry:
      attempts: 10 # number of retries for the job before giving up
      delay: "500ms" # least amount of delay between each retry

有关每个密钥的更完整文档,请参阅 复制批处理作业参考

keyrotate

您可以使用以下示例配置作为构建您自己的自定义密钥旋转批处理作业的起点

keyrotate:
  apiVersion: v1
  bucket: BUCKET
  prefix: PREFIX
  encryption:
    type: sse-s3 # valid values are sse-s3 and sse-kms
    key: <new-kms-key> # valid only for sse-kms
    context: <new-kms-key-context> # valid only for sse-kms

  # optional flags based filtering criteria
  # for all objects
  flags:
    filter:
      newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
      olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
      createdAfter: "date" # match objects created after "date"
      createdBefore: "date" # match objects created before "date"
      tags:
        - key: "name"
          value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
      metadata:
        - key: "content-type"
          value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
      kmskey: "key-id" # match objects with KMS key-id (applicable only for sse-kms)
    notify:
      endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
      token: "Bearer xxxxx" # optional authentication token for the notification endpoint
    retry:
      attempts: 10 # number of retries for the job before giving up
      delay: "500ms" # least amount of delay between each retry

有关每个密钥的更完整文档,请参阅 密钥旋转批处理作业参考

expire

您可以使用以下示例配置作为构建您自己的自定义过期批处理作业的起点

expire:
  apiVersion: v1
  bucket: mybucket # Bucket where this job will expire matching objects from
  prefix: myprefix # (Optional) Prefix under which this job will expire objects matching the rules below.
  rules:
    - type: object  # objects with zero ore more older versions
      name: NAME # match object names that satisfy the wildcard expression.
      olderThan: 70h # match objects older than this value
      createdBefore: "2006-01-02T15:04:05.00Z" # match objects created before "date"
      tags:
        - key: name
          value: pick* # match objects with tag 'name', all values starting with 'pick'
      metadata:
        - key: content-type
          value: image/* # match objects with 'content-type', all values starting with 'image/'
      size:
        lessThan: 10MiB # match objects with size less than this value (e.g. 10MiB)
        greaterThan: 1MiB # match objects with size greater than this value (e.g. 1MiB)
      purge:
          # retainVersions: 0 # (default) delete all versions of the object. This option is the fastest.
          # retainVersions: 5 # keep the latest 5 versions of the object.

    - type: deleted # objects with delete marker as their latest version
      name: NAME # match object names that satisfy the wildcard expression.
      olderThan: 10h # match objects older than this value (e.g. 7d10h31s)
      createdBefore: "2006-01-02T15:04:05.00Z" # match objects created before "date"
      purge:
          # retainVersions: 0 # (default) delete all versions of the object. This option is the fastest.
          # retainVersions: 5 # keep the latest 5 versions of the object including delete markers.

  notify:
    endpoint: https://notify.endpoint # notification endpoint to receive job completion status
    token: Bearer xxxxx # optional authentication token for the notification endpoint

  retry:
    attempts: 10 # number of retries for the job before giving up
    delay: 500ms # least amount of delay between each retry

有关每个密钥的更完整文档,请参阅 过期批处理作业参考