Takashi Iwamoto

脆弱性対応などで、依存パッケージの最低バージョンをpackage.jsonのoverridesに定義した場合、時が経って不要になったら削除したくなりませんか。 そんなときのために、古くなったバージョン定義を検出・削除できるツール「pnpm-override-prune」を作りました。 https://github.com/iwamot/pnpm-override-prune 以下、使い方のご紹介です。 検出 Prismaに、ちょうどよい例があったので題材にしてみます。 $ git clone https://github.com/prisma/prisma.git $ cd pris...

pnpmやaubeの古くなったoverridesを検出・削除する「pnpm-override-prune」

脆弱性対応などで、依存パッケージの最低バージョンをuvのoverride-dependenciesに定義した場合、時が経って不要になったら削除したくなりませんか。constraint-dependenciesも同様です。 そんなときのために、古くなったバージョン定義を検出・削除できるツール「uv-override-prune」を作りました。 https://github.com/iwamot/uv-override-prune 以下、使い方のご紹介です。 インストール uv向けのツールなので、uv tool install でインストールするのが分かりやすいと思います。 $ uv t...

uvの古くなったoverride-dependenciesを検出・削除する「uv-override-prune」

Git 2.54 (2026-04-20) で追加されたconfig-based hooksと、miseのremote Git includes機能を使って、複数リポジトリに対してフック設定の共有を簡単にする仕組みを作りました。どんなものをどんなふうに作ったのか、ご紹介します。 実現できたこと 今回の仕組みにより、フックを利用したいリポジトリで mise run enable-git-hooks を実行することで、.git/config に以下の内容が設定されるようになりました。 [hook "validate"] event = pre-commit ...

Gitのconfig-based hooksをmiseで共有する

AIでコードを量産した結果、管理するGitHubリポジトリが増えて、手間に悩んでいる方も多いのではないでしょうか。いくつものリポジトリで似たような設定変更をくり返すのは、つらいですよね。 運用をDRYにしたくて考えたのが、今回ご紹介する「機能特化リポジトリ」構成です。小さな役割を果たすリポジトリをいくつか配置して、メインとなるリポジトリを支えます。 実装したばかりで、まだ調整が必要かもしれませんが、今のところ気に入っているので紹介させてください。 機能特化リポジトリ 現在、5つの機能特化リポジトリを配置しています。 workflows：再利用可能なワークフロー actions：...

GitHubの運用をDRYにする「機能特化リポジトリ」構成

パワポ作るマンをMCP Apps化してみた

8万デプロイ

I had an Amazon RDS DB instance with public access enabled due to certain requirements. When public access was no longer needed, I migrated it to a private subnet. By using an Egress-Only Internet Gateway (Egress-Only IGW), I completed the migration without adding paid resources like NAT Gateways or VPC endpoints. In this article, I'll share the background and the actual migration steps. Background The reason I wanted to move the DB to a private subnet was that public access was no longer required. Public access had been enabled because Lambda functions in multiple AWS accounts needed to access this DB. While VPC peering was an option, I chose simple public access since IAM authentication alone seemed sufficient. Later, the requirements changed—only Lambda functions in the same account as the DB needed access. This meant I could finally migrate to a private subnet. The Challenge: Connecting to AWS Service Endpoints However, there was one hurdle to overcome: "How do Lambda functions connect to AWS service endpoints?" The Lambda functions needed to access not only the DB but also AWS service endpoints—specifically Athena, STS, and CloudWatch. Attaching a VPC to the Lambda function would enable access to the DB in the private subnet. But it would also cut off the route to service endpoints. My first thought was to use paid NAT Gateways or VPC endpoints. But I wanted to complete the migration without adding costs if possible. Egress-Only IGW: A Cost-Free Solution After some thought, I came up with the solution: Egress-Only IGW. It has the following characteristics: Used only for IPv6 traffic Allows outbound communication from VPC to the internet (inbound connections cannot be initiated from the internet) No charge This means if the AWS service endpoints support IPv6, Lambda functions with VPC attached can connect to them for free. You can check IPv6 support for service endpoints in this documentation. Athena, STS, and CloudWatch all support IPv6 (note: the STS global endpoint does not support IPv6; regional endpoints are required), so the following migration should be possible: Component Before After DB instance Public subnet Private subnet Lambda → DB Internet Within VPC Lambda → AWS services IPv4 IPv6 (via Egress-Only IGW) Migration Steps With the goal set, I proceeded with the following steps: Add IPv6 support to the VPC Update Lambda function configuration Migrate DB to private subnet Delete unnecessary resources 1. Add IPv6 Support to the VPC First, I added IPv6 support to the VPC, including associating CIDR blocks and updating route tables. See the documentation for details. 2. Update Lambda Function Configuration Next, I updated the Lambda function configuration. Key points: Allow IPv6 egress in the security group Set the environment variable AWS_USE_DUALSTACK_ENDPOINT=true (reference) Attach VPC and enable "Allow IPv6 traffic for dual-stack subnets" (reference) The environment variable tells boto3 (used by the Lambda function) to send requests to IPv4/IPv6 dual-stack service endpoints. 3. Migrate DB to Private Subnet Then, I migrated the DB to the private subnet. This was the main task. I expected it to be simple, but you cannot directly migrate a DB from a public subnet to a private subnet. It required some extra steps. Specifically, you need to follow steps like "disable Multi-AZ → edit subnet group → enable Multi-AZ → ..." See this article for details. After migration, I edited the DB's security group: Before: Allow access from 0.0.0.0/0 After: Allow access from the Lambda function's security group 4. Delete Unnecessary Resources Finally, I deleted the now-unnecessary public subnet and the regular Internet Gateway. Migration complete. Conclusion That's how I used Egress-Only IGW to make RDS private without extra cost. It was a great opportunity to work with this feature in a real-world scenario. This example used Lambda, but the approach should apply to other services like ECS as well—assuming the target endpoints support IPv6. If you're choosing a public subnet due to cost-security trade-offs, Egress-Only IGW might be your solution. Give it a try!

How I Made RDS Private Without Extra Cost Using Egress-Only IGW

At my current workplace, we use Seekable OCI (SOCI) to speed up Amazon ECS task launches on AWS Fargate. Recently, we migrated the SOCI Index Manifest version from v1 to v2. In this article, I'll explain why and how we did it. Why We Migrated The reason is simple: v1 support is ending. AWS sent out the following notification: After careful consideration, we have made the decision to discontinue the support for SOCI Index Manifest v1 for Amazon ECS customers using AWS Fargate launch mode, effective February 9, 2026. Until this date, AWS Fargate customers using Index Manifest v1 will be able to leverage existing indexes and create new indexes using Index Manifest v1 to deploy applications. Without migrating to v2, task launches would become slower. How We Migrated The migration method is also straightforward. In short, we "deployed the new SOCI Index Builder that supports v2." We had been using a different solution with the same name, but it was deprecated without v2 support, so we switched. Here are the general steps: 1. Prepare the Tool Clone the SOCI Index Builder Git repository: git clone https://github.com/awslabs/cfn-ecr-aws-soci-index-builder.git cd cfn-ecr-aws-soci-index-builder 2. Edit .taskcat.yml Next, edit .taskcat.yml. This mainly involves changing the deployment region. For example, if you only want to accelerate images in the Tokyo region ECR repository: --- a/.taskcat.yml +++ b/.taskcat.yml @@ -12,11 +12,5 @@ tests: parameters: SociRepositoryImageTagFilters: "*:*" regions: - - us-east-1 - - us-east-2 - - us-west-1 - - us-west-2 - - eu-west-1 - - eu-west-2 - - eu-west-3 + - ap-northeast-1 template: templates/SociIndexBuilder.yml 3. Run taskcat upload Then run taskcat upload. This uploads the CloudFormation templates and Lambda function packages to S3. I recommend running taskcat via Docker rather than installing it locally, as it doesn't support Python 3.14 yet and you probably won't use it very often anyway. Here's an example of a dry run. Remove --dry-run to actually upload: docker run --rm \ -v $(pwd):/mnt \ -v /var/run/docker.sock:/var/run/docker.sock \ -e AWS_ACCESS_KEY_ID \ -e AWS_SECRET_ACCESS_KEY \ -e AWS_SESSION_TOKEN \ -e GIT_PYTHON_REFRESH=quiet \ taskcat/taskcat taskcat upload --dry-run 4. Create the CloudFormation Stack Finally, create a stack using the uploaded CloudFormation template. As documented, the easiest way is through the Management Console. For reference, here's an example of creating a stack in a single region using Terraform: locals { repository_image_tag_filters = ["*:*"] } data "aws_s3_object" "template" { bucket = "&lt;your-s3-bucket-name&gt;" key = "cfn-ecr-aws-soci-index-builder/templates/SociIndexBuilder.yml" } resource "aws_cloudformation_stack" "this" { capabilities = ["CAPABILITY_IAM"] name = "cfn-ecr-aws-soci-index-builder" parameters = { IamPermissionsBoundaryArn = "none" QSS3BucketName = data.aws_s3_object.template.bucket QSS3KeyPrefix = "${split("/", data.aws_s3_object.template.key)[0]}/" SociIndexVersion = "V2" SociRepositoryImageTagFilters = join(",", local.repository_image_tag_filters) } template_url = format( "https://%s.s3.%s.amazonaws.com/%s", data.aws_s3_object.template.bucket, data.aws_s3_object.template.region, data.aws_s3_object.template.key ) } Conclusion That's how we migrated the SOCI Index Manifest from v1 to v2 for ECS/Fargate. If you've been using v1 and haven't migrated yet, I recommend doing so; it's not that much work. Haven't adopted SOCI at all? You can easily speed up your task launches with these four steps. Give it a try!

Migrating SOCI Index Manifest from v1 to v2 for ECS/Fargate

AIエージェント・マイクロサービス時代。AWSでの手軽な構築法を考えて試してみた

これがLambdaレス時代のChatOpsだ！実例で学ぶAmazon Q Developerカスタムアクション活用法

Developer Certificate of Origin、よさそう

復号できなくなると怖いので、AWS KMSキーの削除を「面倒」にしてみた CODT 2025 クロージングイベント版

Strands Agentsは、AWSが公開したオープンソースのAIエージェントSDKです。数行のコードでAIエージェントが作れます。 セッション管理機能を使うと、会話履歴やAIエージェントの状態が永続化できます。組み込みの保存先はローカルファイルシステムとS3です。 今回は、Strands Agentsを単なるS3 KVSクライアントとして使ってみます。LLMは呼び出しません。そういう使い方が便利な場合もあるでしょうから。 Strands Agentsをすでに利用中のアプリケーションで、何かしらのデータを永続化したい場合 プロトタイピングでシンプルなKVSを使いたい場合 1. ...

Strands AgentsをLLM抜きでS3 KVSクライアントとして使ってみる

復号できなくなると怖いので、AWS KMSキーの削除を「面倒」にしてみた

IPA&AWSダブル全冠が明かす、人生を変えた勉強法のすべて

2年でここまで成長！AWSで育てたAI Slack botの軌跡

本日、GitHub Advanced Security認定を受けて合格しました。 https://www.credly.com/badges/9f7bdda5-af4c-4d26-bbd4-f5a373de0f35/public_url 受験料はメンテナー月間のセキュリティチャレンジでもらったバウチャーのおかげで無料でした。本来は99 USDです。 バウチャーをもらうまでのステップは下記のとおりでした。 3つのエクササイズを実施（リポジトリをforkして指示に従う） Introduction to Repository Management Secure Your Repositor...

GitHub Advanced Security認定をタダで取得した方法

Amazon Q CLIでゲームを作るとTシャツがもらえるイベントが開催されているので、ぼくも作ってみました。 盤面に置かれている赤い球を2つの手球で順に弾き、最終的に3つの球で作られる三角形の大きさがスコアになるゲームです。 所要時間は、Amazon Q CLIやPygameのインストールを含めて2時間半ほどでした。ただし、どんなゲームを作るか決めるのに1時間くらい要した形です。 Amazon Q CLIの書いてくれたコードは、下記リポジトリで公開しています。READMEやLICENSEも書いてもらいました。 https://github.com/iwamot/amazon-q-bi...

Amazon Q CLIでゲームが簡単に作れた

Hi, I'm Takashi Iwamoto (@iwamot), an AWS Community Builder (Cloud Operations). When I looked for prior examples of building ChatOps on AWS, most of them used Lambda. However, I had a hunch that we could get rid of Lambda by using the Custom Actions feature of Amazon Q Developer in chat applications (formerly AWS Chatbot), which was announced in November 2023. After trying it out, I was able to build a fairly practical architecture. In this post, I'll walk you through how I built AWS ChatOps without Lambda. Why Avoid Lambda Before we dive in, let me touch on why I want to avoid Lambda. In a word: "Because it makes operations easier." If you don't use Lambda, you never have to worry about runtime end‑of‑life. The more ChatOps targets you have, the more the cost of those EOL updates will hurt. Services Used &amp; Overall Architecture Here's what I actually built. The main AWS services I used are these three: Amazon EventBridge – event detection &amp; routing Amazon SNS – notification delivery Amazon Q Developer in chat applications – Slack notifications &amp; Custom Actions And the overall architecture looks like this: An EventBridge rule detects the CodeDeploy deployment state‑change event. The event is transformed with an input transformer and routed to an SNS topic. The message is delivered to an Amazon Q Developer Slack channel. The notification message and Custom Action buttons appear in Slack. A user picks one of the Custom Actions. The corresponding CodeDeploy command is executed (e.g. continue-deployment, stop-deployment). Creating &amp; Associating Custom Actions To show Custom Action buttons in chat apps like Slack, you first need to create the Custom Actions themselves. There are two ways to define Custom Actions: Create them directly in the chat application Create them with the Amazon Q Developer in chat applications API I chose the API method because I wanted to manage everything as IaC. If you create the action in the chat UI, it seems that the resource is owned by AWS and not created inside your own AWS account. If you go the API route, you'll also need to associate the Custom Actions with the chat channel yourself. The steps are: Call the CreateCustomAction API to create the action. Call the AssociateToConfiguration API to attach it to the chat channel. Below is how I implemented it in Terraform. (It's a bit long, so I've collapsed it.) Definition of Custom Actions locals { custom_actions = { SwitchTasks = { button_text = "🔁 Switch task sets" criteria = [ { operator = "EQUALS" value = "blue-green-deployment" variable_name = "ActionGroup" }, ] variables = { "ActionGroup" = "event.metadata.additionalContext.ActionGroup" "DeploymentId" = "event.metadata.additionalContext.DeploymentId" "Region" = "event.metadata.additionalContext.Region" } command_text = "codedeploy continue-deployment --deployment-id $DeploymentId --region $Region --deployment-wait-type READY_WAIT" } RollbackDeployment = { button_text = "🔙 Rollback" criteria = [ { operator = "EQUALS" value = "blue-green-deployment" variable_name = "ActionGroup" }, ] variables = { "ActionGroup" = "event.metadata.additionalContext.ActionGroup" "DeploymentId" = "event.metadata.additionalContext.DeploymentId" "Region" = "event.metadata.additionalContext.Region" } command_text = "codedeploy stop-deployment --deployment-id $DeploymentId --region $Region --auto-rollback-enabled true" } TerminateOldTask = { button_text = "🧹 Terminate old task set" criteria = [ { operator = "EQUALS" value = "blue-green-deployment" variable_name = "ActionGroup" }, ] variables = { "ActionGroup" = "event.metadata.additionalContext.ActionGroup" "DeploymentId" = "event.metadata.additionalContext.DeploymentId" "Region" = "event.metadata.additionalContext.Region" } command_text = "codedeploy continue-deployment --deployment-id $DeploymentId --region $Region --deployment-wait-type TERMINATION_WAIT" } } } resource "awscc_chatbot_custom_action" "this" { for_each = local.custom_actions action_name = each.key attachments = [ { button_text = each.value.button_text criteria = each.value.criteria notification_type = "Custom" variables = each.value.variables }, ] definition = { command_text = each.value.command_text } tags = [ for key, value in merge(data.aws_default_tags.this.tags, { "Name" = each.key }) : { key = key value = value } ] } Associating the Custom Actions with the Chat Channel (implemented with null_resource) # In this example we set custom_action_groups = ["blue-green-deployment"] variable "custom_action_groups" { description = "A list of Custom Action group names to associate specific AWS Chatbot actions with the specified Slack channel configuration." type = list(string) default = [] } locals { action_group_map = { "blue-green-deployment" = ["SwitchTasks", "RollbackDeployment", "TerminateOldTask"] } selected_actions = distinct(flatten([ for group in var.custom_action_groups : lookup(local.action_group_map, group, []) ])) action_arns = { for k, v in { for action in local.selected_actions : action =&gt; try( one([ for id in data.awscc_chatbot_custom_actions.this.ids : id if endswith(id, "/${action}") ]), null ) } : k =&gt; v if v != null } } resource "null_resource" "associate_actions" { for_each = local.action_arns triggers = { action_arn = each.value chatbot_arn = aws_chatbot_slack_channel_configuration.this.chat_configuration_arn } provisioner "local-exec" { command = "aws chatbot associate-to-configuration --resource ${self.triggers.action_arn} --chat-configuration ${self.triggers.chatbot_arn} --region us-west-2" } provisioner "local-exec" { when = destroy command = "aws chatbot disassociate-from-configuration --resource ${self.triggers.action_arn} --chat-configuration ${self.triggers.chatbot_arn} --region us-west-2" } } EventBridge Rule variable "sns_topic_arn" { description = "The SNS topic to notify when the blue/green deployment is ready for validation." type = string nullable = false } variable "deployment_group_name" { description = "The name of the deployment group targeted by ChatOps." type = string nullable = false } variable "green_url" { description = "The URL for validating the green environment." type = string nullable = false } resource "aws_cloudwatch_event_rule" "this" { description = null event_bus_name = "default" event_pattern = jsonencode( { detail = { deploymentGroup = [ var.deployment_group_name, ] state = [ "READY", ] } detail-type = [ "CodeDeploy Deployment State-change Notification", ] source = [ "aws.codedeploy", ] } ) force_destroy = false name = "${var.deployment_group_name}-deployment-ready" name_prefix = null role_arn = null schedule_expression = null state = "ENABLED" tags = { "Name" = "${var.deployment_group_name}-deployment-ready" } } resource "aws_cloudwatch_event_target" "this" { arn = var.sns_topic_arn event_bus_name = "default" force_destroy = false input = null input_path = null role_arn = aws_iam_role.this.arn rule = aws_cloudwatch_event_rule.this.name target_id = "sns" input_transformer { input_paths = { "account" = "$.account" "deploymentGroup" = "$.detail.deploymentGroup" "deploymentId" = "$.detail.deploymentId" "region" = "$.region" "time" = "$.time" } input_template = replace(replace( jsonencode( { content = { description = trimspace( &lt;&lt;-EOT *Deployment group:* &lt;deploymentGroup&gt; *Deployment ID:* &lt;deploymentId&gt; *Account:* &lt;account&gt; *Region:* &lt;region&gt; *Time:* &lt;time&gt; EOT ) nextSteps = [ "Check if the app works correctly at ${var.green_url}", "If all looks good, switch task sets. If not, trigger a rollback", "Once validated, terminate the original task set", ] textType = "client-markdown" title = ":large_blue_circle::large_green_circle: Deployment is now ready for validation" } metadata = { additionalContext = { ActionGroup = "blue-green-deployment" DeploymentId = "&lt;deploymentId&gt;" Region = "&lt;region&gt;" } enableCustomActions = true } source = "custom" version = "1.0" } ), "\u003c", "&lt;"), "\u003e", "&gt;") } } With this in place, Slack now shows the following notification, and the entire Blue/Green deployment can be operated using just the Custom Action buttons. Remaining Challenge As you can see, this solution works quite well, but one issue is that we can't control the order of the Custom Action buttons. Sometimes they show up as "🧹 Terminate old task set", "🔙 Rollback", "🔁 Switch task sets", in that order. The screenshot above just happened to be in the desired order. If we could control that order, I'm sure this approach would be solid enough to become a go-to ChatOps pattern. I've already submitted a feature request to AWS Support, and I'm hopeful it will be addressed in the future. Conclusion: ChatOps Without Lambda Is Possible That was an example of ChatOps built with Amazon Q Developer in chat applications Custom Actions. When you build it without Lambda, operations get easier. The unresolved button‑ordering issue remains, but if you can live with that, it's already perfectly usable. I plan to expand our ChatOps coverage with this method, and I hope it inspires you to try it out yourself.

AWS ChatOps Without Lambda? Yes, You Can!

名単体テスト 禁断の傀儡（モック）

クォータ監視、AWS Organizations環境でも楽勝です✌️

Cline、めっちゃ便利、お金が飛ぶ💸

開発組織を進化させる！AWSで実践するチームトポロジー

始めないともったいない！SLO運用で得られる3つのメリット

あなたの人生も変わるかも？AWS認定2つで始まったウソみたいな話

効率的な技術組織が作れる！書籍『チームトポロジー』要点まとめ

AWS⼊社という選択肢、⾒えていますか

CodeBuildの失敗ビルドを自動的にLambdaで再試行する

If you want to change the network settings for an Amazon ECS service using blue/green deployment with AWS CodeDeploy, you need to edit the AppSpec file and trigger a new deployment. Without knowing this method, you'd either have to recreate the entire ECS service or give up on changing the network settings. Background For ECS services using blue/green deployment with AWS CodeDeploy, you cannot directly change the network settings. When you try to make changes by calling the ECS UpdateService API, you will encounter the following error: InvalidParameterException: Unable to update network parameters on services with a CODE_DEPLOY deployment controller. Use AWS CodeDeploy to trigger a new deployment. This error message indicates that you need to trigger a new deployment with CodeDeploy, as noted in the documentation. For services using the blue/green (CODE_DEPLOY) deployment controller, only the desired count, deployment configuration, health check grace period, task placement constraints and strategies, enable ECS managed tags option, and propagate tags can be updated using this API. If the network configuration, platform version, task definition, or load balancer need to be updated, create a new AWS CodeDeploy deployment. For more information, see CreateDeployment in the AWS CodeDeploy API Reference. UpdateService - Amazon Elastic Container Service Solution Now that we know we need to trigger a new deployment, the question remains: how exactly should we do that? The answer is to "edit the AppSpec file and then trigger a new deployment." If you trigger it without editing the AppSpec file, the network settings won't change. In the case of appspec.yml, edit it as shown below. NetworkConfiguration: AwsvpcConfiguration: Subnets: ["subnet-1234abcd","subnet-5678abcd"] SecurityGroups: ["sg-12345678"] AssignPublicIp: "ENABLED" AppSpec File example - AWS CodeDeploy Specify the subnets, security groups, and whether or not to assign a public IP address for the ECS tasks in the AppSpec file. When doing so, you need to specify all three: Subnets, SecurityGroups, and AssignPublicIp. All or none of the settings under NetworkConfiguration must be specified. For example, if you want to specify Subnets, you must also specify SecurityGroups and AssignPublicIp. If none is specified, CodeDeploy uses the current network Amazon ECS settings. AppSpec 'resources' section (Amazon ECS and AWS Lambda deployments only) - AWS CodeDeploy After editing the AppSpec file, trigger a new deployment, and the network settings will be applied. The new ECS tasks in the green environment should start with the specified network configuration. Once the deployment is complete, you can safely remove the NetworkConfiguration from the AppSpec file. Conclusion As described, even for Amazon ECS services using blue/green deployment with CodeDeploy, you can change the network settings by editing the AppSpec file and triggering a new deployment. I almost ended up recreating the ECS service because I wasn't aware of this method. I hope this article helps those in a similar situation.

How to Change Network Configurations for Blue/Green Amazon ECS Services

40代後半で開発エンジニアからクラウドインフラエンジニアにキャリアチェンジし、生き残れる自信がようやく持てた話

In this article, I’ll introduce an SLO error budget monitoring system that I built entirely using AWS services. System Overview This system is based on the multi-window, multi-burn-rate alerting method proposed in the Site Reliability Workbook. It sends alerts to Slack when the SLO error budget burn rate exceeds a certain threshold. expr: ( job:slo_errors_per_request:ratio_rate1h{job="myjob"} &gt; (14.4*0.001) and job:slo_errors_per_request:ratio_rate5m{job="myjob"} &gt; (14.4*0.001) ) or ( job:slo_errors_per_request:ratio_rate6h{job="myjob"} &gt; (6*0.001) and job:slo_errors_per_request:ratio_rate30m{job="myjob"} &gt; (6*0.001) ) severity: page expr: ( job:slo_errors_per_request:ratio_rate24h{job="myjob"} &gt; (3*0.001) and job:slo_errors_per_request:ratio_rate2h{job="myjob"} &gt; (3*0.001) ) or ( job:slo_errors_per_request:ratio_rate3d{job="myjob"} &gt; 0.001 and job:slo_errors_per_request:ratio_rate6h{job="myjob"} &gt; 0.001 ) severity: ticket https://sre.google/workbook/alerting-on-slos/ The Site Reliability Workbook recommends combining multiple time windows and burn rates to create more accurate alerts. For example, if you only alert based on a short time window, you may get too many false positives. On the other hand, if you rely on a longer time window, it may take too long for an alert to trigger or recover. By using both short and long windows, the system can generate more precise alerts. For more detailed explanations of this alerting method, I encourage you to check out the Site Reliability Workbook, which is available for free. Personally, I believe that strictly following these guidelines is the best way to monitor service reliability. Why I Implemented This The primary motivation behind this monitoring system was to solve the problem of "alert fatigue." As an operations engineer, I was constantly bombarded with alerts that didn’t actually require any action. Previously, we monitored CPU and memory usage metrics directly, which led to alerts even when the service itself was running smoothly. For example, we would receive a Slack alert if the CPU usage of a web server exceeded 90%, regardless of whether there were any issues with availability or latency. However, even if the CPU usage was high, if the service remained available and responsive, there was no need for intervention. Over time, receiving alerts for issues that didn't require action caused fatigue and increased the risk of missing critical alerts. Missing an important alert could damage service reliability and lead to a loss of users. To address this, I decided to focus only on critical indicators—such as availability and latency—and set up an SLO-based monitoring system that tracks the error budget burn rate. Why I Used AWS Services Only At first, I considered using third-party monitoring tools like New Relic or Datadog. However, after investigating these tools in early 2023, I found that it was either impossible or very difficult to implement the kind of SLO monitoring I needed with their existing features. I worked with support teams from both platforms, but we couldn't come up with a viable solution. On the other hand, by combining AWS services such as Amazon CloudWatch, I could implement the necessary monitoring and at a much lower cost—around $20 per month. Since most of our workloads are hosted on AWS, building the monitoring system directly on AWS made it easier to integrate everything into a single console. AWS also offers a high degree of flexibility in terms of combining services, giving us more room to scale and customize compared to third-party tools like New Relic or Datadog. Architecture Here’s an overview of the system’s architecture: Although it may look a bit complex at first glance, the core workflow is simple: EventBridge Scheduler triggers a Lambda function at regular intervals. The Lambda function runs an Athena query to aggregate ALB access logs. The results are stored in both CloudWatch custom metrics and RDS. CloudWatch composite alarms use these metrics to trigger alerts based on the SLO conditions. The Lambda function is triggered by EventBridge instead of an S3 file save event. This is because when there is heavy traffic, multiple log files are saved to S3, which could result in the aggregation process being run multiple times unnecessarily. We use RDS to store the aggregated results in order to reduce Athena scan costs. For more details on these optimizations and custom metric definitions, please refer to my presentation at JAWS PANKRATION 2024. Results of the Implementation Since we introduced this system, alert fatigue has significantly decreased. I personally noticed a reduction in the number of unnecessary alerts that required no action. Additionally, the development team has become more invested in the reliability of the services. Now, when an alert is triggered, the team takes the initiative to tune performance or even revise the SLOs when necessary. As of September 2023, this system is monitoring 7 services and 12 critical user journeys. Operational Tips To streamline operations, I developed an internal Terraform module that allows us to quickly accommodate additional monitoring requests from the development team. While the module itself is private, this is an actual example of its interface, which could be useful as a reference if you plan to implement a similar module. module "slomon" { source = "git@github.com:enechange/terraform-modules.git//slomon-for-alb?ref=v0.53.0" environment_name = "prod-enechange" alb_access_logs_s3_url = local.alb_access_logs_s3_url sns_topic_names_for_paging = ["cto-incident-enechange"] sns_topic_names_for_ticketing = ["cto-alert-enechange"] critical_user_journeys = { input1_submit = { http_method = "POST" path = "/try/input1_submit" dashboard_order = 1 slo = { availability_target = 95.0 latency_p95_threshold = 4.0 latency_p50_threshold = 3.0 } } } } This Terraform module also includes Athena queries for analyzing historical data, allowing us to quickly set SLOs based on past performance. Future Outlook At present, this monitoring system meets all of our needs. However, if CloudWatch's Application Signals feature evolves to support the required conditions for SLO monitoring, this custom system may no longer be necessary. In that case, we would likely transition to a fully managed solution. Likewise, if New Relic or Datadog introduce features that support the kind of SLO monitoring we need, we may consider switching to them. These APM tools generally offer better capabilities for diagnosing root causes of performance degradation compared to CloudWatch. Ultimately, we will continue to assess the best solution for our needs and adapt our system as new tools and services emerge. Conclusion In this article, I introduced an SLO error budget monitoring system that I built using AWS services only. By implementing this system, we were able to reduce alert fatigue and increase the development team’s focus on service reliability. I hope this article provides useful insights for those dealing with similar challenges around alert fatigue.

Implementing SLO Error Budget Monitoring with AWS Services Only

Key Resultにはアクションを置くべきでない？ 「OKRのKey Resultにはアクションを置くべきでない」との意見を目にします。 Perdoo は Objective を達成するために必要な仮説であり、施策であるアクションのことを Initiative とよんでいる。 OKR の利用において Key Result の内容が Initiative になってしまっているということがよくあるようです。 Objective 頻繁にバズるブロガーになる Key Results 月に2回以上はてなブックマーク50以上の記事がある Initiatives 月に20記事を...

OKRのKey Resultにはアクションを置いてもよい

コールドスタートに4～5秒かかるPythonのLambdaコンテナ関数があり、改善策を調べていたら、AWS公式ブログに下記の記述を見つけました。 AWS が提供するベースイメージは、Lambda サービスによってプロアクティブにキャッシュされます。つまり、ベースイメージは近くの別のアップストリームキャッシュにあるか、ワーカーインスタンスキャッシュにすでに存在しています。はるかに大きいにもかかわらず、キャッシュされないかもしれないサードパーティのベースイメージと比較するとデプロイ時間が短くなる可能性があります。 コンテナイメージとしてパッケージ化された Lambda 関数の最適化 | A...

Lambdaコンテナ関数ではAWS提供ベースイメージを使う

AWS LambdaのPythonコードでBugsnagを使う際、デフォルトの「asynchronous=True」のままだと、通知が飛びませんでした。深く追ってはいませんが、非同期的な通知より先にLambda関数が終了してしまうのでしょう。 「asynchronous=False」を指定したところ、意図どおり通知が飛ぶようになりました。 以下、参考リソースです。 By default, requests are sent asynchronously to BugSnag. If you would like to block until the request is done, y...

AWS LambdaのPythonコードでBugsnagを使う際は「asynchronous=False」にする

Terraformで、特定の条件下でのみブロックを指定したい場面があります。たとえば、AWS Budgetsの予算を示すaws_budgets_budgetでは、予算タイプによってcost_filterブロックの要否が異なります。 条件ごとにリソースを定義してもよいのですが、テンプレートが冗長になりがちです。 下記の例では、try関数を使って、必要なときのみcost_filterブロックを指定してみました。すっきりしていて読みやすく感じます。 locals { budgets = { # RDSリザーブドインスタンスのカバレッジ予算。cost_filtersが必要 r...

Terraformのtry関数をdynamicブロックの作成条件に使う

AWS App Runnerを学ぶためtDiaryをデプロイしてみる

Terraformのawsプロバイダが未対応のAWS Chatbot、awsccなら対応しているとのことで、試してみました。 https://hi1280.hatenablog.com/entry/2022/08/06/191154 既存のSlackチャネルをインポートするところから始めようと、ドキュメントを見たところ、下記のように書かれていました。 Import is supported using the following syntax: $ terraform import awscc_chatbot_slack_channel_configuration.example &l...

awsccでChatbotのSlackチャネルをterraform importする

AWS Transfer Familyサーバーで、SETSTATコマンドを無視できるようになりました。2022年6月のことです。 https://aws.amazon.com/jp/about-aws/whats-new/2022/06/aws-transfer-family-expands-server-configuration-options-set-clients/ ファイルをアップロードするとき、ファイル転送クライアントは SETSTAT コマンドを発行してリモートファイルの属性を変更します。ただし、SETSTAT コマンドはオブジェクトストレージシステムと互換性がないため、...

AWS Transfer FamilyサーバーでSETSTATコマンドを無視する

AWS Transfer Familyのドキュメントに誤りがあったので、報告したら、直してもらえました。 下記のアクセスポリシーが修正対象なのですが、どこが誤りか分かりますか。 { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowListingOfUserFolder", "Action": [ "s3:ListBucket" ], "Effect": "Allow...

AWS Transfer Familyのドキュメントの誤りを直してもらった

CodeBuildトリガーの実体はEventBridgeルールなんですね。ドキュメントにも書いてありました。 CodeBuild は、ビルドトリガーに Amazon EventBridge ルールを使用します https://docs.aws.amazon.com/ja_jp/codebuild/latest/userguide/trigger-create.html なので、TerraformのAWSプロバイダでトリガーを作る場合は、aws_codebuild_projectではなく、aws_cloudwatch_event_ruleやaws_cloudwatch_event_ta...

CodeBuildトリガーの実体はEventBridgeルール

運営中サービスで提供しているフィードの構文が正しいか気になったので、バリデータを探したところ、W3C Feed Validation Serviceが見つかりました。さっそく検証すると、下記の指摘が。 Error element must be an RFC-822 date-time Warning Missing atom:link with rel="self" item should contain a guid element Ensure description precedes content:encoded pubDateの日時がRFC 822形式でなかっ...

W3C Feed Validation Serviceを使ってフィードを修正した

Terraform で list の末尾要素を参照するには、公式ドキュメントにある通り element 関数と length 関数を組み合わせます。 To get the last element from the list use length to find the size of the list (minus 1 as the list is zero-based) and then pick the last element: &gt; element(["a", "b", "c"], length(["a", "b", "c"])-1) c 引用元：element - F...

Terraform で list の末尾要素を参照する

AWSがデジタルバッジの発行を始めたので、さっそく取得してみました。 取得したのは以下2つのバッジです。2022年2月現在は、この「AWS Learning: Block Storage」と「AWS Learning: Object Storage」のみが存在します。 https://www.credly.com/badges/5b9371f0-9c6c-413a-b099-1aff5dafbc17/public_url https://www.credly.com/badges/3c35fc05-c3c1-4d82-b79a-250e5b4714cc/public_url いずれのバッ...

AWSストレージのデジタルバッジを2つ取得した #AWSTraining

Elastic Beanstalk の CLI で使える eb deploy コマンド には --nohang オプションがあります。現時点ではなぜかドキュメントに書かれていないオプションですが、ヘルプに書かれています。 $ eb --version EB CLI 3.20.2 (Python 3.7.2) $ eb deploy --help usage: eb deploy &lt;environment_name&gt; [options ...] Deploys your source code to the environment. positional argument...

eb deploy には --nohang オプションがある

AWS BugBust re:Invent Challenge とは AWS re:Invent 2021 のイベントとして AWS BugBust re:Invent Challenge が開催されました。簡単に言うと「指定されたバグを多く修正すると賞金や賞品がもらえるコンテスト」です。詳しくは AWS ブログの記事をご参照ください。 https://aws.amazon.com/jp/blogs/news/help-make-bugbusting-history-at-aws-reinvent-2021/ 修正したバグの例 ぼくもこのイベントに参加して 129 件のバグを修正...