跳到主要內容
AI News HubLIVE
來源內容 · 翻譯待補全5 分鐘閱讀

待翻譯:A serverless, data-driven Git metrics dashboard using Amazon Quick Sight

文章摘要

AI 服務暫時不可用,以下為來源摘要,待恢復後補全翻譯:Learn how to build a fully serverless pipeline that automatically collects Git metrics from GitHub and GitLab and visualizes them in interactive Amazon Quick Sight dashboards, giving engineering teams near-real-time delivery analytics at low cost.

來源AWS Machine Learning Blog作者: Saurabh Singhal
待翻譯:A serverless, data-driven Git metrics dashboard using Amazon Quick Sight
回報錯誤

更正管道尚未開通,可先複製下方文章資訊留存。

查看更正說明
直接讀正文

AI 服務暫時不可用,以下為來源正文,待恢復後補全翻譯。

Git activity is one of the richest signals engineering teams produce that can provide continuous observability into development analytics. The challenge is extracting these Git metrics at scale, which has traditionally required hand-rolled extract, transform, and load (ETL) jobs, dedicated infrastructure, and ongoing maintenance. Further, with modern development tools becoming more prevalent, teams need a clear way to measure whether these tools are making developers faster, or if the investment is not paying off. The AWS AI-Driven Development Lifecycle (AI-DLC) framework makes a simple point: If you’re using AI coding tools, you need numbers to back it up. Set a baseline, track what changes, and keep watching. Otherwise, you’re flying blind. You won’t know whether AI is speeding things up, only padding the commit counts, or quietly introducing quality issues you did not expect. In this post, we introduce a serverless approach that automates Git metrics collection from GitHub and GitLab to surface live analytics using Amazon Quick Sight. Whether you’re tracking sprint velocity, assessing release readiness, or you need visibility into team patterns, this solution provides near-real-time insights into your Git platform activities. The serverless design abstracts infrastructure management while staying low-cost at scale. Solution overview This solution implements an automated, event-driven pipeline that collects repository metrics from GitHub and GitLab APIs automatically on schedule. It then processes them through a serverless orchestration workflow and persists the results in Amazon Simple Storage Service (Amazon S3) for graphical visualization using interactive Amazon Quick Sight dashboards. The key capabilities include: Intelligent change detection – A dedicated detector function monitors GitHub events API and GitLab activity feeds to determine whether changes like commits, pull requests, issues, repository creation or deletion occurred since the last collection. If no changes are detected, the processing is skipped entirely. Adaptive chunking – For organizations managing more than 20 repositories, the solution automatically partitions the workload into parallel chunks and processes them concurrently using AWS Step Functions Map states. Full and incremental loads – On first execution, the system performs a full load of all repository metadata. Subsequent runs use incremental logic, collecting only data that has changed. A full refresh is automatically triggered every 24 hours to maintain data accuracy. Configurable scheduling – How often metrics get collected is up to you. It’s an AWS CloudFormation parameter that accepts both rate() and cron() expressions, so you can set whatever schedule fits. The solution aligns with the AI-DLC framework’s observability pillar by providing continuous, automated measurement of development activity. You can establish velocity baselines before adopting AI tools, track changes during rollout, and quantify improvements over time. Architecture The architecture uses six core AWS services to build a fully managed, event-driven pipeline: Amazon EventBridge Scheduler: A scheduled rule starts the workflow at the interval you configured. It runs the collection cycle automatically without any external triggers or manual intervention. AWS Step Functions: A Step Functions state machine orchestrates the entire collection workflow. It first invokes the change detector to determine whether new activity has occurred. Based on the result, it decides between a full load and an incremental load. A full load collects all repository metadata from scratch on the first run and every 24 hours thereafter. An incremental load collects only data that changed since the last run, which reduces API calls and execution time. The workflow then evaluates the number of active repositories. If the count exceeds the chunking threshold (default: 20), it partitions the repositories into equal-sized chunks and processes them concurrently using a Step Functions Map state. Each chunk is handled by a separate AWS Lambda invocation. If the count is below the threshold, a single direct invocation collects all repositories. The workflow includes comprehensive retry logic with exponential backoff for transient API failures. AWS Lambda: Change detector – github-change-detector: This Lambda function queries the activity feeds to determine the load type. It monitors six categories of events: push events, pull requests, issue creation and updates, repository creation, repository deletion, and contributor changes. If no relevant events are found since the last check, the workflow terminates early. Metrics collector – github-metrics-collector: This Lambda function performs the actual data collection. It operates in three modes: collect_all for direct processing, collect_chunk for processing a subset of repositories during parallel execution, and aggregate for combining the chunk results. AWS Secrets Manager: Git tokens are stored securely in Secrets Manager. Both Lambda functions retrieve tokens at runtime, so that credentials are never hard coded or passed as environment variables. Amazon S3: Collected metrics are stored on Amazon S3 with versioning and server-side encryption enabled. The output includes a structured JSON file containing the full API response with nested repository details, and a flattened CSV file optimized for analytics. Both files include fields such as repository name, total commits, open and closed pull requests, open and closed issues, number of contributors, primary language, last activity timestamp, and repository creation date. S3 serves as the durable, cost-effective data store for all historical metrics. Amazon Quick Sight: Reads the collected CSV data from the S3 data store or through Amazon Athena for SQL-based queries. It loads the data into SPICE (Super-fast, Parallel, In-memory Calculation Engine), a fast in-memory layer, to build interactive dashboards. You can build visualizations for summary metrics, pull request (PR) trends, development activity over time, per-repository drilldowns, and contributor analysis. The following figure provides an overview of the solution architecture using the listed services. Figure 1: Architecture overview Before moving on to the step-by-step walkthrough, let us look at the prerequisites required for you to follow along. Prerequisites An active AWS account with permissions to create CloudFormation stacks, Lambda functions, S3 buckets, AWS Identity and Access Management (IAM) roles, Step Functions state machines, Amazon EventBridge rules, and Secrets Manager secrets. AWS Command Line Interface (AWS CLI) v2. A GitHub or GitLab account with repositories to monitor, and the ability to generate Personal Access Tokens. An active Amazon Quick Sight subscription on either Standard or Enterprise edition for dashboard creation. Least-privilege permissions for deploying the required resources: cloudformation:, s3:, lambda {CreateFunction,UpdateFunctionCode, UpdateFunctionConfiguration,GetFunction, PutFunctionConcurrency}, states:{CreateStateMachine,UpdateStateMachine,DescribeStateMachine}, events:{PutRule,PutTargets,DescribeRule}, sqs:{CreateQueue,GetQueueAttributes, SetQueueAttributes}, sns:{CreateTopic,GetTopicAttributes,SetTopicAttributes}, kms:{CreateKey, CreateAlias, PutKeyPolicy, EnableKeyRotation}, secretsmanager:{CreateSecret,GetSecretValue}, and iam:{CreateRole, PutRolePolicy, AttachRolePolicy, PassRole}. Clone the solution from this GitHub repository. Navigate to the cloned directory, subsequent steps reference file paths relative to this root. Walkthrough In this section, we elaborate the steps that you can follow to deploy a sample dashboard in your test environment. Step 1: Create personal access tokens Before deploying the infrastructure, you need API tokens for each Git platform you plan to monitor. For GitHub: Navigate to GitHub, Settings, Developer settings, Personal access tokens and choose Tokens (classic). Choose Generate new token (classic). Under scopes, select only the minimum required permissions: repo (read-only access to repository metadata, commits, and pull requests) and read:org (read-only access to organization membership). Copy the generated token and store it securely. For GitLab: Navigate to GitLab, Settings and Access Tokens. Create a new token with scopes and copy the generated token. The following image shows the GitHub developer settings page where you can generate the new tokens: Figure 2: GitHub developer settings page to generate new personal access tokens Step 2: Store tokens in AWS Secrets Manager Store each token as a secret in AWS Secrets Manager using the AWS CLI. See the following sample commands: # Store GitHub token aws secretsmanager create-secret \ --name git-dashboard/github-token \ --description "GitHub Personal Access Token for Git Dashboard" \ --secret-string "ghp_YOUR_TOKEN_HERE" \ --region us-east-1 # Store GitLab token aws secretsmanager create-secret \ --name git-dashboard/gitlab-token \ --description "GitLab Personal Access Token for Git Dashboard" \ --secret-string "glpat-YOUR_TOKEN_HERE" \ --region us-east-1 Step 3: Deploy the CloudFormation stack Download this sample CloudFormation template to follow along and deploy the solution in your non-production environment. Option 1: Deploy using the AWS CLI. Replace placeholder values enclosed in angle brackets () with your actual AWS account ID, Region, and the secret ARNs from Step 2: aws cloudformation create-stack \ --stack-name git-metrics-pipeline \ --template-body file://infrastructure/template.yaml \ --parameters \ ParameterKey=BucketName,ParameterValue=git-metrics-- \ ParameterKey=ScheduleExpression,ParameterValue="rate(10 minutes)" \ ParameterKey=ChunkingThreshold,ParameterValue=20 \ ParameterKey=GitHubTokenSecretArn,ParameterValue= \ ParameterKey=GitLabTokenSecretArn,ParameterValue= \ ParameterKey=EnabledPlatforms,ParameterValue="github,gitlab" \ --capabilities CAPABILITY_NAMED_IAM Option 2: Deploy using the AWS Management Console. Navigate to the AWS CloudFormation console. Choose Create stack, With new resources (standard). Upload the template, fill in parameters, and choose Submit. Stack reaches CREATE_COMPLETE status in 3-5 minutes. It creates the following resources: an S3 bucket with versioning and encryption enabled, two Lambda functions (Python 3.13 runtime), an IAM execution role with least-privilege permissions, a Step Functions state machine, an Amazon EventBridge schedule rule, and associated IAM roles. The following image shows the sample Create stack parameters and configuration: Figure 3: CloudFormation console showing the Create stack configuration page Step 4: Upload the Lambda deployment package The CloudFormation template deploys placeholder code for the Lambda functions. Update them with the actual logic using the pre-built deployment package from the cloned repository. The file lambda-package.zip is located in the deployment/ directory of the repository you cloned in the prerequisites. Replace in the following commands with the S3 bucket name created by your CloudFormation stack: # Upload package to S3 aws s3 cp deployment/lambda-package.zip s3:///lambda/lambda-package.zip # Update Detector Lambda aws lambda update-function-code \ --function-name github-change-detector \ --s3-bucket \ --s3-key lambda/lambda-package.zip # Update Collector Lambda aws lambda update-function-code \ --function-name github-metrics-collector \ --s3-bucket \ --s3-key lambda/lambda-package.zip Step 5: Execute and validate Trigger the workflow manually to verify end-to-end functionality. Replace with your 12-digit AWS account ID and with your deployment AWS Region (for example, us-east-1): aws stepfunctions start-execution \ --state-machine-arn arn:aws:states:::stateMachine:github-metrics-workflow \ --input '{}' Monitor the e [truncated for AI cost control]

展開要點與分析

文章情報

工程師中級

要點

  • AI 服務暫時不可用,系統已先保留來源內容與降級後設資料。
  • Learn how to build a fully serverless pipeline that automatically collects Git metrics from GitHub and GitLab and visualizes them in interactive Amazon Quick Sight dashboards, giv…

技術影響

可能影響 Agent 架構、工具呼叫、工作流自動化和產品整合。

要點與分析由自動化流程生成,可能有誤,請結合原始來源核實。