The road

Multi-Tenant Bedrock Agents Security with Cedar

Sat, 06 Jun 2026 00:00:00 +0000

TL;DR (30-Second Read)

With Amazon Bedrock AgentCore now generally available — including AgentCore Identity for agent authentication and AgentCore Policy, which enforces Cedar rules by intercepting every tool call before execution — the security design for multi-tenant SaaS on Bedrock Agents has reached an inflection point. This blueprint addresses the hardest problem in agentic SaaS: a Large Language Model (LLM) that, through prompt injection or hallucination, crosses tenant boundaries or escalates privileges. The answer is a zero-trust, low-latency, two-layer authorization architecture with real-time quota enforcement, built on Amazon Verified Permissions (AVP) and the Cedar policy engine.

S3 Vectors vs OpenSearch: Decision Tree from 30+ Projects

Wed, 27 May 2026 00:00:00 +0000

Choosing a vector store on AWS for generative AI (GenAI) workloads used to be a one-line decision: pick Amazon OpenSearch Service or its serverless variant (AOSS) and move on. That changed when Amazon S3 Vectors went GA in 2025. By storing vector data directly in S3 and pricing it on a fully consumption-based model, S3 Vectors has reset the cost-performance frontier for vector search.

This post is not a rehash of the official documentation. It distills selection and tuning experience across more than 30 production GenAI projects shipped over the past year. You will get a decision tree, the cost-crossover math between the two services, and the specific migration pitfalls that bite hardest in practice — including the cosine-distance range mismatch and the metadata structure constraints.

MCP OAuth on AgentCore Gateway + Cognito via APIGW Façade

Tue, 19 May 2026 00:00:00 +0000

Introduction

Amazon Bedrock AgentCore Gateway is the most pragmatic way to host a Model Context Protocol server on AWS today. Declare your tools as OpenAPI or as Lambda targets, get a managed multi-target MCP endpoint, and inherit AWS-native authentication via a customJwtAuthorizer. For machine-to-machine traffic that pattern is excellent.

The moment you ask an interactive MCP client — Claude Code, Cursor, the MCP Inspector — to talk to that same gateway with a per-user OAuth flow, the seams show. AgentCore Gateway expects a JWT and trusts whatever issuer you wired into its authorizer. Pair it with Amazon Cognito and the wiring works for the server side. It does not work for the client side, because Cognito is an OIDC identity provider, not an MCP-compliant authorization server. The two are not the same thing.

Claude Platform on AWS vs. Bedrock: A Decision Tree

Wed, 13 May 2026 00:00:00 +0000

"Use Bedrock" was a one-line answer six months ago. As of May 11, 2026, it's not. Anthropic and AWS shipped Claude Platform on AWS to general availability — Anthropic's native developer platform, accessed through your AWS account, billed through AWS Marketplace, and operated by Anthropic outside the AWS security boundary.

There are now three ways to run Claude when AWS is anywhere in the picture:

Amazon Bedrock — AWS operates inference; AWS is the data processor.
Claude Platform on AWS — Anthropic operates inference; Anthropic and AWS are independent data processors. Auth, billing, and audit go through AWS-native plumbing.
Claude Platform direct (claude.com) — Anthropic everything. AWS is not in the loop.

The choice has real architectural implications: feature availability, who holds your data, what your CloudTrail trail actually records, whether HIPAA workloads can run there, and whether your AWS EDP commit can absorb the spend. Pick wrong and you'll either re-platform when a beta you need only exists on one path, or re-architect when an auditor asks why model inference is leaving the AWS security boundary.

Agent Toolkit for AWS: What It Changes for Claude Code

Tue, 12 May 2026 00:00:00 +0000

If you've been using Claude Code for AWS development, you've probably seen the pattern: you paste a CloudFormation snippet into your session, Claude suggests something plausible, you deploy it, and the stack events stream lights up with CREATE_FAILED on a property the model couldn't have known about — because its training data stopped months ago.

The usual workaround has been hand-rolling context into CLAUDE.md: copying service endpoint quirks, IAM condition key syntax, and PrivateLink DNS formats that the model gets wrong. It works, but it's manual, fragile, and grows without bound.

Self-Hosted GitHub Runners on AWS Spot for AI Dev Teams

Fri, 08 May 2026 00:00:00 +0000

Autonomous AI dev teams move the bottleneck. When a dispatcher fans out work to dev and review agents every 5 minutes, the constraint is no longer human attention — it is the CI/CD pipeline that gates every PR. Each agent push triggers builds, tests, E2E verification, and bot reviews. With even a small team of agents iterating in parallel, GitHub Actions minutes become the dominant operational cost.

Hosted runners are convenient, but they are billed per-minute and capped at 2 cores on the standard tier. For a team of agents that pushes dozens of times an hour and runs container builds + Playwright suites, the math stops working. The lever is self-hosted runners on AWS spot EC2 — bigger instances, multi-architecture, scaling to zero when idle.

Track Claude Code Cost Per Project with Bedrock Tagging

Wed, 29 Apr 2026 00:00:00 +0000

If you run claude against Amazon Bedrock across a dozen repos, your bill arrives as one opaque number. Until recently, the workaround was clunky — create an application inference profile per project, swap them by hand, hope you remembered which one was active. In April 2026, AWS shipped native per-principal cost attribution for Bedrock: every InvokeModel call now carries the caller's IAM principal and session tags straight into Cost Explorer and CUR 2.0. That is enough to turn a ~60-line shell recipe into a per-repo billing system.

Transcribing Long Podcasts and Meetings with FunASR

Tue, 28 Apr 2026 00:00:00 +0000

Two recordings sat on my disk waiting to be turned into searchable text. A 4-hour 13-minute discussion from a TGO founders' group — eight speakers, Chinese, Zoom audio. A 1-hour 8-minute podcast episode (屠龙之术 Vol.94 × 知本论) where two hosts spent the whole hour dissecting OpenClaw — its positioning, the AI-agent narrative, and investors' reactions.

Both were full of specific claims, speaker-attributed opinions, and Chinese brand terms I wanted to grep later. Neither would transcribe cleanly with the tools I tried first.

From Solo AI Engineer to Autonomous Dev Team

Tue, 10 Mar 2026 00:00:00 +0000

In a previous post, the AI Digital Engineer pattern was introduced, featuring a single Claude Code agent guided by Skills and enforced by Hooks to execute a complete engineering workflow. This approach demonstrated effectiveness in delivering production-ready code with guaranteed quality gates.

However, a fundamental limitation emerged: one agent performing all tasks.

This limitation highlighted the need for a multi-agent system, where a team of AI agents, each with a distinct role, could collaborate autonomously to transform GitHub issues into merged pull requests without human intervention.

Serverless Multi-Tenant OpenHands on AWS with Fargate

Mon, 02 Mar 2026 00:00:00 +0000

In a previous post, I introduced an AWS CDK project for deploying OpenHands on EC2, featuring Cognito authentication and Aurora PostgreSQL. While this architecture successfully facilitated initial deployment, operating a shared AI coding platform for a team revealed three fundamental limitations:

Shared Resources: All sandbox containers executed on a single EC2 instance, leading to contention for CPU and memory.
Persistent Cost: The EC2 instance incurred approximately $375/month, regardless of platform utilization.
Lack of Tenant Isolation: Sandbox containers possessed network access to each other.

The v1.0.0 release of openhands-infra addresses these limitations by fully replacing EC2 with ECS Fargate. This revised architecture provides per-conversation isolation through dedicated Fargate tasks, eliminates idle costs by stopping tasks when inactive, and prevents cross-tenant communication via granular security groups.

This post details the architectural evolution and the AWS services that enable these capabilities.

AI Digital Engineer: End-to-End Delivery with Claude Code

Sat, 31 Jan 2026 00:00:00 +0000

What if an AI could operate like a senior software engineer - not just writing code, but following the complete engineering process from design through deployment? This post introduces the AI Digital Engineer pattern: a system that transforms Claude Code from an interactive assistant into an autonomous engineer capable of delivering production-ready software.

Deploying OpenHands AI Platform on AWS with CDK

Mon, 26 Jan 2026 00:00:00 +0000

OpenHands is an open-source AI-driven development platform that enables AI agents to write code, fix bugs, and execute complex development tasks autonomously. The default setup works well for local development, but what if you want to run it for a team or make it accessible from anywhere?

This post introduces an AWS CDK project that extends OpenHands beyond single-user local usage. It adds multi-user authentication, persistent storage, and automatic recovery capabilities—transforming OpenHands into a shared service that your team can access from anywhere.

The Rise of Cloud-Based AI Coding Agents

The AI coding landscape has evolved rapidly. While IDE-integrated tools like GitHub Copilot focus on code completion, a new category of autonomous AI coding agents has emerged—platforms that can independently plan, write, test, and deploy code with minimal human intervention.

Commercial cloud platforms now offer managed AI coding experiences:

Devin (Cognition AI): The first widely-known autonomous AI software engineer, handling complete tasks from planning to deployment
Claude Code: Anthropic's agentic coding tool that runs in your terminal, with Claude.ai offering Artifacts for interactive code generation
OpenAI Codex: OpenAI's cloud-based coding agent running in sandboxed environments with GitHub integration
v0 (Vercel): Cloud-based platform for generating full-stack applications with one-click deployment

These platforms provide convenience but come with trade-offs: vendor lock-in, data privacy concerns, limited customization, and recurring subscription costs.

Why self-host OpenHands?

Consideration	Cloud Platforms	Self-Hosted OpenHands
Data privacy	Data processed by third parties	Your data stays in your AWS account
Customization	Limited to vendor features	Full control over configuration
Cost model	Per-seat subscription	Infrastructure cost (scales with usage)
LLM choice	Vendor-selected models	Any model (Bedrock, OpenAI, local)
Integration	Vendor-provided integrations	Custom integrations via API

For organizations with strict compliance requirements, existing AWS infrastructure, or teams that prefer open-source solutions, self-hosting OpenHands provides the autonomous coding capabilities without the constraints of commercial platforms.

Secure AWS Credentials with credential_process

Fri, 02 Jan 2026 00:00:00 +0000

Managing AWS credentials securely is a fundamental challenge for developers. Storing plain text access keys in ~/.aws/credentials creates significant security risks, especially when backing up dotfiles to version control systems. This post introduces credential_process, a powerful AWS CLI feature that allows you to source credentials from external processes, enabling encrypted credential storage while maintaining seamless AWS access.

The Problem with Plain Text Credentials

The traditional approach stores AWS credentials in ~/.aws/credentials:

OIDC External Identity Source for AWS IAM Identity Center

Wed, 31 Dec 2025 00:00:00 +0000

AWS IAM Identity Center (formerly AWS SSO) provides centralized access management for AWS accounts and applications. While it natively supports SAML 2.0 for external identity providers, many organizations prefer OIDC-based authentication through providers like Amazon Cognito. This post demonstrates how to use Cloudflare Access as a SAML bridge between Amazon Cognito and AWS IAM Identity Center with automatic just-in-time (JIT) user provisioning.

Desktop Notifications for Claude Code: Never Miss a Completed Task

Tue, 02 Dec 2025 00:00:00 +0000

When working with Claude Code on complex tasks, you often switch to other work while waiting for completion. The challenge? Knowing exactly when Claude finishes so you can review the results promptly. This post shows you how to configure desktop notifications that alert you the moment Claude Code completes a task.

MCP OAuth Evolution: SEP-991 Simplifies Client Registration

Tue, 02 Dec 2025 00:00:00 +0000

The Problem with Dynamic Client Registration

In my previous deep-dive into MCP authorization, I analyzed how the protocol builds on OAuth 2.1 with mandatory PKCE, Resource Indicators (RFC 8707), and the "Discovery Trifecta" of RFC 7591, 8414, and 9728. Dynamic Client Registration (DCR) was positioned as the key enabler for MCP's federated ecosystem.

However, DCR has significant practical limitations:

Challenge	Impact
Requires AS support for public registration API	Many identity providers don't offer this
Forces OAuth proxy infrastructure	Added complexity when AS lacks DCR
Manual IT involvement	End users need admin help for each registration

The MCP ecosystem faces a unique challenge: unbounded clients connecting to unbounded servers with no prior relationship. DCR, while standardized, often requires workarounds in practice.

Implementing MCP OAuth 2.1 with Keycloak on AWS

Fri, 21 Nov 2025 00:00:00 +0000

Introduction

The Model Context Protocol (MCP) ecosystem mandates OAuth 2.1-compliant authorization servers to facilitate secure, federated access to AI model services. MCP clients, such as Claude Code, Cursor, and VS Code extensions, rely on modern OAuth specifications including Dynamic Client Registration (RFC 7591), PKCE (RFC 7636), and crucially, Resource Indicators (RFC 8707) for audience-restricted tokens.

However, most Identity-as-a-Service (IDaaS) providers, including the open-source Keycloak platform, currently lack full RFC 8707 support. Keycloak, while robust in OAuth 2.0 capabilities, employs a proprietary audience parameter in contrast to the standardized resource parameter defined in RFC 8707. For a comprehensive analysis of this compatibility landscape, refer to my previous post: Technical Deconstruction of MCP Authorization: A Deep Dive into OAuth 2.1 and IETF RFC Specifications.

Xiaozhi ESP32 MCP Gateway with Amazon Bedrock AgentCore

Mon, 17 Nov 2025 00:00:00 +0000

The Xiaozhi hardware is an impressive ESP32-based AI voice assistant capable of offline wake-up, multi-language support, and cloud connectivity. But what if you want your Xiaozhi device to access multiple AI tools, APIs, and services without managing complex integrations on the hardware side? This is where Amazon Bedrock AgentCore Gateway shines as a unified aggregation layer for Model Context Protocol (MCP) servers.

In this guide, I'll walk you through building a distributed MCP architecture that connects Xiaozhi hardware to multiple cloud services through a single WebSocket connection, leveraging AgentCore Gateway to aggregate tools ranging from simple calculators to complex RESTful APIs like real-time football data.

Beyond Prompts: 4 Context Engineering Secrets for Claude Code

Fri, 14 Nov 2025 00:00:00 +0000

You can stop hoping your Large Language Model (LLM) follows complex instructions. Context Engineering is the strategic practice of curating what enters the model's limited attention budget. To build reliable AI agents, you must master these four deterministic context patterns.

This post explores the key insights from my presentation "Claude Code Skill(s): Mastering AI-Powered Development", focusing on practical patterns that transform unpredictable AI interactions into deterministic, production-ready workflows.

What is Context Engineering?

Context Engineering is the art and science of strategically managing what information an LLM processes within its limited attention window. Unlike traditional prompt engineering, which focuses on crafting individual requests, context engineering treats the entire development environment as a programmable context system.

Technical Deconstruction of MCP Authorization: A Deep Dive into OAuth 2.1 and IETF RFC Specifications

Wed, 12 Nov 2025 00:00:00 +0000

Executive Summary

This article provides a deep-dive technical analysis of the Model Context Protocol (MCP) authorization flow. The central insight is that MCP's authorization model is not a generic application of OAuth 2.0 but a sophisticated implementation of the emerging OAuth 2.1 standard.

The MCP protocol deliberately rejects the flexible but less secure patterns of the original 2012 OAuth framework (RFC 6749). Instead, it adopts a modern, secure-by-default, and dynamic protocol stack built on three pillars:

How to Fix Shift+Enter in VS Code Remote SSH for Claude Code

Thu, 06 Nov 2025 00:00:00 +0000

The Problem: Premature Input Submission

When working with Claude Code in a VS Code Remote SSH session (e.g., connecting from a macOS or Windows client to a remote Linux host), a common frustration arises: pressing Shift+Enter in the terminal is supposed to create a new line for multi-line input. Instead, it often submits the current line prematurely.

This behavior disrupts the workflow for crafting complex, multi-line prompts, leading to fragmented interactions and inefficient token usage.

Amazon Quick Suite Deep Dive: Build AI-Powered Business Intelligence on AWS

Wed, 29 Oct 2025 00:00:00 +0000

Introduction

Business intelligence has long been the domain of specialists, requiring complex tools and time-consuming analysis. But what if you could simply ask your data questions in plain English and receive comprehensive, visualized answers in seconds? What if you could automate your weekly reporting with a simple conversation?

Amazon Quick Suite is AWS's answer. It's a generative AI-powered business intelligence platform designed to democratize data analysis, making it accessible to everyone in your organization—no machine learning expertise required.

Build on AWS Faster with Claude Code and AWS Skills

Sun, 26 Oct 2025 00:00:00 +0000

Introduction

Building on AWS is powerful but complex. What if your AI assistant had deep AWS expertise built-in? That's what AWS Skills brings to Claude Code.

AWS Skills is a plugin that transforms Claude Code into an intelligent AWS development partner. It understands CDK best practices, estimates costs before you deploy, and guides you through serverless patterns. This post shows you how to build a serverless REST API using Claude Code supercharged with AWS Skills.

Upgrade to Claude Agent SDK: A Quick Migration Guide from Claude Code

Tue, 30 Sep 2025 00:00:00 +0000

Heads up, builders! You may have noticed that Anthropic has rebranded the Claude Code SDK to the new Claude Agent SDK.

This is more than just a new name. As the official announcement explains, this change reflects a strategic focus on making it easier than ever to build, debug, and deploy powerful AI agents.

If you've been following our deep dive into building agentic applications, you'll be happy to know that the migration is incredibly straightforward. All the core concepts and powerful features like conversation management, MCP integration, and response streaming are still there.

Building Agentic Applications with Claude Code: A Developer's Guide to AI-Powered Automation

Mon, 29 Sep 2025 00:00:00 +0000

Introduction

If you've been following the AI space, you know that "agents" are the next big thing. These autonomous, intelligent systems promise to revolutionize how we automate complex tasks. But building them can be a daunting undertaking, often involving a tangled web of conversation management, tool integration, and state tracking.

What if I told you there's a framework that handles the heavy lifting, letting you focus on your agent's unique logic? Enter Claude Code. It has evolved beyond a simple CLI tool into a powerful AI application framework for building sophisticated AI applications.

Leveraging MCP Client's OAuthClientProvider for Seamless AWS AgentCore Authentication

Thu, 04 Sep 2025 00:00:00 +0000

Overview

Building on my previous exploration of connecting to MCP servers hosted on AWS AgentCore, I've been working extensively with the native MCP SDK's OAuth Client Provider to streamline authentication workflows. The MCP SDK's built-in OAuth support has evolved significantly, offering robust solutions for both interactive user authentication and machine-to-machine (M2M) flows.

In this follow-up article, I'll share the key improvements and special techniques I've discovered for using the MCP Client's OAuthClientProvider with AWS AgentCore, including handling AgentCore's unique behavior with 403 responses, implementing M2M authentication flows, and leveraging automatic token refresh capabilities.

How invoking remote MCP servers hosted on AWS AgentCore

Fri, 22 Aug 2025 00:00:00 +0000

Overview

Recently, I've been exploring AWS AgentCore's new capability to host Model Context Protocol (MCP) servers, and I wanted to share my experience with connecting to these remote servers as a client. The Model Context Protocol is an open standard that enables AI assistants to securely connect with external data sources and tools, and AWS AgentCore provides a managed hosting environment for these servers with built-in authentication and scaling capabilities.

In this article, I'll walk through the process of invoking MCP servers hosted on AWS AgentCore Runtime or proxied via AgentCore Gateway, covering different authentication methods, client implementation patterns, and practical considerations. What struck me most about this approach is how it bridges the gap between local development and enterprise-grade deployment while maintaining the flexibility that makes MCP so powerful.

Build Agentic Chatbot on AWS with Amazon Bedrock

Mon, 07 Apr 2025 00:00:00 +0000

Overview

In this article, I'll share my experience building an agentic chatbot on AWS using Amazon Bedrock, Amplify Gen2, and Amplify AI kit. This project, called Industry Assistant Portal, serves as an internal industry assistant that provides industry-specific AWS solutions guidance. The chatbot leverages Amazon Bedrock's powerful foundation models and knowledge base capabilities to deliver contextually relevant information about AWS industry solutions.

The journey of building this chatbot taught me valuable lessons about implementing agentic AI systems that can reason, plan, and execute complex tasks while maintaining context awareness. I'll cover the architecture, implementation details, challenges faced, and key learnings from this project.

2025 AI Developer Tools Benchmark: Comprehensive IDE & Assistant Comparison

Fri, 17 Jan 2025 00:00:00 +0000

Overview

This comprehensive benchmark evaluates the capabilities of 10 leading AI-powered developer tools and IDEs. The focus is on their ability to autonomously complete real-world programming tasks through natural language conversations, minimizing the need for manual coding. The evaluation excludes common features like code explanation, completion, unit testing, and documentation generation to focus on advanced AI capabilities.

Testing Period: Late December 2024 to Mid-January 2025
Tools Tested: 10 major AI development assistants
Tasks Evaluated: 3 real-world programming scenarios
Note: Several tools were tested across multiple versions during this period.

Create Amazing Images with Amazon Nova and Model Context Protocol

Tue, 31 Dec 2024 00:00:00 +0000

Ever wished you could conjure up the perfect images for your blog posts or articles without leaving your editor? Wouldn't it be amazing to generate professional-quality visuals with just a few keystrokes while you're in your creative flow?

Well, grab your virtual paintbrush because we're about to dive into how you can do exactly that using GenAI's image generation capabilities with Model Context Protocol (MCP) server!

Nine Essential Tips of AWS Amplify for Boosting Development Productivity

Tue, 24 Dec 2024 00:00:00 +0000

AWS Amplify is a powerful set of tools and services for developing, hosting, and managing serverless applications. With the recent launch of Amplify Gen 2¹², the platform has evolved significantly to enhance the developer experience. In this guide, we'll explore nine essential tips that will help you maximize your productivity with AWS Amplify, covering everything from authentication and infrastructure management to AI integration and deployment.

Understanding Amplify Gen 2

Before diving into the tips, let's understand what makes Amplify Gen 2 special. It introduces a code-first developer experience that enables building fullstack applications using TypeScript. Key benefits include:

AI 真能编程了吗？

Wed, 13 Nov 2024 00:00:00 +0000

如果你平时关注科技圈的动态，那么你一定在朋友圈或直播中看到过这样的吸睛话题：

编程小白用 AI 编程，人人都能成为编程高手

零基础也能跨界开发

编程小白的逆袭：借助 AI 打造 xxx

在自媒体的推波助澜下，AI 编程已成为一种潮流，甚至被包装成人人都可掌握的技能。当然，这背后也少不了培训公司借机贩卖焦虑的推动。

Deep Dive Clickstream Analytics Series: Data Pipeline Observability

Mon, 21 Oct 2024 00:00:00 +0000

In this post, we will explore the observability features of our clickstream solution. Observability is crucial for understanding the health of your data pipeline, identifying issues promptly, and ensuring optimal performance. We'll cover the monitoring, logging, and troubleshooting capabilities built into the solution.

Overview

The clickstream analytics solution incorporates several observability features to help users monitor and maintain their data pipelines:

Logging
Custom CloudWatch dashboards
Automated alerting
Pipeline health checks
Troubleshooting tools

Let's delve into each of these components.

Deep Dive Clickstream Analytics Series: Reporting

Sat, 19 Oct 2024 00:00:00 +0000

In this post, we will explore the reporting module of our clickstream solution. This module leverages Amazon QuickSight to provide powerful visualization and analysis capabilities for clickstream data, enabling users to gain valuable insights from their data.

Using Amazon Bedrock as a Custom OpenAI Server Alternative in Cursor

Fri, 11 Oct 2024 00:00:00 +0000

Cursor is a powerful AI-assisted programming Integrated Development Environment (IDE) that comes with built-in Large Language Model (LLM) capabilities to aid in coding. With the introduction of Amazon's Bedrock Claude 3.5 model, developers now have access to advanced alternatives. In this post, we'll explore how to set up a custom gateway for Amazon Bedrock with the Claude Sonnet 3.5 foundation model and use it in Cursor as a custom model provider. This approach allows us to harness the cutting-edge language model capabilities of Amazon Bedrock within the familiar Cursor environment, potentially offering enhanced performance and cost-effectiveness for AI-assisted coding tasks.

Access Bedrock Claude 3/3.5 Models with Alfred OpenAI ChatGPT Workflow

Thu, 10 Oct 2024 00:00:00 +0000

Many Alfred users enjoy the convenience of the OpenAI ChatGPT workflow for quick AI-powered assistance. However, with the introduction of Amazon's Bedrock Claude 3 and 3.5 models, some may want to leverage these powerful alternatives. In this post, we'll explore how to set up a custom gateway to access Bedrock Claude models instead of the OpenAI API for your Alfred OpenAI workflow.

Why Use Bedrock Claude?

Potentially lower costs compared to OpenAI's API
Access to the latest Claude models
Data privacy considerations (your data stays within AWS)
Integration with other AWS services

Setting Up the Bedrock Access Gateway

Bedrock Access Gateway provides OpenAI-compatible RESTful APIs for Amazon Bedrock. It supports streaming responses via server-sent events (SSE) and is compatible with various Bedrock model families, including Anthropic Claude 3 (Haiku/Sonnet/Opus), Claude 3.5 Sonnet, Meta Llama 3, Mistral, and more features.

Deep dive clickstream analytic series: Data Modeling

Sun, 22 Sep 2024 00:00:00 +0000

In this post, we will delve into the data modeling module of our clickstream solution. This module is an optional component that creates data models in the Amazon Redshift data warehouse and calculates reporting dimensions based on the event, session, and user factor tables generated in the data processing module.

Overview Architecture

Overview architecture for data modeling module

The overview architecture demonstrates how the solution orchestrates loading clickstream data into the Amazon Redshift data warehouse and triggers data modeling within Redshift.

Deep dive clickstream analytic series: Data Processing

Sat, 14 Sep 2024 00:00:00 +0000

In this post, we will delve into the data processing module of our clickstream solution. This module is an optional component that normalizes raw clickstream events by cleaning, transforming, and enriching them to fit the standard clickstream data schema defined in the solution. It's designed for flexibility, reliability, high performance, and cost-effectiveness.

Overview Architecture

Overview architecture for data process module

The data processing is designed for batch or micro-batch data processing to optimize performance and cost-effectiveness. It's primarily an Apache Spark application running on Amazon EMR Serverless to achieve a balance between high performance and cost. It offers the following capabilities:

Deep dive clickstream analytic series: Data Ingestion

Sun, 08 Sep 2024 00:00:00 +0000

In this post, we will delve into the data ingestion service of our clickstream solution. This service is a vital part of the clickstream analytics system. It is designed to be reliable, resilient, high-performing, flexible, and cost-effective. It plays a key role in capturing clickstream data from various sources and delivering it to downstream processing and modeling components.

Overview Architecture

Overview architecture for data ingestion service

Deep dive clickstream analytic series: Serverless web console

Wed, 04 Sep 2024 00:00:00 +0000

This post explores the web console module of the clickstream solution.

The web console allows users to create and manage projects with their data pipeline, which ingests, processes, analyzes, and visualizes clickstream data. In version 1.1, the Analytics Studio was introduced for business analysts, enabling them to view metrics dashboards, explore clickstream data, design customized dashboards, and manage metadata without requiring in-depth knowledge of data warehouses and SQL.

One code base for different architectures

The web console is a web application built using AWS serverless technologies, as demonstrated in the Build serverless web application with AWS Serverless series.

How to build a clickstream analytic system for small businesses to large-scale events

Tue, 03 Sep 2024 00:00:00 +0000

In the last couple of months, I led a team to build a comprehensive and open-sourced solution that helps customers analyze clickstream events on the cloud. The solution provides data autonomy, allowing users full access to raw data, near real-time ingestion, flexible configurations, and cost-effectiveness. It is a system that utilizes serverless services to cater to various customers, whether small businesses or large-scale events with massive data volumes, offering fully managed services with minimal operational efforts or the flexibility to use preferred open-source technical stacks.

Analyzing Clickstream Events Using Amazon Athena UDFs

Sat, 17 Aug 2024 00:00:00 +0000

In today's digital age, businesses are constantly seeking ways to understand and analyze user behavior on their websites. Clickstream events provide valuable insights into how users interact with a website, and analyzing this data can help businesses make informed decisions to improve user experience and drive conversions.

Clickstream Analytics on AWS collects, ingests, analyzes, and visualizes clickstream events from your websites and mobile applications. The solution manages an ingestion endpoint to receive clickstream events, which are multiple events in a batch sent by the solution‘s SDKs.

Scan Your Code with Ephemeral SonarQube in GitHub Actions

Sun, 12 May 2024 00:00:00 +0000

As developers, we all know the importance of maintaining high code quality standards. One powerful tool that can help us achieve this is SonarQube, a renowned platform for continuous code quality inspection. However, setting up and maintaining a dedicated SonarQube instance can be a cumbersome task, requiring significant resources and ongoing maintenance.

Fortunately, GitHub Actions offers a convenient solution by allowing us to spin up an ephemeral (short-lived) SonarQube instance directly within our workflow. This approach streamlines the process, eliminating the need for a permanent SonarQube server while still reaping the benefits of its code analysis capabilities.

Avoiding Pitfalls When Using Amazon DynamoDB Interface VPC Endpoints

Sat, 04 May 2024 00:00:00 +0000

Amazon DynamoDB now supports AWS PrivateLink as of March 19, 2024. This feature allows you to securely access DynamoDB from your Amazon Virtual Private Cloud (VPC) without exposing your traffic to the public internet.

However, unlike VPC endpoints for other AWS managed services, the AWS PrivateLink for Amazon DynamoDB does not support the Private DNS feature. This means that if your subnets are configured with only a DynamoDB Interface VPC endpoint, the public DNS name of the DynamoDB service (e.g., dynamodb.us-east-1.amazonaws.com in the us-east-1 region) cannot be resolved in those subnets.

Redshift Serverless: Cost Deep Dive and Use Cases

Sat, 24 Feb 2024 00:00:00 +0000

Serverless computing is all the rage, promising pay-as-you-go magic and freedom from infrastructure woes. But what about serverless for data warehouses? Let's delve into the fascinating (and sometimes confusing) world of Redshift Serverless: its cost structure, ideal use cases, and situations where it might not be the best fit.

Cost Breakdown: Beyond the Illusion of Free

Redshift Serverless offers a compelling promise: only pay for what you use. But like any good magic trick, there's more to the story. Here's the primary cost breakdown:

Custom compliance implementation in AWS CDK

Tue, 09 Jan 2024 00:00:00 +0000

AWS CDK accelerates cloud development using common programming languages to model your applications. I had a series of posts using CDK to demonstrate Building serverless web applications with AWS Serverless. Because CDK uses a programming language to model your application, you can encapsulate your library via Constructs, and then reuse it crossing the entire application.

Meanwhile, you can create your own constructs to encapsulate the compliance requirements to simplify the code. For example, in our solution, I used the construct SolutionFunction to force using the same Node.js version(18.x), architecture(ARM64), Lambda logging configuration(JSON log), environment variables for Powertools Logger and so on crossing all NodejsFunction. In addition, using Aspects and escape hatches to make sure the application meets the compliance requirements.

Awesome AWS CLI

Sun, 07 Jan 2024 00:00:00 +0000

Disclaimer: the cover image was generated by Amazon Bedrock's Titan Image Generator G1.

AWS CLI is a swiss knife for orchestrating the operations of AWS resources. Especially, the filter option could help your filter and transform the output then combine with other Linux commands together.

This post collects the CLI usages to resolve my AWS operation needs.

Delete the legacy versions of a service catalog product

AWS Service Catalog has default 100 versions per product. Below is a one line command to delete the legacy versions.

Build serverless web application with AWS Lambda web adapter

Thu, 26 Oct 2023 00:00:00 +0000

Disclaimer: the cover image was generated by StableDiffusionXL with prompts 'cover image, spring boot, flask framework running in aws lambda'.

When deploying and operating a web application on the cloud, you prefer to use your favorite programming language and web framework. Also, you want to benefit from Serverless technologies for stability, scalability, cost optimization, and operation excellence.

AWS Lambda Web Adapter is a tool that perfectly meets your expectations. It lifts and shifts the web application based on your preferred language and web framework, including FastAPI, Flask, Django, Express.js, Next.js, Spring Boot, Nginx, PHP, Rust, Golang Gin, Laravel, ASP.NET, and so on! You don't have to change any code to migrate your application to Lambda runtime. It also supports WebSocket and streaming features that work well with your LLM applications.

Verbose logging for AWS JS SDK v3

Sun, 10 Sep 2023 00:00:00 +0000

When programming with the AWS SDK, developers sometimes want to debug a specific HTTP request when invoking an SDK API. Due to the poor documentation of AWS JS SDK v3, it takes a lot of work to find a way to print the verbose logging of AWS SDK by asking it to the LLMs.

Below is a practical tip for enabling verbose logging for AWS JS SDK v3.

Solution 1 - specify a custom logger for AWS SDK clients

 1import { DescribeParametersCommand, SSMClient } from "@aws-sdk/client-ssm";
 2import * as log4js from "log4js";
 3
 4log4js.configure({
 5 appenders: { out: { type: "stdout" } },
 6 categories: { default: { appenders: ["out"], level: "debug" } },
 7});
 8
 9const logger = log4js.getLogger();
10
11const ssmClient = new SSMClient({
12 logger: logger,
13});

Solution 2 - use middleware to hook the life cyele of request

 1import { DescribeParametersCommand, SSMClient } from "@aws-sdk/client-ssm";
 2
 3const logRequestMiddleware = (next: any, _context: any) => async (args: any) => {
 4 console.log('Request:', args.request);
 5 return next(args);
 6};
 7
 8const ssmClient = new SSMClient({
 9});
10
11ssmClient.middlewareStack.add(logRequestMiddleware, { step: 'finalizeRequest' });

See complete working example gist below,

Define your API via OpenAPI definition on AWS

Thu, 27 Oct 2022 00:00:00 +0000

Application Programming Interfaces(APIs) is a critical part of the web service, Werner Vogel, the CTO of AWS had a great 6 Rules for Good API Design presentation in 2021 re:Invent keynote.

In AWS the developers could manage and proxy the APIs via Amazon API Gateway. The developers can use console, CLI, API or IaC code(for example, Terraform/CloudFormation/CDK) to provisioning the API resources on AWS. However some developers might flavor with using OpenAPI specification to define the APIs. It enables multiple services/tools to understand the APIs' specification, such as Postman. Amazon API Gateway supports this use case, you can import the existing OpenAPI definition as API.

Setup DevOps pipeline with few code

Wed, 14 Sep 2022 00:00:00 +0000

DevOps pipeline is a key component of project operation, it helps you automate steps in your software delivery process.

Amazon itself has rich expirence on DevOps with large scale services, it shares the lesson and learn from operating the Amazon's services. You can read this summary post written in Chinese.

Also AWS provides fully managed SaaS services for the lifecycle of software development, including AWS CodePipeline for automating continuous delivery pipelines, AWS CodeCommit for securely hosting highly scalable private Git repositories, AWS CodeArtifact for artifact management, AWS CodeBuild for building and testing code with continuous scaling and AWS CodeDeploy for automating code deployments to maintain application uptime.

Federated OIDC login with Cognito and Amplify

Mon, 12 Sep 2022 00:00:00 +0000

When working on either 2C application or 2B service, the customers do not want to or is not allowed to sign up the new account, they can login the application via existing IdP or enterprise SSO. So, building the application supports the federated OIDC login to address such requirements.

This post extends the capability of Todolist application protected by Amazon Cognito, using Auth0 as the third party OpenID Connect provider introduces the external user pool.

Protect website with Cognito

Sun, 04 Sep 2022 00:00:00 +0000

Previous post we demonstrated how distributing and securely deploying the website to global end users. The authentication and authorization are always mandatory features of web application. Amazon Cognito is a managed AWS serverless service helping the applications to implement AuthN and AuthZ, with Cognito the applications securely scales to millions of users(up to 50,000 free users) supporting identity and access management standards, such as OAuth 2.0, SAML 2.0, and OpenID Connect.

Distribute the website globally

Fri, 02 Sep 2022 00:00:00 +0000

It's a well known pattern to distribute the website via CDN globally, it reduces the latency of the site and improve the availibity and security leveraging the infrastructure of cloud provider.

Using CDN service CloudFront and simple storage S3 on AWS hosts the static website. It well fits the SPA(single page application) framework technologies, for example, React, Vue and Angularjs. There are lots of existing project and code snippets to sharing this pattern, such as CloudFront to S3 and API Gateway and AWS S3 / React Website Pattern.

Build no code restful HTTP API with API Gateway and DynamoDB

Sat, 27 Aug 2022 00:00:00 +0000

Most web applications are using Restful APIs to interactive with the backend services. In the TODO application, it's the straight forward to get, update and delete the items from backend database. Amazon DynamoDB is a key-value database, it fits for this scenario with scalability and optimized pay-as-you-go cost. Also Amazon API Gateway has built-in integration with AWS serivces, the restful API can be transformed to the request to DynamoDB APIs. Using this combination you can provide the restful APIs only provisioning AWS resources without writing the CRUD code!

Build serverless web application with AWS Serverless

Fri, 26 Aug 2022 00:00:00 +0000

Building web application is a common use case, leveraging cloud services could accelerate the builders to develop and deploy the services. With AWS serverless services, the application can easily get the capabilities like security, highly availability, scalability, resiliency and cost optimized.

FluxCD GitOps debugging tip

Thu, 16 Jun 2022 00:00:00 +0000

After enabling E2E testing of FluxCD powered GitOps continuous deployment, the feedback of new commits are quite slow. Because you have to wait for the E2E testing result, lots of time cost on setuping the environment and provisioning your development from scrath.

Inspired by E2E testing in Github actions, the DevOps engineers can build local debugging environment in Kind or minikube.

使用外部Secrets Manager管理Kubernetes密钥

Sun, 12 Jun 2022 00:00:00 +0000

背景

密钥的管理对于使用 GitOps 方式做持续发布是一个挑战，特别是当目标部署平台是 Kubernetes 的时候。 K8S 使用声明式配置管理最终状态，而K8S中的密钥仅仅是将密钥内容做了base64格式的编码。在基于 Flux 的 GitOps 实战介绍了使用Bitnami Sealed Secrets加密密钥内容，可以安全的将加密后的Kubernetes Manifest文件提交到Git代码仓库，由Sealed Secrets发现这些SealedSecret的密码，并解密后动态的创建K8S原生Secrets对象。

基于 Flux 的 GitOps 管理 Crossplane 部署及资源

Wed, 01 Jun 2022 00:00:00 +0000

背景

在Flux 部署实战的总结展望中有一个方向是如何将云上基础设施资源同Kubernetes内资源统一管理，而Crossplane提供了一个高度可扩展的后端，使用声明式程序同时编排应用程序和基础设施，不用关心它们在哪里运行。

近期 AWS 官方博客宣布了 AWS Blueprints for Crossplane，为客户提供了在 Amazon EKS 上应用 Crossplane 的参考实现。

Publish your AWS CDK applications via AWS CloudFormation templates

Sun, 15 May 2022 00:00:00 +0000

AWS CDK is a great abstract to accelerate managing the cloud infrastructure as code. The journey will be enjoyful with leveraging the Construct Hub to use the high level contributions from AWS partners and commnunity.

Use Case

AWS CloudFormation is one of the underly technologies of AWS CDK to manage the cloud infrastructure. It easily to enable the IT administrators even business operators whom has no/limited developer skills to develop the end-to-end solutions with one-click user experience.

So it's a use case for effectively developing the Cloud Application via AWS CDK, then publishing it as CloudFormation template with better user experimental experience.

基于 Flux 的 GitOps 实战（下）

Sun, 08 May 2022 00:00:00 +0000

在上篇介绍基于 CNCF 下的 GitOps 工具 FluxCD v2 实现了管理多账户的 Kubernetes 集群的共享组件，Secrets 使用的最佳实践， GitOps 流水线事件同 IM(Slack) 的集成，以及对 GitOps 代码的 CI 流程。

本文将围绕如何使用 Flux 的多租户管理最佳实践，打造基于 GitOps 工作流程的共享服务平台，实现租户(业务/应用团队)可自助的持续部署。

基于 Flux 的 GitOps 实战（上）

Fri, 22 Apr 2022 00:00:00 +0000

在前文介绍了 GitOps 的概念，Kubernetes 上 GitOps 最佳实践以及对比了 CNCF 基金会下云原生的 GitOps 工具（ArgoCD 和 Flux）。本篇将带你按照 Flux 的最佳实践在跨VPC跨账户的 Kubernetes 上实践 GitOps 的持续集成，轻松管理数十数百乃至更多的集群及部署在上面的应用。

Kuberentes 上 GitOps 最佳实践

Wed, 30 Mar 2022 00:00:00 +0000

今天 Kuberentes 已经成为IT基础设施的重要玩家，容器编排领域的事实标准。写于3年前的文章不要自建 Kuberentes 的观点已经被绝大多数的企业所认可和接受。

然而同众多企业接触中发现，企业有很高的意愿采用 Kuberentes 管理工作负载，并且已有大量的企业已经将 Kuberentes 用于生产环境。但如何对多套不同阶段的 Kuberentes 集群来做持续部署，做到高安全性、权限分离、可审计、保证业务团队的敏捷等需求感到困惑。目前客户实现方式非常多样，并没有很好的遵循业界的最佳实践。

Find out the most costly resources in your AWS account

Sun, 20 Feb 2022 00:00:00 +0000

As a builder in cloud, you might feel confused about which resources cost mostly in your account.

In AWS, you can quickly find out which services even functionality cost a lot via AWS Billing or AWS Cost Explorer. However sometimes it sucks on finding out which functions cost mostly if you have hundreds of Lambda functions, or which metrics/log groups cost mostly in Amazon CloudWatch.

Grant federated users accessing kubernetes resources in EKS console

Wed, 09 Feb 2022 00:00:00 +0000

Though you're administrator of your AWS account, you probably see below warnings when viewing your cluster in EKS console.

Your current user or role does not have access to Kubernetes objects on this EKS cluster.

Publishing npm packages to multiple registries with Projen

Fri, 04 Feb 2022 00:00:00 +0000

Construct Hub is a web portal to collect the constructs for AWS CDK, CDK8s and CDKtf. The construct could support multiple programming languages, such as Javascript/TypeScript, Python, Java and C#. Actually the construct is developed by TypeScript, then it's compiled as across languages library by jsii! Any npm/pypi package with certain tags will be discovered by Construct Hub, the package will be automatically recognized as construct package and listed in Construct Hub.

Projen is a project generator to create project with simplifying the project configuration to support dependencies management, building, unit testing, code style linting, CI/CD via Github actions PR and actions. So projen supports the construct project out of box, which configures construct project with jsii configuration that build the construct to across languages library, though publish the packages to kinds of package registries, such as npmjs, pypi and maven central.

AWS上构建共享自服务平台服务去中心化研发团队

Sun, 26 Dec 2021 00:00:00 +0000

近期在一个 Webinar 分享了如何在 AWS 上服务去中心化研发团队构建共享服务平台，核心观点总结如下，

应用程序弹性设计

Sun, 28 Nov 2021 00:00:00 +0000

AWS架构的完善(AWS Well-Architected)框架涉及了五大支柱，其中可靠性支柱要求侧重于确保工作负载在预期的时间内正确、一致地执行其预期功能。这要求应用程序系统具备弹性设计，可从故障中快速恢复，以便满足业务和客户需求。然而设计、开发、且验证具备弹性设计的应用程序，对经验和实践能力都有很高的要求。利用成熟的经验和良好的工具将加快构建符合预期的弹性应用程序。

元宇宙风口下的机会

Mon, 22 Nov 2021 00:00:00 +0000

元宇宙是近期的热点话题，近期做了些学习了解，将一些学习内容总结为一个deck。分享如下，

AWS上的混沌工程

Sun, 21 Nov 2021 00:00:00 +0000

混沌工程是一种帮助系统满足弹性需求的技术，它起源于Netflix的工程实践，著名的猴子军团。

AWS一直提倡架构的完善(AWS Well-Architected)，混沌工程正是卓越运营和可靠性支柱的实践。因此在 re:Invent 2020 AWS发布了Fault Injection Simulator服务来简化开发者在AWS上的混动工程实践。

Turn off Filevault on macOS

Sun, 31 Oct 2021 00:00:00 +0000

I'm trying to upgrade my Macbook Pro to macOS Monterey, however the installation can not be started due to the disk is encrypted by Filevault 😕 I have to turn off Filevault to disable disk encrpytion before installing macOS Monterey.

I found this support article on how turning off Filevault, but it does not work at all. There is nothing hint or error message after clicking the option Turn off Filevault.

Mirror Helm Charts to AWS ECR

Mon, 27 Sep 2021 00:00:00 +0000

I met a case to mirror existing Helm charts to another repository. It might be caused by network availability or compliance requirements.

There are multiple ways to host a Helm repository, for example, Nexus OSS Repository, Github Pages, AWS ECR and so on.

Amazon Elastic Container Registry (Amazon ECR) is a fully managed container registry that makes it easy to store, manage, share, and deploy your container images and artifacts anywhere. It's built with scale and secure. In my case I'm using this existing service to mirror the Helm charts.

The practise of Amazon Neptune

Fri, 03 Sep 2021 00:00:00 +0000

Amazon Neptune is a managed Graph database on AWS, whose compute and storage is decoupled like Amazon Aurora. Neptune leverages popular open-source APIs such as Gremlin and SPARQL, and easily migrate existing applications.

The update of Sonatype Nexus repository OSS on AWS solution

Thu, 24 Jun 2021 00:00:00 +0000

Last year I shared the production-ready, cloud native solution to deploy Sonatype Nexus Repository OSS on AWS.

在AWS上快速部署专用的NAT实例

Fri, 16 Apr 2021 00:00:00 +0000

本方案的起因是，一个源代码托管在Github上的项目fix一个重要的bug后，在AWS上的持续部署流水线一直失败。分析日志后，发现流水线中的数个步骤需要克隆源代码，但是访问Github的网络非常不稳定，这数个流水线任务持续因连接超时，连接拒绝等网络错误而失败。而流水线任务大量使用了CodeBuild, Lambda等AWS托管服务，无法为执行环境配置可靠的网络连接。

Effective AWS CDK for AWS CloudFormation

Wed, 16 Dec 2020 00:00:00 +0000

Infrastructure as Code is the trend to manage the resources of application. AWS CloudFormation is the managed service offering the IaC capability on AWS since 2011. CloudFormation uses the declarative language to manage your AWS resources with the style what you get is what you declare.

However there are cons of CloudFormation as a declarative language,

the readability and maintenance for applications involving lots of resources
the reuseable of code, CloudFormation modules released in re:Invent 2020 might help mitigate it

AWS CDK provides the programming way to define the infra in code by your preferred programming languages, such as Typescript, Javascript, Python, Java and C#. AWS CDK will synthesis the code to CloudFormation template, then deploying the stack via AWS CloudFormation service. It benefits the Devops engineers manage the infra on AWS as programming application, having version control, code review, unit testing, integration testing and CI/CD pipelines, the deployment still depends on the mature CloudFormation service to rolling update the resources and rollback when failing.

For solution development, using CDK indeed improves the productivity then publish the deployment assets as CloudFormation templates.

Though CDK application can be synthesized to CloudFormation template, there are still some differences blocking the synthesized templates to be deployed across multiple AWS regions.

This post will share the tips on how effectively writing AWS CDK application then deploying the application by CloudFormation across multiple regions.

亚马逊的部署最佳实践

Sat, 28 Nov 2020 00:00:00 +0000

近期Amazon Builders Library发布了数篇文章介绍亚马逊如何实践持续部署，同时分享了亚马逊在部署方面的最佳实践。

这里将这三篇文章核心内容做个概述，方便大家按需细读。

跨账号跨区域部署AWS CDK编排的应用

Wed, 14 Oct 2020 00:00:00 +0000

AWS CDK是编排部署AWS云上资源最佳的工具之一。基于AWS CDK的应用应该如何实践DevOps持续集成和部署呢？

通常我们有下面几种方法，

Deploy Sonatype Nexus repository OSS on EKS

Tue, 16 Jun 2020 00:00:00 +0000

Sonatype Nexus repository OSS is an artifact repository that supports most software repositories such as Maven, Pypi, Npmjs, Rubygems, Yum, Apt, Docker registry and etc. In the enterprise Nexus repository is widely used for storing proprietary artifacts and caching the artifacts for speeding up the devops.

无服务器架构的Docker镜像数据分析应用

Mon, 04 May 2020 00:00:00 +0000

近期对Docker镜像做了些数据分析，这里分享一下利用云原生技术快速且低成本的实现任意数量的数据分析。

之前通过文章介绍了不用拉取Docker镜像就可获取镜像的大小的一种方法，通过其中的示例脚本，我们可以获取到待分析的原始数据。

比如nginx镜像的部分原始数据(csv格式)如下，

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


1.18.0-alpine,sha256:676b8117782d9e8c20af8e1b19356f64acc76c981f3a65c66e33a9874877892a,amd64,linux,null,null,"sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08",2813316
1.18.0-alpine,sha256:676b8117782d9e8c20af8e1b19356f64acc76c981f3a65c66e33a9874877892a,amd64,linux,null,null,"sha256:6ade829cd166df9b2331da48e3e60342aef9f95e1e45cde8d20e6b01be7e6d9a",6477096
1.18.0-alpine,sha256:70feed62d5204358ed500463c0953dce6c269a0ebeef147a107422a2c78799a9,arm,linux,v6,null,"sha256:b9e3228833e92f0688e0f87234e75965e62e47cfbb9ca8cc5fa19c2e7cd13f80",2619936
1.18.0-alpine,sha256:70feed62d5204358ed500463c0953dce6c269a0ebeef147a107422a2c78799a9,arm,linux,v6,null,"sha256:a03f81873d278ad248976b107883f0452d33c6f907ebcdd832a6041f1d33118a",6080562
1.18.0-alpine,sha256:2ba714ccbdc4c2a7b5a5673ebbc8f28e159cf2687a664d540dcb91d325934f32,arm64,linux,v8,null,"sha256:29e5d40040c18c692ed73df24511071725b74956ca1a61fe6056a651d86a13bd",2724424
1.18.0-alpine,sha256:2ba714ccbdc4c2a7b5a5673ebbc8f28e159cf2687a664d540dcb91d325934f32,arm64,linux,v8,null,"sha256:806787fcd4f9e2f814506fb53e81b6fb33f9eea04e5b537b31fa5fb601a497ee",6423816
1.18.0-alpine,sha256:6d6f19360150548bbb568ecd3e1affabbdce0fcc39156e70fbae8a0aa656541a,386,linux,null,null,"sha256:2826c1e79865da7e0da0a993a2a38db61c3911e05b5df617439a86d4deac90fb",2808418
1.18.0-alpine,sha256:6d6f19360150548bbb568ecd3e1affabbdce0fcc39156e70fbae8a0aa656541a,386,linux,null,null,"sha256:f2ab0e3b0ff04d1695df322540631708c42b0a68925788de2290c9497e44fef3",6845295
1.18.0-alpine,sha256:c0684c6ee14c7383e4ef1d458edf3535cd62b432eeba6b03ddf0d880633207da,ppc64le,linux,null,null,"sha256:9a8fdc5b698322331ee7eba7dd6f66f3a4e956554db22dd1e834d519415b4f8e",2821843
1.18.0-alpine,sha256:c0684c6ee14c7383e4ef1d458edf3535cd62b432eeba6b03ddf0d880633207da,ppc64le,linux,null,null,"sha256:30a37aac8b54a38e14e378f5122186373cf233951783587517243e342728a828",6746511
1.18.0-alpine,sha256:714439fec7e1f55c29b57552213e45c96bbfeefddea2b3b30d7568591966c914,s390x,linux,null,null,"sha256:7184c046fdf17da4c16ca482e5ede36e1f2d41ac8cea9c036e488fd149d6e8e7",2582859
1.18.0-alpine,sha256:714439fec7e1f55c29b57552213e45c96bbfeefddea2b3b30d7568591966c914,s390x,linux,null,null,"sha256:214dff8a034aad01facf6cf63613ed78e9d23d9a6345f1dee2ad871d6f94b689",6569410

各列的含义分别是，镜像tag, 镜像Digest, 镜像对应平台的Architecture, 镜像对应平台的OS, 镜像对应平台的变种（例如，ARM的v7, v8等）, 镜像对应平台的OS版本, 镜像组成层的Digest, 镜像组成层的大小。

上面nginx镜像的示例数据，告诉我们镜像名nginx且tag为1.18.0-alpine的镜像包含了amd64-linux, arm-linux-v6, arm64-linux-v8, 386-linux, ppc64le-linux以及s390x-linux共5种Arch合计6个版本的镜像。且每个平台的对应镜像包含了两个层以及这两个层的大小。

当我们有了成百数千甚至海量镜像的原始数据后，如何能快速且低成本的分析这些数据呢？

Get the size of Docker image without pulling image

Sat, 02 May 2020 00:00:00 +0000

Recently I had a requirement to stats the size of some Docker images. It would be waste if pulling them all firstly then calculating the size of each image. Also you know the docker image consists of some Docker layers that probably are shared by other images. It's hard to get the disk usage if only sum the size of each image.

Is there any way to get the size of Docker image without pulling it?

oh-my-zsh性能调优思路

Wed, 22 Apr 2020 00:00:00 +0000

Z shell搭配oh-my-zsh自定义配置已成为众多Linux/Macosx用户的标准terminal配置。

最近遇到在zsh中执行任意命令都变得特别慢(哪怕简单执行ls也要花费肉眼可见的1，2秒钟)，这里记录下如何排查Z shell下启用oh-my-zsh的性能问题。

基于CodeCommit代码管理的无服务器架构Devops

Thu, 26 Mar 2020 00:00:00 +0000

Github/Gitlab已经成为众多开发者非常熟悉的代码协作平台，通过他们参与开源项目或实施企业内部项目协作。

AWS也提供了托管的、基于Git、安全且高可用的代码服务CodeCommit。CodeCommit主要针对企业用户场景，所以他并没有社交功能以及代码仓库fork功能，是否CodeCommit就无法实现Github基于Pull Request的协同工作模式呢？

AWS发布更快、更便宜、更易用的HTTP APIs

Fri, 13 Mar 2020 00:00:00 +0000

AWS在3月12日正式发布了新一代的API网关 -- HTTP APIs。AWS发布的第一代API Gateway服务已经快5年了，通过这些年来大规模服务客户的心得以及客户反馈，由此重新构建了更快（相比第一代网关60%的延迟减少）、更便宜（至少节省71%的费用）、更易用的第二代网关服务。

AWS Cloud Debugging初探

Thu, 26 Dec 2019 00:00:00 +0000

在re:Invent 2019之前，AWS Toolkit发布了Cloud Debugging beta功能。该功能支持在IntelliJ IDEs(IntelliJ, PyCharm, Webstorm, 以及 Rider)中远程调试 ECS Fargate 容器中执行的应用程序。

AWS Batch简介

Wed, 25 Dec 2019 00:00:00 +0000

AWS Batch是一个全托管的批处理调度服务，它可为用户管理所有基础设施，从而避免了预置、管理、监控和扩展批处理计算作业所带来的复杂性。当然AWS Batch已与 AWS 平台原生集成，让用户能够利用 AWS 的扩展、联网和访问管理功能。让用户轻松运行能够安全地从 AWS 数据存储（如 Amazon S3 和 Amazon DynamoDB）中检索数据并向其中写入数据的作业。AWS Batch可根据所提交的批处理作业的数量和资源要求预置计算资源并优化作业分配。能够将计算资源动态扩展至运行批处理作业所需的任何数量，从而不必受固定容量集群的限制。AWS Batch还可以利用 Spot 实例，从而进一步降低运行批处理作业产生的费用。

AWS Batch服务本身是免费的，仅收取实际使用的 EC2 实例费用。

免费邮件转发服务

Wed, 18 Dec 2019 00:00:00 +0000

在拥有域名后，通常希望创建一些自有域名下的邮箱来收取不同用途的邮件，同时不希望为这部分功能付费😃。使用免费的企业邮箱(比如网易企业邮箱、阿里云企业邮箱)是一种选择。这时就需要配置邮件地址和邮件客户端来收取邮件，如果有多个邮箱地址，配置会特别麻烦。有时，这些企业邮箱的收件服务会莫名其妙的丢失一些邮件。

实战Aliyun EDAS应用迁移AWS

Mon, 02 Dec 2019 00:00:00 +0000

近期实践了将阿里云EDAS微服务应用迁移到AWS上，在这里分享一下迁移方案。

该方案涉及了以下三个方面，

微服务应用集群。在AWS上采用的ECS集群部署微服务应用，通过Cloudmap实现服务注册发现，App Mesh实现服务间流量控制。更加详尽的微服务迁移要点和对应方案，详见下面的deck。
Devops pipeline。使用托管的CodePipeline，CodeBuild实现CI/CD。
Infra as Code。利用AWS强大的Infra as Code能力，将云上的基础设施和微服务应用编排通过CDK代码实现。

下面是迁移方案的deck。完整且可部署的PoC代码，点这里。

Read More

AWS RDS数据库日志分析及展示

Tue, 05 Nov 2019 00:00:00 +0000

托管的RDS数据库已经是云计算服务中非常成熟的服务，绝大部分的云计算用户会采用RDS服务来提升数据库服务的可用性同时减少数据库的各类运维事务。

AWS RDS服务支持开启和查询各类的数据库日志，包括常规日志、慢日志、错误日志和审计日志。但RDS服务默认提供的日志查看工具仅仅类似文本查看器，无法针对日志数据做统计和查看历史滚动的存档。

使用Openswan连接AWS VPC

Mon, 16 Sep 2019 00:00:00 +0000

业务上云之后，经常也有需求将多云、数据中心或办公室的私有网络同云端的私有网络建立连接。AWS Site-to-Site VPN正是AWS提供的托管VPN服务，我们可以在另一端的私有网络通过Openswan同AWS VPC网络建立基于IPSec协议的安全连接。

AWS CDK简介

Sun, 08 Sep 2019 00:00:00 +0000

Infrastructure as Code(架构即代码)一直是衡量公有云是否支持良好运维能力的重要指标。作为云计算领先的AWS，通过服务CloudFormation来编排云环境中的基础设施资源。不过由于CloudFormation是使用YAML/JSON编写的声明式语言，不善于处理逻辑，编写繁琐且不利于调试排错，对于新上手的Devops工程师来说也有不小的学习曲线。三方开源的工具Terraform同样没有很好解决CloudFormation存在的这些问题。

无服务器架构的域名重定向服务

Tue, 27 Aug 2019 00:00:00 +0000

业务时常有需求将某个域名(A)的访问重定向到其他域名(B)，即使实现这样一个很简单的需求通常也需要部署Web服务器（例如Nginx），为域名A的请求返回302响应，并提供新的Location地址重定向到域名B。现在基于云计算服务，我们可以使用一些托管服务来实现同样的事情，无需管理服务器和维护应用，同时做到最低成本实现该需求。

接下来将介绍如何利用AWS上的服务实现该需求。

Amazon Alexa Android版本国内登录问题

Wed, 14 Aug 2019 00:00:00 +0000

近期需要做一些Alexa上的开发，在手机上安装了Amazon Alexa，一直得到下面这样的错误提示而无法登录。

Connection Timed Out.

使用AWS S3作为MacOSX时间机器(Time Machine)的备份存储

Sun, 30 Jun 2019 00:00:00 +0000

个人电脑数据备份一直都是一个强烈的需求。使用网盘等云存储产品可以部分满足数据的备份需求，仍然无法做到使用便利性和很高的数据安全保障。

MacOSX上系统内置了备份解决方案 -- 时间机器(Time Machine)。Time Machine支持AirPort Time Capsule，NAS存储或者外置的存储设备。然而这些备份方案都依赖于硬件设备，有容量限制或不便于移动。在云计算已经大行其道的今天，有没有使用云计算厂商对象存储作为目标存储的备份方案，为MacOSX数据备份提供无限容量、高度的安全性的云方案？经过一番搜索，既找到了开源免费的工具Restic，也有付费软件Arq。无论Restic还是Arq提供的是独立的三方工具来实现备份到云端存储或从云端恢复，有没有将Time Machine和云端储存结合在一起的方案呢？

Spring Cloud Function -- 跨Serverless平台的函数计算框架

Fri, 28 Jun 2019 00:00:00 +0000

基于serverless框架的钉钉回调函数中介绍了serverless framework，一款支持跨云厂商/Serverless平台的部署工具。但是函数代码还是需要针对不同的serverless平台作对应的适配。而Spring Clound Function就是针对这种情况专门开发的跨serverless平台的框架，实现一套代码通过不同的打包实现跨serverless平台。Spring Clound Function目前支持AWS Lambda, Microsoft Azure Function以及Apache OpenWhisk。

公有云对比

Wed, 26 Jun 2019 00:00:00 +0000

AWS是全球云计算领域的领跑者，它在计算、存储、网络等方面都做出了很多创新，同时也是其他云计算厂商学习及模仿的对象。

阿里云是目前国内市场份额最大的云计算厂商，其份额超过了第二至五位厂商的总和，份额领先优势比AWS在全球还要显著，同时全球份额也超过IBM来到第四。

本文将对AWS和阿里云核心服务做一个简要对比，以及这两家厂商发展方向的一些个人见解。

Serverless framework 101

Thu, 16 May 2019 00:00:00 +0000

Serverless Framework是一个开源命令行工具。他提供函数脚手架、流程自动化、最佳实践等帮助开发、部署跨云厂商的托管无服务器计算服务(官方已支持aws, Azure, GCP, IBM Cloud等各种厂商的无服务器计算)。同时支持使用插件来扩展各种功能，比如支持更多云厂商无服务器计算服务，例如阿里云的函数计算。

这里使用基于函数计算的钉钉回调函数接口示例来演示如何使用Serverless Framework将一个无服务器函数部署到AWS Lambda。

AWS Lambda Layer实践

Tue, 14 May 2019 00:00:00 +0000

在基于函数计算的钉钉回调函数接口中使用钉钉回调函数案例实践了AWS Lambda无服务函数。该示例中，我们将自定义的函数代码及依赖的第三方库（比如json处理库jackson, 钉钉openapi加密库, aws dynamodb client等）整体打包为一个部署包，上传到lamdba代码仓库用于函数执行。

然而实际项目中，其实有大量的相关函数可能会共享这些基础依赖库、三方函数库(比如headless chrome(Puppeteer), pandoc, OCR library -- Tesseract等等)或者使用自定义runtime(如官方未支持的java11)的需求。AWS Lambda在去年底发布了Lambda layers功能来满足上述这些实际开发中的需求。

接下来，让我们看看如何将前文中的函数依赖放置到一个单独的layer中，作为不同函数的共享依赖库。

QCon2019北京站回顾

Thu, 09 May 2019 00:00:00 +0000

这周参加了QCon 2019北京站，这里记录下部分印象深刻的主题以及个人感受。

QCon是由InfoQ主办的综合性技术盛会，主题涵盖了大前端、高可用架构、容器技术、大数据、机器学习等各种热门技术主题。其中也不乏下一代分布式应用、混沌工程等前沿有意思的主题，后面会详细介绍相关的主题演讲。

Spring Cloud or Cloud Native

Mon, 29 Apr 2019 00:00:00 +0000

基于Java的Spring Cloud是由Java最大开源生态Spring社区推出的Out-of-Box分布式微服务解决方案，自2016年发布起就被众多开发者看好。Java作为广为流行的服务端编程语言，Spring Cloud也就越来越多的被用于微服务开发。

Spring Cloud集成了Netflix OSS开源项目实现了很多功能(或作为实现之一)，包括服务治理、网关路由、客户端负载均衡、服务间调用、断路器等。Spring Cloud Netflix将很多生产级别微服务能力开箱即用的带到了Spring Cloud架构下的微服务中，帮助开发者快速的构建满足12要素的应用。

在去年底发布的Spring Cloud Greenwich版本中宣布Spring Cloud Netflix中重要的组件Hystrix、Ribbon、Zuul 1等由于上游开源项目进入维护状态，对应的Spring Cloud Netflix项目也进入到维护状态。这些项目将不再适合用于长期维护的产品中！

同时随着近年云计算的发展，特别是Kubernetes成为容器编排平台的事实标准，加上Service Mesh(服务网格)对微服务的服务治理和流量控制，为云原生应用提供了更为现代、平台无关的解决方案。

为Kubernetes中任意应用添加基于oauth2的认证保护 (下)

Mon, 08 Apr 2019 00:00:00 +0000

本文是为Kubernetes中任意应用添加基于oauth2的认证保护的下篇，将图文详解如何使用基于钉钉认证的oauth2 proxy为自身本没有认证授权功能的Web站点实现认证及授权。

基于函数计算的钉钉回调函数接口

Tue, 02 Apr 2019 00:00:00 +0000

由于企业内部管理的需要，用到了钉钉的业务事件回调能力，正好将这个轻量级的接口使用无服务器技术来实现部署，以应对流量无规律下的动态扩展伸缩、按需使用、按量计费等需求。

无服务器计算101

Mon, 01 Apr 2019 00:00:00 +0000

Serverless Computing(无服务器计算)是目前最被看好的云端计算执行模型。其最大的好处是提供分布式弹性可伸缩的计算执行环境，仅为实际使用资源付费，并且将应用维护者从常规的运维事务中解放出来，更利于专注到具体的业务上。

在主流的应用部署方式下，无论是使用云主机还是Kubernetes作为运行环境，都会有大量运维层面的事务需要考虑和处理，并且应用程序需要按照分布式程序的设计准则来应对应用的水平伸缩。同时随着云计算服务的发展和完善，云计算厂商提供了越来越多的基础服务，例如API网关、对象存储、消息队列、日志、监控等服务，函数计算可以完美的同其他云服务集成，帮助用户快速实现出生产级别的弹性可伸缩的应用。

为Kubernetes中任意应用添加基于oauth2的认证保护 (上)

Sun, 03 Feb 2019 00:00:00 +0000

企业随着业务的发展，必然会部署各种各样的IT系统。出于安全性的考虑，一些系统仅可企业内部使用，甚至仅开放给企业部分部门员工使用。

这些IT系统大致可分为两类，

系统本身不支持任何认证机制，例如资讯或文档类系统。需要增加认证保护，能够限制非企业员工访问即可。系统运维通常的做法是，为站点设置HTTP Basic认证保护。由于HTTP Basic认证是通过预设的用户、密码认证，认证信息比较容易泄露。即使定期更换密码，但需要额外的机制通知用户密码的变更，用户体验也不好。
系统自身支持认证，甚至支持多种认证机制。比如最常用的开源CI/CD工具，Jenkins内置支持本地数据库认证、通过插件支持多种第三方系统集成认证。如果大量的IT系统都有一套独立的用户管理，随着企业的员工的变更，用户的增删等操作对系统管理员来说是不小的工作量。同时，也很容易由于人为疏忽，造成资产、数据的安全隐患。

假设企业自身已经有了一套OA系统包含员工、组织结构管理，例如，国内目前最为普及流行的钉钉或企业微信。我们完全可以提供一套基于oauth 2.0协议的认证方式，让以上两类IT系统使用企业已有的OA系统(钉钉或企业微信)来实现登录认证。做到这一点后，企业无论有多少IT系统都不再需要额外管理用户的成本，并且也避免了数据安全隐患。

IAM最佳实践

Fri, 01 Feb 2019 00:00:00 +0000

企业使用公有云服务的第一件事情就是创建云帐号，有了帐号之后如何让企业员工安全合规的使用云帐号下的各种资源是开启云之旅后的第一个考验。

云计算厂商针对企业上云后面临的第一个需求已经推出了完善的解决方案--Identity and Access Management。IAM可以帮助云帐号安全地控制对云计算服务资源的访问。企业可以使用IAM控制对哪个用户进行身份验证 (登录) 和授权 (具有权限) 以使用资源。

云厂商是否提供完善的IAM服务可以作为整体产品解决方案是否成熟的一个衡量指标，比如AWS的IAM和阿里云的访问控制都是较为成熟完善的产品。国内某个以AI能力为卖点的云厂商，在IAM产品方面几乎为零，很难相信对安全合规有需求的企业会完整使用他的云产品作为解决方案。

不要自建Kubernetes

Tue, 22 Jan 2019 00:00:00 +0000

这是“如何高效使用云服务”系列文章的首篇分享。可能有朋友好奇为什么不是从云计算最基础的服务--计算资源ECS/EC2讲起呢？在Cloud Native已经被越来越接受的今天，基于Kubernetes部署、编排应用的方式已经是业界的事实标准。无论是互联网巨头，传统500强企业，还是创业团队都在使用或规划使用Kubernetes作为应用程序的自动化部署、可扩展管理平台。在云计算平台，虚拟机越来越不需要单独的管理，在绝大多数的业务场景下，它们只是作为容器集群所管理的计算资源。甚至虚拟机的创建到销毁整个生命周期管理都可以由Kubernetes根据集群的负载来自动完成。

所有主流的云计算厂商都在解决方案中力推托管的Kubernetes，AWS的EKS，Azure上的AKS，当然少不了Google家GCP上的Kubernetes Engine。国内阿里云，腾讯云等每一个公有云玩家也都基于开源Kubernetes推出了托管服务。如果一家云计算厂商在提供托管Kubernetes这一服务上没跟上业界的步伐，将来极大可能被淘汰出这个市场。

真的会用云服务吗？

Mon, 07 Jan 2019 00:00:00 +0000

这是“如何高效使用云服务”系列文章的引子。该系列将讲述如何利用各种公有云服务来安全合规、高质量、快速、低成本的打造产品/系统，帮助企业（特别是中小微创业团队）在人少，钱缺的情况下做到最高效率。

2018北京ArchSummit回顾

Thu, 13 Dec 2018 00:00:00 +0000

上周参加了ArchSummit(全球架构师峰会)，在这里记录下部分参加的主题以及个人感受。

如何修复Jenkins CI无法读取存在的任务配置

Wed, 12 Oct 2016 00:00:00 +0000

V秘开发团队一直使用着Jenkins CI来持续集成V秘服务的新功能和各种改进。近日，Jenkins CI在重启之后，很多已有任务的配置无法被Jenkins CI完整的加载，导致很多功能无法使用。导致我们整个网站的各种服务无法被升级更新了:-(

MongoDB中如何找出慢查询

Thu, 29 Sep 2016 00:00:00 +0000

MongoDB是目前最为流行的NoSQL数据库之一。V秘的后台数据就是保存在MongoDB中的哦;)

尽管MongoDB的性能为业界称道，但任何数据库系统使用中都存在着慢查询的问题。慢查询的性能问题，可能是由于使用非最优的查询语句，不正确的索引或其他配置原因导致的。但开发人员或数据库维护人员首先要找出这些低效的查询，才能做出对应的查询优化。

在MongoDB中实现慢查询的profile是非常容易，因为MongoDB内置了profile开关来记录执行时间触发了profile条件的查询。

参照db.setProfileLevel()的文档，通过以下命令就可以记录执行时长超过300ms的查询。

1db.setProfilingLevel(1, 300)

当慢查询被重现后，可以通过查找system.profile collection来查看执行时长超过300ms的查询。

被profiler记录下来慢查询record看起来如下，

Docker Swarm mode(v1.12.x)的一些使用限制

Tue, 20 Sep 2016 00:00:00 +0000

Swarm mode在Docker v1.12中正式发布，Swarm mode带来了诸如Docker集群，容器编排，多主机网络等激动人心的特性。V秘团队也尝试着将各种后台服务部署到Docker Swarm Cluster获取更好的弹性计算能力。

创建于Docker Swarm的服务无法在Ubuntu 14.04 LTS中运行

Tue, 13 Sep 2016 00:00:00 +0000

V秘团队一直致力于用技术改善产品。V秘后台的各种服务一直是通过完善的Devops流程自动部署到Docker容器集群。随着Swarm mode在Docker v1.12中正式发布，Swarm mode带来了诸如Docker集群，多主机网络等激动人心的特性。我们也在尝试将V秘服务部署到Docker Swarm Cluster获取更好的弹性计算能力。

基于Angularjs单页面应用的SEO优化

Thu, 19 May 2016 00:00:00 +0000

在之前的文章我曾提到基于Angularjs的单页面应用在用户体验上的种种好处。然而任何事情都不是完美的，Angular和类似的框架通过应用内做页面路由的实现给SEO（也俗称搜索引擎优化）带来了不少麻烦。

首先，我们来看看页面内路由是如何实现的。默认Angularjs生成的页面uri类型如下，

http://mydomain.com/#/app/page1

浏览器请求上面这个uri的时候，实际发送给服务器的请求地址是http://mydomain.com/, web服务器会将默认的页面响应给浏览器，比如index.html或index.php等。

Spring框架下的分布式session管理

Sat, 07 May 2016 00:00:00 +0000

在微服务和容器等技术的帮助下，Web应用可以较为容易的进行水平扩展，来部署更多的应用实例来提升请求处理数QPS。当Web服务有状态的时候，如何在集群下管理用户session成为新的待解决问题。

V秘是如何构建的

Thu, 07 Apr 2016 00:00:00 +0000

春天来了，V秘大家庭也新增了两位10后的传人。新爸爸经过一番忙乱后，希望在这里与大家分享V秘的架构，共同探讨如何快速的构建高可用，高性能的Web服务。

V秘致力于提供最好的在线视频制作云平台。让用户随时随地零门槛的快速制作出高质量高清晰度的视频，来纪念记录生活中有意义的时刻，同时将这份快乐传递给更多的家人朋友一起分享。

然而要可靠的可扩展的实现这样看似简单的需求，其背后确由众多知名开源技术，可靠的云服务，不间歇的监控运维来实现和保证的。

V秘架构的基本目标就是要实现，

服务的高扩展性。有有效可靠的方法支撑数万并发到数十万，百万及更多的并发请求。
服务的高可用性。各种服务都是多实例的集群，某些服务故障后，集群中的其他实例仍然能够提供服务。
服务的自动化构建。从代码到服务部署上线是一套自动化的流程，越少的人工介入保证了服务的可用性。
系统的实时监控。7x24小时的监控保证服务的可用性，当监控到数据异常或服务停止运行能及时告警引入人工运维团队。

更多细节请参阅下面的slides,

说一说阿里云ossfs

Fri, 26 Feb 2016 00:00:00 +0000

阿里云提供的对象或者文件存储叫OSS，为应用程序提供了海量存储，按需付费等服务。应用程序则需要通过Aliyun OSS的各语言SDK才能操作（读，写，遍历等）OSS中的文件。

对运维人员来说，做一些数据维护工作的时候，通过SDK操作OSS中的文件就会比较麻烦。在linux/unix环境下，通常有一些工具把远程文件系统或云盘挂载为本地文件。在网络状况比较好的情况下，操作远程文件就像操作本地文件一样。例如，把Amazon S3，Dropbox云盘，可通过ssh登录的远程服务器上的磁盘挂载为本地文件系统。

如何使用微信公众平台的临时素材

Wed, 27 Jan 2016 00:00:00 +0000

微信给公众平台提供了素材管理的接口，通过这一系列接口可以上传，接收以及管理图片，视频等多媒体文件。其中又分为临时和永久两种类型。永久素材有总量的限制，临时素材微信服务器只给保存3天。

最近V秘刚好有个同微信用户互动的场景，为用户美化微信拍摄的小视频。V秘后台服务器收到用户发送过来小视频（微信将其认做临时素材），将其美化处理后，再将美化的视频上传为临时素材，最终美化后的视频作为视频类型的客服消息被推送给用户。整个流程很简洁，用户发送小视频后，就坐等观看美化后的小视频了。

然而最终经过V秘开发团队的实践及测试，得出的结论是，

##微信公众平台的临时素材不能用！绝对的鸡肋！

公众平台上传素材的API以及使用已有素材发送视频消息API都很健壮。但问题出在了微信后台资源的服务上面。

单页面应用(single page application)中使用微信支付

Sun, 24 Jan 2016 00:00:00 +0000

随着AngularJS等前端MVC框架的流行，AJAX的异步请求数据结合H5的push state等特性，极大的改善了网站的用户体验和页面加载性能。这类网站应用通常只有一个入口页面，通过应用内路由到不同的页面，所以俗称单页面(signle page application)应用。页面URL看起来如下，

文件系统的Inode耗尽，会导致Docker编译镜像出现'No space left on device'错误

Thu, 21 Jan 2016 00:00:00 +0000

最近在提交前端代码后，前端代码的自动发布老是失败。失败的原因多是编译Docker镜像时在执行COPY语句拷贝文件到镜像文件系统时，扔出了'No space left on device'这个错误。这个错误根据描述非常好理解，就是docker文件系统所在磁盘没有了空间。

Daemon hell in Jenkins

Tue, 21 Jul 2015 09:32:00 +0800

Recently I wrote a Linux like initd script to start/stop my web application.

The script works well when running it in shell of linux. The web application will run in background by daemon.

However I found both daemon and web application(java) exited immediately if I started the script in Jenkins as a shell step of build process.

I put below simple script in 'Execute shell' block,

1daemon --name=test-daemon -- sleep 200sleep 60

The process 'daemon' and 'sleep 200' should exit after 200 seconds the 'sleep' exits. The jenkins job will be finished in 60 secs.

The symptoms of Java broken in Mac OSX 10.10 and fix solution

Mon, 13 Jul 2015 19:11:00 +0800

After uninstalling some applications from my Mac OSX, I found the applications that depends on JRE totally does not work. I noticed below symptoms,

Eclipse Mars can not be launched, even though I specified the launching vm to another one(`java -version` still work). The SWT native library failed to resolve the dependencies to '/System/Library/Frameworks/JavaVM.framework/Versions/A/JavaVM' which does not exists.
I tried to reinstall Oracle 1.8.0_u45 via both brew and dmg image downloaded from Oracle website, both ways were failed as well.
The Mac pkg Installer can not be started due to dylib broken. It means I can't install any pkg via GUI. The command line(such as sudo installer -verboseR -target / -pkg /Volumes/OS\ X\ 10.10.4\ Update\ Combo/OSXUpdCombo10.10.4.pkg) still works.

Finally I realized the problem was caused by I uninstalled the out of date Apple Java 6. Looks like all of above failures are required the system built-in Java. It really does not make sense the Oracle 1.8 installer script to depend on the out of date Java.

Run groovy script via Jenkins CLI

Mon, 13 May 2013 13:44:00 +0800

Jenkins supports ssh authentication in CLI.

Below is a command to verify that I am authenticated:

1
2java -jar jenkins-cli.jar -s http://myserver/jenkins who-am-i
3
4    Authenticated as: myuser
5    Authorities:
6        authenticated

However you still would meet permission error when running groovy script in CLI.

1
2java -jar jenkins-cli.jar -s http://myserver/jenkins groovysh 'jenkins.model.Jenkins.instance.pluginManager.plugins.each { println("${it.longName} - ${it.version}") };'
3
4Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
5at $Proxy2.main(Unknown Source)
6at hudson.cli.CLI.execute(CLI.java:271)
7at hudson.cli.CLI._main(CLI.java:417)
8at hudson.cli.CLI.main(CLI.java:322)

It's a bug of Jenkins. The workaround is create a groovy script, then run that script via Jenkins CLI.

Solr boost examples

Sat, 11 May 2013 13:16:00 +0800

The index has a field named 'create_time' that is the timestamp of document created time. The query string can boost the latest created document like below,

{!boost b=recip(ms(NOW,create_time),3.16e-11,0.08,0.05)}name:keyword

There is another field named 'important' that indicates whether the document is important or not. The query string can boost the document is important like below,

q={!boost b=$importfunc}name:keyword&importfunc=query({!v='important:true'})

Above query string uses a sub query in boost function.

Finally I want to boost both above two fields, and 'important' field has higher priority. The query string looks like below,

Django's unicdoe encode error

Thu, 16 Aug 2012 09:11:00 +0800

It's a common and ugly problem when using non-ascii characters in Django.

The general solution is below,

put # -- coding: utf-8 -- at beginning of every python source files that are using utf-8 characters
declare every string variable as unicode, such as str_var = u'中文字符'
add a __unicode__ method in your model classes
if you are running server on apache/mod_wsgi or ngnix, you need configure web server to use utf-8 encoding

The workaround of making Zend CE/Zend debugger work on mountain lion

Sat, 28 Jul 2012 20:49:00 +0800

I installed both Zend CE and zend debugger of Eclipse on my Mac. Both of them work well in Mac lion. However they don't work any more after I upgraded my Mac to mountain lion.

After some investigation I found some extensions of Zend PHP can't be loaded due to shared library dependency can't be found in mountain lion. The xslt module of PHP depends on some system libraries(suc as /usr/local/libxslt-1.1.23/lib/libxslt.1.dylib) that have been removed by mountain lion.

Dual monitors on Ubuntu

Mon, 19 Mar 2012 20:04:00 +0800

I had two monitors for my workstation. One is 22' and the another is 17'. I used the small one as a extend desktop.

Today I get a another 23' monitor to replace the small one. However the resolution of the 23' monitor can't be changed after pluging it in. It always used the resolution matching the 17' one.

Both 'Setting - Display' and 'AMD Catalyst control' can't adjust it as higher resolution.

Embedding an HTTP server in Equinox

Mon, 05 Mar 2012 19:25:00 +0800

I want to create a test server for my application. Using embedding Http server in equinox is my first option.

I had experience using simple http service implementation of equinox, however I want to play with Jetty this time.

Following the guide of Equinox server, I can't running a Jetty server with my servlet in Eclipse Indigo. Obviously the guide is out of date.

After tuning it, I found below bundles are minimum collection to run Jetty inside OSGi runtime.

Acess Intranet without VPN

Fri, 17 Feb 2012 17:47:00 +0800

Sometimes I need access the Intranet of company, however I don't like to create VPN connection. The connection is slow, waste time to create the connection and have to change password due to security policy.

My workstation is Linux, which has a lot of utility tools to help me access Intranet at home without VPN.

Firstly I set up a ssh server on my personal computer. It's quite easy if you are using Linux, for Windows I installed Copssh.
Then register a free domain name and configure it in my router. And let router forward port 22(or any port you wan to use) to my personal computer.
In my working Linux machine, create a ssh tunnel to my personal computer. Must use the public/private key for authenticating. For example,

How to reuse the existing OpenID accounts after the host name of Gerrit server is changed

Wed, 15 Feb 2012 11:36:00 +0800

An internal Gerrit server was moved, so the hostname of server is changed. However we are using OpenID for user control, the OpenID provider(such as Google account) will generate a new token for the new server(hostname changing will impact the identity token of Google account) when we login Gerrit with same OpenID account. Gerrit will create a new internal account by default even though my OpenID account has existed in the system and has a lot of activities.

JRE/JDK's certificate issue and solution

Thu, 24 Nov 2011 15:58:00 +0800

The problem came from I tried to set up send mail server(SMTP) for my Gerrit server. My Gerrit server is using OpenID for user authorization, so I registered a new email account to send notification from Gerrit.

Most of email service providers require the secure authorization when using its SMTP server to send mail. However the root CA of my email provider is not added into the default certificate of JRE. So Gerrit always failed to send email due to ssl verification exception.

The tips of Maven/Tycho building crossplatform RCP and repository

Tue, 08 Nov 2011 16:16:00 +0800

I successfully converted our product build from PDE build to Maven/Tycho. Something is worth to be documented here.

There are several examples and posts to demonstrate how using Tycho building your Eclipse plug-ins, features, applications and products. The most helpful example is the demo of Tycho project.

Below are some traps I met when building my project by Tycho,

product build
Our product is based on plug-ins, however we added the 'featurelist' in build.properties of PDE build to include some root binary for the product. However Tycho doesn't support this type of build, we create some features as the placeholder of plug-ins. Then change the product as features based. You have to manually remove the plugins tag in .product definition file, otherwise Tycho will fail on strange error if the .produce has both features and plugins tag. Then configure the director plugin as not installing features.

Read More

Migration Clearcase to Git -- part 2

Tue, 25 Oct 2011 19:45:00 +0800

Several days ago I had a post to record the unsuccessful experience to migrate source code from Clearcase to Git.

We have a new way after doing some brain storms. This way still is not a perfect solution, but it's much better than previous trial.

Use clearexport_ccase to export the source folder to intermittent data. See documentation of Clearcase admin.
Create a temporary vob for importing the data later. See example.
Import the data into temporary vob. See example.
Repeat step 1 to 3 for importing all necessary data into temporary vob.
Use the SVN Importer to import the temporary vob as Subversion repository.
Last steps refer to a documentation of succeeded migration case of one of Eclipse project from Subversion to Git.

Git definitely is greatest SCM tool now. The size of Subversion repository is around 10GB, finally the Git repository is less than 700MB, which saves more than 10 times disk space. It's awesome!

Migrate Clearcase to Git

Mon, 17 Oct 2011 19:50:00 +0800

I tried to migrate the source code of project from Clearcase to Git repository. As far as I know there is no elegant solution for such migration. For purpose of this migration, I want to keep the history and label of files in Clearcase after migrating to Git repository.

There are mature tools to migrate CVS/SVN repository to Git, so I tried to use Subversion as a bridge for my migration.

p2 query performance

Mon, 17 Oct 2011 19:34:00 +0800

Our p2 based on installer suffered performance issue when querying IUs from repositories. Though the repositories have a large number of IUs to be queried, but we find the performance of using QL is unacceptable in some special scenarios.

I published several different methods to find the expected IUs. Thomas pointed out the better expression of QL and finally helped us to find out the our repository without IIndexProvider implementation.

IIndexProvider implementation of a repository is quite important to improve the performance of QL, especially use the 'traverse' clause to query something.

Create an import library for building application in MinGW

Fri, 12 Aug 2011 10:51:00 +0800

Yesterday I modified an existing c++ application for Windows. And its default build environment is Makefile and MinGW.

However I used a newly Windows API that is not included by header files of MinGW.

First of all, I copied the constant definition from header file of Windows SDK, and defined the Windows API method as a extern C method. So it's no problem to compile the code in MinGW.

Secondly I have to fix the link issue. Because the symbol of the Windows API also can't be found by gcc link.

Customize PDE build

Fri, 22 Jul 2011 13:59:00 +0800

The documentation of PDE has a chapter for this topic. Basically it's simply. Copy the template scripts what you want from templates/headless-build folder under org.eclipse.pde.build plug-in to your build configuration directory that is the folder has build.properties file.

However I found the variables listed in template 'customAssembly.xml' can't be used in the runtime. I filed bug 346370 against it.

Using the certificate of Windows code signing to sign jars

Mon, 18 Jul 2011 21:03:00 +0800

I did sign the jars via reusing the existing certificate of Windows code signing several months ago. Writing it down for further reference.

Whatever your purpose of reusing the existing Windows code certificate, I only document the way from technical perspective.

After buying the certificate of Windows code signing from CA, you will get a .pvk file that stores both the certificate and private key. PVK file is the PKCS12 format[1], however java uses JKS format by default. So you need convert the pvk file to JKS keystore and certificate.

Unlock the locked profile if firefox/thunderbird crash

Wed, 11 May 2011 13:19:00 +0800

I met that firefox/thunderbird complained another its instance running even if no a running firefox/thunderbird process. Finally let them run again after removing the '.parentlock' file in their default profile.

strace utility helps me a lot to find the solution.

strace -f -e file firfox

Eclipse P2's import/export capability

Fri, 22 Apr 2011 15:56:00 +0800

I implemented the replication tool at the end of 2009, then published it to Eclipse Marketplace in May 2010. However it's not pervasively used due to users have to install that plug-in firstly.

I searched a similar request on bugzilla, then I initialized my contribution in the early of this year. Finally it was accepted and will release as part of eclipse itself since Eclipse 3.7 M7! I hope it would benefit the users of Eclipse more and more.

[vim] delete the lines not contain words

Wed, 05 Jan 2011 14:25:00 +0800

:g!/some expression/d

Inside P2's profile (2) - the fragment matches all osgi bundles

Tue, 28 Dec 2010 11:33:00 +0800

Recently our installer met a strange bug, it didn't uninstall all legacy bundles after updating to new version. Finally I found it's due to a magic fragment is missing in the profile due to some causes.

            installBundle(bundle:${artifact})


            uninstallBundle(bundle:${artifact})

            setStartLevel(startLevel:4);

It has 'hostRequirements' element that represents it's a fragment IU and match all the eclipse's plug-ins in that profile. And this fragment defines the touch point actions for its hosts that will do installBundle action during 'install' phrase and uninstallBundle action during 'uninstall' phrase. It's a very good way to remove the duplicate touch point definitions for all eclipse's plug-ins in the profile.

Inside P2's profile (1) - inclusion rules

Tue, 28 Dec 2010 11:13:00 +0800

You would see some interesting properties at the bottom of eclipse's profile.

For example,

It attaches a property named 'org.eclipse.equinox.p2.internal.inclusion.rules' with value 'STRICT' on the IU 'org.eclipse.sdk.ide' with version 3.6.1.M20100909-0800.

It's a very important property for the p2 engine. It means the IU 'org.eclipse.sdk.ide' has been explicitly installed into the profile, so it's not allowed be implicitly updated or removed.

For example,
We have top feature IU 'org.eclipse.sdk.ide' that represents the Eclipse SDK, 'org.eclipse.pde.feature' that represents the Plug-in Development Tool and 'org.eclipse.jdt.feature' that represents the Java Development Tool. And both JDT and PDT are part of Eclipse SDK, so 'org.eclipse.pde.feature' and 'org.eclipse.jdt.feature' are required by 'org.eclipse.sdk.ide'.

stack overflow protector

Thu, 23 Dec 2010 13:25:00 +0800

Latest gcc compiler enables the stack overflow protector that is since GLIBC 2.4. So the library or executable is compiled by latest gcc could be loaded or executed in RHEL4 or Solaris 9 that only have GLIBC 2.3. Hence using option '-fno-stack-protector' to compile the library or executable to make sure it could be executed in older linux release.

g++ -fno-stack-protector -o test.o test

the loop name for 'for' clause in java

Tue, 21 Sep 2010 10:46:00 +0800

Recently I just know such a useful syntax usage of java.

aLoopName: for (;;) {   // ...  while (someCondition)   // ...    if (otherCondition)       continue aLoopName;

rename command

Sun, 19 Sep 2010 11:40:00 +0800

It's a powerful command to rename files in a batch.

Usage:

rename 's/(\d+)$/$1\.txt/' * rename add '.txt' extension name for all files that ends with number.

applying proxy for the softwares without proxy support in linux

Mon, 13 Sep 2010 13:14:00 +0800

If you have http proxy, set it to system environment,

export http_proxy=http://127.0.0.1:8000 Then start the application in that same terminal.

If the proxy is socks proxy, use 'tsocks' to wrap the application in terminal.

Food and Drinking

Mon, 30 Aug 2010 15:11:00 +0800

Honestly speaking, you have eaten the best delicious food if you're living in China. Though we have more and more concerns on the safety of food, we have to recognize that Chinese food is more delicious than others.

The cuisine is simple in Austria. People always use pork, beef, flour, tomato, potato and few green vegetables. So they surprised Chinese cost several hours to make the food.

Working Workspace

Fri, 27 Aug 2010 17:29:00 +0800

Joel posted a blog related to how to hire the great programmers. One of his key points is building comfortable workspaces.

I believe every programmer loves the workspace like Google and Fog Creek. The workspace of Google has been very famous due to its French chef, gymnasium and big sofas. Why is Fog Creek? It's the company created by Joel, he also practiced his theory on his company. Ruan YiFeng posted a blog for it. I bet you would envy the guys working in that office.

Transportation

Fri, 27 Aug 2010 16:20:00 +0800

Salzburg is a small city and is on the banks of the Salzach River. It's easy to go through the city by bus in 30 minutes.

Salzach River

outline view

Riding the bicycle is a very good way to enjoy the beautiful sight of the city. You could see many kids with parents riding bicycle in the sunny weekend.
The train station and a major bus transient station are the same one that is called as 'main station' by local residents. it's not far from the office of company, about 20 minutes by foot.

Working style

Fri, 20 Aug 2010 17:45:00 +0800

There are 20+ staffs in Salzburg office. Most of them are developers, one is administrator of office.
Generally the staffs in Salzburg work more flexible than the staffs working in Beijing.
Some of them live in German. Even though it's not so far as Salzburg, they also need come to Salzburg by train. So sometimes they work at home, use internet and phone as communication tool.
And they have different responsibilities for products. For example, Helmut works on installer, Matthias and Michael are responsibility for QFT testing, Martin N. is focus on license API developing. So everybody has himself schedule, he can decide when he come to office and when leave office based on his working schedule. Nobody cares when you come/leave office or how long you work every day. I believe all of them do well on their jobs.
Furthermore you can work with your dogs together if nobody takes care of them at home.

day 1

Tue, 10 Aug 2010 19:24:00 +0800

度过长途飞行的旅程不是一件容易的事。要在狭小的空间里待上近10个小时，好在是两人出行，半睡半聊的打发过了时间。
到达维也纳之后，出了登机通道看到的居然是一个酒吧类的餐馆。感觉很稀奇，也很有味道。
周围当然少不了免税店和商铺，但远没有首都机场那样的规模。总体感觉就像国内大型超市购物出口一样，而且人也不多。

另外赞一下维也纳机场的Wifi，简单配置就K了，哪像国内的，又要移动号码，还要短信获取，搞半天也没弄定。
转机之前还有段时间，就在机场里到处逛了逛。顺便在一个吧解决了晚餐，同时也尝了杯当地啤酒。

去萨尔茨堡的飞机还是带螺旋桨的，头次坐这样的老式飞机。

Discovering the p2 API

Mon, 17 May 2010 22:35:00 +0800

Check out this SlideShare Presentation:

Discovering the p2 API

View more presentations from Sonatype.

P2 replication tool lives on Eclipse Marketplace

Mon, 17 May 2010 15:41:00 +0800

Days ago I updated my p2 replication tool. It's easier to install it in your Eclipse.
A new component named 'Eclipse marketplace' is added into Eclipse SDK since Helios, which is an application store for Eclipse. People could be easy to install third party plug-ins into their Eclipse.

You can launch marketplace via 'Help' - 'Eclipse Marketplace...', then search key word 'p2' or 'replication' to find the tool. Finally click next to install it.

useful network utility tools

Thu, 11 Mar 2010 14:53:00 +0800

1. tcpdump
tcpdump -n port 80 -i eth0|lo
monitor all package transferred on 80 port on the network interface eth0/lo
2. netstat
netstat -anp|grep java
trace all network traffic on the process named java
netstat -anp|grep 128.224.159.xxx
trace all network traffic on the host whose ip address is 128.224.159.xxx
3. nslookup
nslookup 206.191.52.46
look up the domain name whose ip address is 206.191.52.46

[tip]Find -exec tip

Wed, 24 Feb 2010 13:46:00 +0800

Using -exec command like below, need add escape character for semicolon that separated two commands in shell

1find directory/ -type d -exec chmod a+x {} \\;

Feb 24, 2010 - update:

1find . -maxdepth 4 -type d  -name 'g-vxworks' 2>/dev/null -print

Jan. 7, 2024 - update: You might see No such file or directory when combining --exec rm to delete the found files.

You can add -depth option to mitigate the message.

special characters in p2 touchpoint instruction

Fri, 05 Feb 2010 16:15:00 +0800

I suffered p2 installation failed on the configure parse. Becase I try to add vm arguments for my application.
For example, I added '-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8272' in the product configuration.
P2 will fail when parsing the argument, because it contains ':' and ',' that should be escaped.
It works again after replacing it to '-agentlib${#58}jdwp=transport=dt_socket${#44}server=y${#44}suspend=n${#44}address=8272'.
The more detail note could be found in p2 touchpoint wiki.
And I also opened bug to request improving it.

mount windows share folder

Fri, 29 Jan 2010 13:22:00 +0800

mount -t cifs -o username=xxx,password=xxx,workgroup=xx,iocharset=utf8 //share.domain/folder /localfolder

the way to dump hex file

Wed, 27 Jan 2010 17:27:00 +0800

1. open the file in vim, :%!xxd
2. hexdump

How to get the name of running test case in JUnit4

Mon, 11 Jan 2010 14:57:00 +0800

public class NameRuleTest {     @Rule public TestName name = new TestName();         @Test public void testA() {                 assertEquals("testA", name.getMethodName());         }        @Test public void testB() {                 assertEquals("testB", name.getMethodName());         }}

How p2 UI handles with license agreements

Wed, 30 Dec 2009 16:56:00 +0800

P2 install wizard firstly query the repository to find out the root installable unit(as well as top installable). Then p2 recalculate the dependency and try to search the requirements in all available repositories after user submits their installation request. Go to the license agreement page if all the dependencies are satisfied.

P2 agreement page obtains all the units to be installed from the operands of provision plan. The number always is much greater than the number submitted by user. Because the submitted IUs only are the root IUs.

[tip]ssh key

Tue, 29 Dec 2009 11:33:00 +0800

Setup SSH without password.
a) execute "ssh-keygen -t rsa" under your linux/unix login to get the RSA keys.
(press Enter for all)
You will get 2 files uner ~/.ssh/, id_rsa and id_rsa.pub
b) Put the public key id_rsa.pub to your remote host: ~/.ssh/authorized_keys If the remote host share the same nfs, just try " cat id_rsa.pub >> ~/.ssh/authorized_keys"
* Remember to modify hostname or ip info in ~/.ssh/authorized_keys to "", so that you can login from any host without password in your NIS domain.
For example:
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4Ri5J0s1BL/+mR7RfAuDW6FY2P6ILc61Zvw1BdDkvHMFrTzaC/AUMw33H7biAMCXuCleakCuSoV8ZDiGHYs4wOVvet5sDmphkwdiC4xTekdl3dRNvGjMVbvFUta/Y5CiayL6YIu47Ro6Vvu4Mutsrv/13pTlifrEz+NTR/+bzMb9nTniCwiryMyYod3E46b8WvS8yE3WK+tH4BZE8bjiCwdvAzSdPyk/OFNrlBNuF1yewwnxv1roRD3UalT2+7O4kfEG9sMvvBHjuX2l7xlUe3stBftYpigBbwGmmadxjRpNIlk88t5xKcQX6nSu7V8HI3GWPHI0D+ISIlbfU5Sunw== kzhu0@

[Eclipse][P2]P2 replication plug-in

Fri, 25 Dec 2009 16:43:00 +0800

I wrote a plug-in to simplify the process to install the same plug-ins in different platform or different workstation.
Anyone is interested in it, pls follow below guide to freely use it.
http://code.google.com/p/kane-toolkit/wiki/P2Replication

Enjoy it.

[eclipse]How Equinox load bundles

Wed, 02 Dec 2009 14:58:00 +0800

How Equinox load bundles

Equinox launcher is responsible to start OSGi framework. The system bundle would be created and marked as installed when initializing the framework. Equinox also tries to install the installed bundles if finding them in persistence data during the initializing period. Of course there is no extra bundles would be installed when launching Equinox first time.

Then Equinox launcher would install the bundles specified by vm's system property 'osgi.bundles'. And start the initial bundles that are marked as early start. For example, let's have a look at the configuration/config.ini of Eclipse, you would find a line similar as below,
osgi.bundles=reference\:file\:org.eclipse.equinox.simpleconfigurator_1.0.200.v20090831.jar@1\:start
It means the start level of bundle 'org.eclipse.equinox.simpleconfigurator_1.0.200.v20090831.jar' is 1, and it would be started after installing it.

[Eclipse][P2]Learn p2 step by step

Thu, 12 Nov 2009 17:48:00 +0800

Learn p2 step by step

See this link for detail

Learn P2 step by step

p2 concept

首先来理解p2引入的几个概念[1]

p2 / Agent
The provisioning infrastructure on client machines
Installable Unit (IU)
Metadata that describes things that can be installed/configured
Artifact
The actual content being installed/configured(e.g., bundle JARs)
Repository
A store of metadata or artifacts

Read More

[tip]ssh forward

Wed, 28 Oct 2009 15:29:00 +0800

ssh -qTfnN -D LocalPort remotehost

All the added options are for a ssh session that's used for tunneling.

-q :- be very quite, we are acting only as a tunnel.
-T :- Do not allocate a pseudo tty, we are only acting a tunnel.
-f :- move the ssh process to background, as we don't want to interact with this ssh session directly.
-N :- Do not execute remote command.
-n :- redirect standard input to /dev/null.

Simulate p2 self host in Eclipse run

Tue, 27 Oct 2009 16:12:00 +0800

-Dosgi.install.area=<launcher's folder>
-Declipse.p2.profile=

Eclipse/OSGi preference

Thu, 22 Oct 2009 15:29:00 +0800

The IPreferenceStore API of Eclipse is based on OSGi's preferences service. Equinox implements several scope context for different preferences, such DefaultScope, InstanceScope and ConfigurationScope. The IPreferenceStore is the wrapper of instance scope for back-compatibility. It stored the data in workspace(osgi.data.area).

The workspace folder would be created when launching RCP application if it doesn't exist. But we can use argument '-data @none' to suppress the creation of workspace. If that, the instance scope/IPreferenceStore can't store any value any more.

The usage of Eclipse's Proxy API

Wed, 21 Oct 2009 16:53:00 +0800

Eclipse platform register an OSGi service 'IProxyService' to manage network connection, which has capability to set proxy setting. There are three types of proxy working mode,

Direct(no proxy),
Manual(specified by user),
Native(using OS's proxy setting, such as gnome-proxy, IE).

There are three types of proxy supported by IProxyService. They're http, https and socks.

It also allows to add/remove ip address from white list, which are accessed without connecting proxy.

End users can manage the proxy setting of Eclipse via Preference - General - Network Connections. Eclipse would do persistence of user's setting. Other components of Eclipse also use those proxy settings to access network, such as ECF.

[tip]Turn off automatically scanning disk when reboot

Wed, 11 Feb 2009 17:40:00 +0800

tune2fs -i 0 -c 0 /dev/sdx

How to set default input method for GNOME

Mon, 09 Feb 2009 11:24:00 +0800

add below lines in ~/.gnomerc

export XMODIFIERS="@im=fcitx"
export GTK_IM_MODULE="xim"

[HowTo]How to set up jre environment in firefox

Mon, 29 Dec 2008 12:52:00 +0800

create symbol link under lib/plugins of firefox to link jre/plugins/i386/ns**/libjavaplugin_oji.so

[Debug][tip]How to ignore specified signal when debugging program via gdb

Wed, 10 Dec 2008 15:02:00 +0800

Such as,
(gdb) handle SIGPIPE nostop noprint pass

[HowTo][tip]How to adjust the font size of Notes editor

Thu, 04 Dec 2008 13:11:00 +0800

Close Notes
Double click c:\notes\notes.ini to open it.
Add one new line "Display_font_adjustment=n" after the third line in notes.ini file. "n" is the number.It can be 1or 2 or 3....and the font will be larger with the number increasing.
Launch note

[OSGi][Eclipse]Add custom jar or path into Equinox Framework

Sun, 28 Sep 2008 13:23:00 +0800

Set vm arguments 'osgi.framework.extensions' and 'osgi.frameworkClassPath' when vm starts. If those value are set, those jar or path would be added into the classloader when starting EclipseStarter.

See org.eclipse.equinox.launcher.Main for more details in the source code of Eclipse 3.4.
Best Regards
Kane

[OSGi]How to acquire the fragments of specified bundle

Fri, 01 Aug 2008 16:22:00 +0800

The answer is very simple, using the service 'org.eclipse.service.PackageAdmin'.

[Eclipse]Equinox's classloader and its URL schema

Thu, 17 Jul 2008 17:42:00 +0800

Equinox uses the adaptor hooks to implement the class loader.
See http://wiki.eclipse.org/Adaptor_Hooks for more detail

BaseClassLoadingHook would search the native code on itself. If it find the file in that jar file, it would extract the native library into its storage folder.

EclipseClassLoadingHook defines some variables to search the native library. Belows are built-in variables:

result.add("ws/" + info.getWS() + "/"); //$NON-NLS-1$ //$NON-NLS-2$
result.add("os/" + info.getOS() + "/" + info.getOSArch() + "/"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$
result.add("os/" + info.getOS() + "/"); //$NON-NLS-1$ //$NON-NLS-2$

[tip][vim]排版小技巧

Tue, 15 Jul 2008 11:00:00 +0800

1. 将要排版的文字贴到vim了
2. set textwidth=70
3. visual模式下选择要排版的文字,按gq, 就变成70字母1行的格式了

[tip]convert dos format to unix

Fri, 23 May 2008 18:03:00 +0800

In vi/vim,

set file format=unix

or dos2unix, unix2dos

[tip]Makefile

Thu, 22 May 2008 20:37:00 +0800

Build c/c++ project always need third party library on linux, such as gtk+, glib. Writing their absolute path in Makefile is not flexible way. You can use pkg-config instead of the absolute path. Below is code snippet:

GTK_LIB=$(shell pkg-config --libs gtk+-2.0)
GTK_INC=$(shell pkg-config --cflags gtk+-2.0)

gcc -o yourlibrary.so $(GTK_INC) $(GTK_LIB)

[OSGi][Equinox]URL Handlers Service

Wed, 16 Apr 2008 14:20:00 +0800

OSGi provides a mechanism to let user contribute custom schemes automatically. It avoid some restriction with Java facilities for extending the handlers. The more detail could be found from OSGi specification R4, which has description how OSGi implements URL Handler Service.

Use a sample to illustrate how to contribute your scheme(protocol):

1. register your URLStreamHandlerService implementation, which must contain a property named "url.handler.protocol". below register my scheme 'smb'
public void start(BundleContext context) throws Exception {
Hashtable properties = new Hashtable();
properties.put( URLConstants.URL_HANDLER_PROTOCOL, new String[] { "smb" } );
context.registerService(URLStreamHandlerService.class.getName(), new SmbURLHandler(), properties );
}
2. your URL Handler extends AbstractURLStreamHandlerService, and implements abstract function 'openConnection(URL)'
public class SmbURLHandler extends AbstractURLStreamHandlerService {

[OSGi][Equinox]the Bundle-NativeCode implementation in Equinox

Mon, 31 Mar 2008 17:36:00 +0800

OSGi Spec defines Bundle-NativeCode header to contain a specification of native code libraries contained in that bundle. All magic things are initialized by org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLibrary(String) and org.eclipse.osgi.framework.internal.core.BundleLoader.findLibrary(String). Then BundleLoader uses the org.eclipse.osgi.baseadaptor.BaseData(an implementation of BundleData) to find the library path, if the bundle is NOT a jar file, it would directly get the absolute path of library. Otherwise, the BaseData would extract the library file if it could NOT find it in OSGi bundle storage(located in ${data}/org.eclipse.osgi/bundles/[bundle_id]/.cp/). Refer to org.eclipse.osgi.baseadaptor.BaseData.findLibrary(String) for more detail.

[shell]Learning Note - 3/19/08

Wed, 19 Mar 2008 14:35:00 +0800

1> 是输出正确数据， 2> 则是错误数据输出项目, 若要同时写入同一个档案需要使用 2>&1 /dev/null 是什么呢？基本上，那就有点像是一个『黑洞』的垃圾桶功能！当你输入的任何东西导向到这个虚拟的垃圾桶装置时，『他就会凭空消失不见了～～』

[Eclipse]How to use qualifier string when exporting features and plug-ins

Tue, 11 Mar 2008 13:30:00 +0800

You must see the qualifier string property when exporting your features and plug-ins by Eclipse pde. But specified qualifier string won't appear after you export the features successfully.

If you want to use the qualifier string, you must define your feature and plug-in version like below:
1.0.0.qualifier, 2.2.2.qaulifier

万恶的注册表

Tue, 03 Apr 2007 13:16:00 +0800

最近几天被一个注册表相关的defect搞的焦头烂额。

背景是这样的，产品在安装的时候需要通过修改注册表注册文件关联等信息。在先前安装程序基于InstallShield时工作正确，但在最近安装程序改用MSI后，我们写入注册表的信息没有被写到所期望的位置。

通过各种试验，查找资料，终于搞明白原因。我们修改注册表的进程不是当前用户进程，而是系统进程，因此写入到HKEY_CURRENT_USER下的数据不能被写入到当前登陆用户下。

We should not use "HKEY_CURRENT_USER" to retrival current user's registry key value. Because Windows Services always startup before user login. It may happen some error or loading the wrong setting profile. If you still insist on using the current user registry key setting, please refer "RegOpenCurrentUser".

最后只好将这些数据写到了Local Machine键值下。

[Eclipse]get rid of the menus of eclipse platform

Fri, 09 Mar 2007 13:46:00 +0800

When you develop a rich client application base on eclipse framework, and your application require eclipse platform feature, you would find that your application has some menu items contributed by eclipse platform. Those menu items are defined by several plug-ins' implementation of actionSet extention point. In fact Eclipse provides an activity mechanism to suppress the extension points which you don't want to use. However, you must know the identification name of extension points which you want to suppress. It's a hard work to find out all of them from dozens of plugins. so, I wrote a utility function to list all the extension points of specified name.

加载jar文件里的本地库

Wed, 24 Jan 2007 14:19:00 +0800

java程序开发中经常用到JNI调用本地library, 同时又希望将library同class文件编译成一个jar文件以方便deploy.

但是JDK的classloader不支持从jar文件中加载library, 一个变通的方法就是jar里的library以临时文件的方式写到临时目录或java.library目录.

附上两篇文档链接 :

**Load Library inside a jar file
**

使用JNI时，装载本地库的小技巧

[Eclipse]Eclipse update support

Wed, 17 Jan 2007 13:51:00 +0800

Those days my work is focus on eclipse's update. Now I understand the general mechanism and meet some issues when using it in development work.

The update mechanism includes four major types: install, enable, disable and uninstall. And all of those operations can be executed by command line, such as installing a feature can use following line:
-application org.eclipse.update.core.standaloneUpdate -command install -featureId my.feature -version 1.0.0 -from file:/v:/local_updateSite/ -to file:/v:/eclipse/.
The installation process would copy the feature and plugins which are included by the feature to the local site from the update site, then execute the feature's global install handler if it has one.

[Eclipse]The call sequence between partActivated and menu update

Thu, 19 Oct 2006 12:42:00 +0800

I met a defect that dynamically created menu items disappear after creating a new viewPart. It caused me overtime last Friday. Today I find the root cause.

The scenario is:

open first document, the items are shown well
open another document, the items disappear

The requirement is that showing the menu items while current part is document, otherwise hide them.

So our implementation is:

when current document part is deactivated, set menu items invisible

Read More

[debug][java]Remote debug in Eclipse

Wed, 18 Oct 2006 12:53:00 +0800

I need use remote debug in our project, however just some experience in Weblogic were found from internet. After my investigation, I got some experience about using Eclipse remote debug RCP.
There are two important parameters for jvm. And we must launch remote java app with those two parameters.
-Xdebug //tells jvm starting with debug mode
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=1044 //transport=dt_socket represents communication with socket, address=1044 represents that the port number is 1044
Then there are 3 steps in local env:
1.import source code into eclipse's project
2.Debug-Remote Java Application, see attachement as a sample
3.insert breakpoint,