Picsha AI

Picsha AI Image Delivery Primer

This document provides a comprehensive overview of how the Picsha AI platform handles the entire lifecycle of an asset, from the moment of upload (ingestion) to its final optimized delivery to end-users globally via our CDN.

1. The Ingestion Phase (What We Do on Upload)

When an asset is uploaded to Picsha, it triggers an asynchronous job in the picsha-ai-ingest Fargate worker. Because web browsers natively support only a limited set of image and document formats, the ingestion phase is primarily responsible for creating web-compatible "proxies" and optimized samples of the original file.

AI-Powered Extraction & Analysis

Before rendering previews, the asset undergoes deep metadata extraction and AI analysis:

  • Metadata via ExifTool: A persistent exiftool-vendored background process pool accurately extracts EXIF data, embedded color profiles, and true MIME types (using magic-byte detection) even if the file lacks an extension.
  • Object & Facial Analysis (AWS Rekognition): To ensure API compatibility, the ingest worker temporarily scales down large files to compliant JPEGs before sending them to AWS Rekognition. This extracts object labels and maps detected faces into specific Rekognition Collections.
  • Scene Analysis (Amazon Titan): For image embeddings, we use the Amazon Titan Multimodal Framework. This generates rich vector embeddings used by OpenSearch for semantic queries.
  • AI Summaries (Anthropic Claude): We pass the extracted textual content (or proxy image) to Anthropic's Claude via Bedrock to generate robust semantic summaries.

Making Samples of Photos

We process images using Sharp and ImageMagick depending on their complexity. The ingest worker immediately and automatically generates a standard set of browser-compatible derivative samples for every asset:

  1. proxy.jpg (The Universal Proxy): For "complex" media formats that browsers cannot render directly (e.g., HEIC/HEIF from iPhones, RAW files like CR2/NEF, and design files like PSD, AI, EPS), the ingest pipeline uses ImageMagick to flatten and decode the file into a high-quality JPEG. (Note: While legacy systems explicitly invoked Ghostscript for EPS parsing, the modern ingestion pipeline calls ImageMagick, which seamlessly delegates EPS and PostScript rendering to the underlying Ghostscript engine). This proxy acts as the source of truth for all future delivery transformations.
  2. optimized.webp: A web-ready, highly compressed delivery format optimized for fast loading on standard screens.
  3. thumb.webp: A lightweight, 150px preview used extensively in the platform's grid layouts and UI.

Creating PDF Representations of Docs and Markdown Files

For non-image text assets, the goal is to generate a visual, browser-viewable "poster" proxy so users can preview the document natively in the UI without downloading it. The DocumentService handles this conversion:

  • Office Documents (.docx, .doc, .rtf, .pptx, .xlsx, .csv): We utilize a headless LibreOffice container (soffice --headless --convert-to pdf) to faithfully convert these documents and spreadsheets into a PDF representation.
  • Markdown & HTML Files (.md, .html): Markdown files undergo a specialized pipeline. First, the raw text is parsed using marked into styled HTML, complete with GitHub-flavored typography, code blocks, and layout wrappers. Once the HTML document is generated (or if the asset was natively HTML), it is passed into LibreOffice to convert the styled HTML into a polished, visual PDF proxy.

These PDF proxies are ultimately surfaced to the frontend as the visual representation (poster.jpg) of the textual asset.

2. The Delivery Phase (CloudFront & Lambda)

Asset delivery is managed by the picsha-ai-image-delivery serverless service. It employs an AWS API Gateway + AWS Lambda architecture sitting behind a globally distributed Amazon CloudFront CDN (cdn.picsha.ai).

Dynamic Sourcing and Transformation in Lambda

When a request reaches the Lambda function (e.g., /render/{assetId}?w=500&h=500&pos=attention), the imageDeliveryService performs the following steps:

  1. Source Determination: It intelligently decides which S3 object to pull. For standard images, it pulls the original. For complex images (HEIC/RAW), it dynamically swaps the source to the assets/${id}/proxy.jpg generated during ingestion. For documents and videos, it pulls the poster.jpg.
  2. On-the-Fly Processing: Using the Sharp library, the Lambda applies all requested query parameters. This includes resizing, "face gravity" cropping, blurring, watermarking, text compositing, background operations (removal/replacement), and even AI-driven generative editing via the signature MIMI natural language orchestrator (supporting both native 4K generation via mimi_mode=4k and a free preview-faithful 4K export upscale of the 2K render via upscale=4k).
  3. Response: The final transformed image is returned synchronously to the client. For standard payloads, this is a Base64-encoded binary response. However, if the generated image (such as a 4K render) exceeds the AWS Lambda 6MB response payload limit, the Lambda instantly uploads it to a short-lived S3 bucket and returns a 302 Found redirect to a Presigned URL, completely bypassing standard Serverless payload bottlenecks.

Sampling in the CDN (Caching Strategy)

To ensure the Lambda function is not overwhelmed and to provide sub-50ms response times to end-users, we rely heavily on CloudFront edge caching.

Where and How We Sample:

  • CloudFront is configured to forward Query Strings (ForwardedValues: QueryString: true).
  • In the context of the CDN, sampling refers to caching the unique permutations of the image transformations. The "cache key" for CloudFront is the exact combination of the URL path, the specific query parameters requested, AND the client's Accept header (to support format=auto negotiation safely).
  • Example: A request for ?w=800 is sampled and cached separately from a request for ?w=800&blur=10.
  • TTL: Every unique sample generated by the Lambda is cached at the CloudFront edge locations for 31,536,000 seconds (1 Year) (DefaultTTL: 31536000) and marked as immutable in the Cache-Control headers.

This architecture guarantees that the expensive compute required for complex transformations (or retrieving proxy.jpg files) only happens once per unique parameter combination, after which it is served directly from the CDN edge node closest to the user.