Picsha AI

Ingestion Pipeline

The moment an asset hits our S3 storage ingress, Picsha AI fires off an asynchronous task to our powerful Queue Worker (picsha-ai-ingest). This isolated node performs heavy lifting: identifying magic bytes, rendering proxies, extracting text, and generating rich multimodal AI embeddings.

Because we process diverse asset classes ranging from flat images to 40MP RAW files, multi-page PDFs, and MP4 videos, the pipeline behaves dynamically depending on the detected MIME type.

Phase 1: Security & Detection

1. Magic Byte Analysis We never trust file extensions or the generic application/octet-stream MIME types uploaded by web clients. The moment an asset is ingested, we use ExifTool to inspect the literal file headers (magic bytes) to securely lock down its true format.

2. Antivirus & Malware Scanning (ClamAV Sidecar) To safeguard user data and satisfy isolated security boundaries, Picsha integrates a high-speed malware scanning pipeline using a ClamAV TCP Daemon sidecar.

  • On by Default, Opt-Out Available: Antivirus scanning is enabled by default, so no unscanned file ever enters your library. Developers ingesting only trusted internal content can opt out per upload by setting "antivirus_scan": false in their upload configuration.
  • Granular Pay-As-You-Go Pricing: Scans are billed at a flat rate of $1.00 per 1,000 scans ($0.001 each) recorded under the antivirus_scan billing event, with 0 monthly free units (billed from first usage).
  • Quarantine Isolation: If malware is detected, the ingest worker automatically quarantines the file, marks the asset status as 'infected', appends detailed scanning telemetry (ClamAV engine version and virus signature) inside the rawExif block, and throws a processing error to abort subsequent AI pipelines, preventing any compromised assets from ever reaching downstream CDN delivery or client applications.

3. AWS Rekognition Content Moderation (Safety Check) To minimize processing latency and avoid unnecessary customer charges, AWS Rekognition Content Moderation is disabled by default. Developers and clients can explicitly enable it by specifying "content_moderation": true inside the upload JSON configuration.

  • Automatic Isolation: When enabled, if unsafe labels (such as explicit, highly suggestive, violent, or hateful imagery) are detected on physical image derivatives with a confidence score of $\ge 80%$, the worker overrides the asset's system status to 'pending_moderation' in both Neon PostgreSQL and OpenSearch.
  • Delivery Lockdowns: Downstream media delivery pipelines (/v1/assets/:id/render or /v1/fetch) automatically block access to assets in 'pending_moderation' or 'rejected' states, throwing a 403 Forbidden error.
  • Margin & Performance Optimization: By default, skipping this Rekognition safety check entirely avoids external API calls, reducing overall resource consumption and processing time.

Phase 2: Derivative Generation

Every media type requires unique processing steps to sanitize it for ultra-fast web delivery.

📸 Standard & Complex Image Handling

  • Web Images (JPEG/PNG): We generate a .webp optimized web delivery version and an ultra-fast 150px grid thumbnail.
  • Complex Images (HEIC, RAW, PSD, EPS, AI): Browsers cannot render these. We first safely drop the image into an ImageMagick/LibRaw memory pipeline to extract the primary layer and flatten it into a universally readable high-quality JPEG Proxy. We then route that Proxy into the standard optimization pipeline to generate .webp variants.
  • Background Removal Cutout (opt-in): When the upload config sets "remove_background": true, a dedicated segmentation model extracts the subject into a transparent alpha cutout PNG, stored alongside the original. Cutouts are deduplicated by content hash — identical pixels are only ever processed (and billed) once — and power instant bg_rem extraction and bg_asset scene compositing at render time with zero inference in the delivery path.

📄 Document Handling (PDF, DOCX, PPTX, XLSX, TXT, MD, CSV, HTML, and more)

  • PDF Conversion: If a supported document format (like .docx, .pptx, .xlsx, .md, .csv, .html, etc.) or raw text file is found, we immediately convert it into a standard web.pdf to ensure uniform cross-device viewer compatibility.
  • Poster Extraction: The first page of the document is rendered down into a high-res poster.jpg, which serves as a cover image for our grid views.
  • Text Extraction: The raw text within the document is scraped and securely buffered into memory.

🎥 Video & Audio Handling

  • HLS Streaming: If the adaptive_stream flag is utilized, the video triggers a dedicated AWS Elemental MediaConvert workflow. This transcodes the massive mp4/mov into staggered .m3u8 chunks (1080p, 720p, 480p) allowing for seamless buffering on the client.
  • Cover Image: A snapshot from the video timeline is extracted and saved as the asset's visual poster.jpg.
  • Audio Extraction: If the asset is a video, the internal audio track is temporarily extracted and uploaded as an MP3 file for speech processing.

Phase 3: Artificial Intelligence

With the clean derivatives generated, we invoke our multimodal LLM architecture to construct the Search Engine indexing.

1. Vision Analysis & Formatting (Rekognition & Anthropic Claude) Any asset possessing a physical image derivative (Photos, Document Posters, Video Posters) is pushed to AWS Rekognition. This executes bounded-box inference scaling for facial recognition and, if explicitly enabled in the upload configuration, content safety/moderation checks. We simultaneously pass the proxy to the Anthropic Claude module to read and summarize the visual context natively.

2. Transcript & Document AI

  • Audio/Video: Audio is fed into Amazon Transcribe for full textual transcripts.
  • Documents & Transcripts: The resulting text (or document text) is then forwarded to Anthropic Claude to cleanly summarize the massive multi-page payloads into actionable paragraphs.

3. Image Quality Metrics & Face Attributes Every physical image derivative is also scored with a set of deterministic, no-inference quality metrics (enabled by default; opt out per upload with "quality_metrics": false). All scores are 0–100 and computed locally in the worker — no external API calls, no per-metric billing:

  • sharpness — variance-of-Laplacian blur detection (blurry photos land well below sharp ones)
  • brightness and contrast — luminance mean and spread
  • clippedHighlightsPct / clippedShadowsPct — percentage of blown-white and crushed-black pixels

When face indexing has found faces in the image, the worker additionally runs AWS Rekognition face-attribute detection and records per-face signals: eyes open, smile, sunglasses, head pose (yaw), per-face brightness/sharpness, and the dominant emotion. Together these power shot selection — picking the best frame from a burst, culling blinks and blurs, or letting an AI layout engine (like the Picsha Books "book brain") choose hero images automatically.

Both are persisted inside the asset's ai_analysis record: quality scores are projected into OpenSearch (so search can rank by them), while the more verbose faceDetails array lives in PostgreSQL only and is returned by the asset detail and batch-analysis endpoints. Failures in this step are non-fatal — an asset never errors out because a quality pass hiccuped.

4. The Unified Multimodal Embedding (Titan) Finally, the "holy grail" of our Ingest architecture occurs. We take the Proxy Image + the generated Text Summary/Transcript and fuse them together within the Amazon Titan Multimodal Framework. This calculates a massive 1024-dimensional vector that represents both the visual pixels and the profound textual context identically in mathematical space.

Phase 4: Persistence

The final dimensions, EXIF data, GPS coordinates, textual transcripts, and multimodal vectors are pushed simultaneously to Neon (PostgreSQL) for user-metadata management, and AWS OpenSearch for blistering conversational Search Indexing.

If cache warming is requested via the API (render_on_upload), the worker dispatches HTTP requests back through our CDN endpoints so the listed renders are already generated and cached at the edge before your first user asks for them — no cold-start transform on the critical path.