Picsha AI

Ingestion Pipeline

The moment an asset hits our S3 storage ingress, Picsha AI fires off an asynchronous task to our powerful Queue Worker (picsha-ai-ingest). This isolated node performs heavy lifting: identifying magic bytes, rendering proxies, extracting text, and generating rich multimodal AI embeddings.

Because we process diverse asset classes ranging from flat images to 40MP RAW files, multi-page PDFs, and MP4 videos, the pipeline behaves dynamically depending on the detected MIME type.

Phase 1: Security & Detection

1. Magic Byte Analysis We never trust file extensions or the generic application/octet-stream MIME types uploaded by web clients. The moment an asset is ingested, we use ExifTool to inspect the literal file headers (magic bytes) to securely lock down its true format.

2. Antivirus & Malware Scanning (ClamAV Sidecar) To safeguard user data and satisfy isolated security boundaries, Picsha integrates a high-speed malware scanning pipeline using a ClamAV TCP Daemon sidecar.

  • Selective Toggle: Antivirus scanning is disabled by default to avoid unexpected billing charges. Developers can explicitly enable it for external files by setting "antivirus_scan": true in their upload configuration. It can also be enabled globally via the Picsha Developer Dashboard.
  • Granular Pay-As-You-Go Pricing: Scans are billed at a flat rate of $1.00 per 1,000 scans ($0.001 each) recorded under the antivirus_scan billing event, with 0 monthly free units (billed from first usage).
  • Quarantine Isolation: If malware is detected, the ingest worker automatically quarantines the file, marks the asset status as 'infected', appends detailed scanning telemetry (ClamAV engine version and virus signature) inside the rawExif block, and throws a processing error to abort subsequent AI pipelines, preventing any compromised assets from ever reaching downstream CDN delivery or client applications.

3. AWS Rekognition Content Moderation (Safety Check) To minimize processing latency and avoid unnecessary customer charges, AWS Rekognition Content Moderation is disabled by default. Developers and clients can explicitly enable it by specifying "content_moderation": true inside the upload JSON configuration.

  • Automatic Isolation: When enabled, if unsafe labels (such as explicit, highly suggestive, violent, or hateful imagery) are detected on physical image derivatives with a confidence score of $\ge 80%$, the worker overrides the asset's system status to 'pending_moderation' in both Neon PostgreSQL and OpenSearch.
  • Delivery Lockdowns: Downstream media delivery pipelines (/v1/assets/:id/render or /v1/fetch) automatically block access to assets in 'pending_moderation' or 'rejected' states, throwing a 403 Forbidden error.
  • Margin & Performance Optimization: By default, skipping this Rekognition safety check entirely avoids external API calls, reducing overall resource consumption and processing time.

Phase 2: Derivative Generation

Every media type requires unique processing steps to sanitize it for ultra-fast web delivery.

📸 Standard & Complex Image Handling

  • Web Images (JPEG/PNG): We generate a .webp optimized web delivery version and an ultra-fast 150px grid thumbnail.
  • Complex Images (HEIC, RAW, PSD, EPS, AI): Browsers cannot render these. We first safely drop the image into an ImageMagick/LibRaw memory pipeline to extract the primary layer and flatten it into a universally readable high-quality JPEG Proxy. We then route that Proxy into the standard optimization pipeline to generate .webp variants.
  • Background Removal Cutout (opt-in): When the upload config sets "remove_background": true, a dedicated segmentation model extracts the subject into a transparent alpha cutout PNG, stored alongside the original. Cutouts are deduplicated by content hash — identical pixels are only ever processed (and billed) once — and power instant bg_rem extraction and bg_asset scene compositing at render time with zero inference in the delivery path.

📄 Document Handling (PDF, DOCX, PPTX, XLSX, TXT, MD, CSV, HTML, and more)

  • PDF Conversion: If a supported document format (like .docx, .pptx, .xlsx, .md, .csv, .html, etc.) or raw text file is found, we immediately convert it into a standard web.pdf to ensure uniform cross-device viewer compatibility.
  • Poster Extraction: The first page of the document is rendered down into a high-res poster.jpg, which serves as a cover image for our grid views.
  • Text Extraction: The raw text within the document is scraped and securely buffered into memory.

🎥 Video & Audio Handling

  • HLS Streaming: If the adaptive_stream flag is utilized, the video triggers a dedicated AWS Elemental MediaConvert workflow. This transcodes the massive mp4/mov into staggered .m3u8 chunks (1080p, 720p, 480p) allowing for seamless buffering on the client.
  • Cover Image: A snapshot from the video timeline is extracted and saved as the asset's visual poster.jpg.
  • Audio Extraction: If the asset is a video, the internal audio track is temporarily extracted and uploaded as an MP3 file for speech processing.

Phase 3: Artificial Intelligence

With the clean derivatives generated, we invoke our multimodal LLM architecture to construct the Search Engine indexing.

1. Vision Analysis & Formatting (Rekognition & Anthropic Claude) Any asset possessing a physical image derivative (Photos, Document Posters, Video Posters) is pushed to AWS Rekognition. This executes bounded-box inference scaling for facial recognition and, if explicitly enabled in the upload configuration, content safety/moderation checks. We simultaneously pass the proxy to the Anthropic Claude module to read and summarize the visual context natively.

2. Transcript & Document AI

  • Audio/Video: Audio is fed into Amazon Transcribe for full textual transcripts.
  • Documents & Transcripts: The resulting text (or document text) is then forwarded to Anthropic Claude to cleanly summarize the massive multi-page payloads into actionable paragraphs.

3. The Unified Multimodal Embedding (Titan) Finally, the "holy grail" of our Ingest architecture occurs. We take the Proxy Image + the generated Text Summary/Transcript and fuse them together within the Amazon Titan Multimodal Framework. This calculates a massive 1024-dimensional vector that represents both the visual pixels and the profound textual context identically in mathematical space.

Phase 4: Persistence

The final dimensions, EXIF data, GPS coordinates, textual transcripts, and multimodal vectors are pushed simultaneously to Neon (PostgreSQL) for user-metadata management, and AWS OpenSearch for blistering conversational Search Indexing.

If cache warming is requested via the API, the worker dispatches HTTP requests back through our CDN endpoints to guarantee sub-50ms cold-starts.