Creating an AI Image Generation Pipeline for Creative Analytics

Context

The client is building an AI-powered creative analytics solution that helps users analyze their Ads performance across various platforms, such as Google or Meta. I joined them to create a new core module that allows end users to generate creative images using AI, based on their own content and brand details.

The product required adding a new create flow based on image creation, spanning user-facing UX, credit-based billing, all the way to async jobs handling the actual AI generation and brand management.

My Approach

After having extensive discussions with various stakeholders as well as the technical team, I came up with an execution plan where I did the following:

  • Designed various Postgres migrations to add the necessary tables and stored procedures to the existing Supabase database to ensure schema integrity and efficient resource usage on complex operations and queries.
  • Integrated Google’s AI Gemini models to handle complex workflows, custom business rules, and QA based checks to ensure the quality of the AI-generated images.
  • Wired the backend service to a NextJS and Tailwind based frontend with a seamless UI flow for users to navigate the product module from managing their data and assets to downloading their generated images.
  • Designed a queue-based processing pipeline using Redis and BullMQ to enforce per-user concurrency limits and isolate workloads, preventing heavy users from degrading system-wide performance.
  • Built production readiness into the system from day one by making the app run via Docker and pm2 to ensure reproducibility, scaling, and ease of deployment, integrating logging to help with observability, as well as providing developers with tooling to make local development a breeze through aspects such as live reloading and database seeding.
  • Collaborated with a team of skilled developers as well as stakeholders to deliver the features as required within weeks before the set deadline, using various tools and following the agile methodology.
  • Authored parts of the internal documentation aimed at helping other developers navigate the product roadmap as well as the specification and reasoning behind all decisions taken during development.

Challenges

  • Requirements evolved during development, which was addressed by designing a flexible data model and decoupled pipeline components to minimise rework when the scope shifted.
  • AI outputs were inconsistent and occasionally unusable, which was mitigated through structured prompts and a validation and retry layer before any image generation job was marked as complete.
  • Clear async communication for a remote team across time zones was handled by having well-scoped PRs and keeping the internal specification document aligned despite the distributed setup.

Outcome

  • A robust database schema so that new image generation workflows, billing models, or brand asset structures can be added without reworking the core data model.
  • A service designed to scale through using BullMQ’s group concurrency, CDN-delivered media, and SSE-based progress events to ensure the system handles many concurrent users without requiring expensive infrastructure or complex workarounds.
  • The full module, schema, microservice, AI pipeline, and UI were delivered within the agreed timeline.

Learnings

Generative AI is a fast paced and evolving field with new models, techniques, and tools emerging frequently, and using it to build useful software as well as integrating it to existing workflows and systems that we use and rely on daily requires deliberate planning and setting of guard rails to ensure the reliability of the system, the safety of the end user, as well as making sure it is an economically feasible solution to a real world and pressing problem.

Tech Stack

  • Next.js , Node.js , Supabase (PostgreSQL) , Redis , BullMQ Pro , Tailwind CSS , Docker , PM2 , Bunny CDN.