WiseSpend: The AI Receipt Scanner Killing Budget Apps

Stop manually tracking expenses. Stop ignoring your spending. Stop wondering where your paycheck disappeared.

Here's the brutal truth: 80% of people abandon budgeting apps within 90 days. Not because they're lazy. Because traditional finance tools treat symptoms, not the disease. They show you what you spent, never what it cost your future self.

That $6.50 latte? Your budgeting app shrugs. But what if that same purchase meant 15 minutes of retirement gone? What if it pushed your house deposit two weeks further away?

This is where WiseSpend enters—and why developers and financially frustrated users are paying attention. Built for the Comet AI Agents Hackathon, WiseSpend isn't another expense tracker. It's an agentic, AI-first financial coach that transforms receipt images into Socratic coaching sessions, goal-impact calculations, and behavior-changing insights using Google's Gemini 3 multimodal AI.

The secret weapon? Every reasoning step is fully traced and optimized through Opik, Comet's observability platform. This isn't prototype-grade code. This is production-ready agent architecture with evaluation-driven improvement baked in.

Ready to understand why your next 100 purchases could be radically different? Let's dissect what makes WiseSpend genuinely disruptive.

What Is WiseSpend?

WiseSpend is an open-source, full-stack financial intelligence platform created by developer olathedev for the Comet "Commit to Change" AI Agents Hackathon. It combines multimodal AI vision, agentic reasoning chains, and production observability to solve a problem that has stumped fintech for decades: making people actually care about their spending before the money leaves their account.

The repository lives at github.com/olathedev/wise-spend, and it's gaining traction for one simple reason—it works differently.

Why It's Trending Now

Three converging forces make WiseSpend's timing perfect:

1. Gemini 3's multimodal leap: Google's latest model can parse unstructured receipt images with near-human accuracy, distinguishing intent ("necessity vs. luxury") rather than performing dumb OCR.

2. Agentic AI maturity: 2024-2025 marks the shift from chatbots to autonomous agents. WiseSpend's reasoning chain—receipt scan → image analysis → item extraction → context analysis → goal impact → Socratic coaching—is a textbook agent architecture.

3. Observability as competitive advantage: With Opik integration, WiseSpend isn't a black box. Every AI decision is traceable, evaluable, and improvable. This matters when you're giving financial advice.

The project demonstrates true agent behavior: autonomous reasoning, tool use (vision API, calculations), memory (user history, goals), and continuous improvement through evaluation metrics.

Key Features That Separate WiseSpend from the Herd

📸 Multimodal Receipt Scanning with Intent Recognition

Most receipt scanners perform OCR and call it a day. WiseSpend leverages Gemini 3 Vision to understand context. It doesn't just read "Coffee $6.50"—it recognizes this as discretionary spending, compares it to your historical patterns, and flags anomalies.

Technical depth: The system preprocesses images, runs multimodal analysis with structured prompts, then extracts categorized line items. Every step creates nested Opik spans with full token usage tracking.

🤔 Socratic Financial Coaching (Not Nagging)

The killer UX insight: people ignore commands but respond to questions they answer themselves.

Instead of "You spent too much on coffee," WiseSpend asks: "Is that $6.50 latte worth 15 minutes of your future retirement?" This technique—derived from the Socratic method—drives behavioral change through self-discovered insight rather than external judgment.

The coaching tone is deliberately tuned via Opik Agent Optimizer to be "supportive but firm"—empathetic enough to maintain engagement, direct enough to provoke action.

🎯 Real-Time Goal Impact Calculation

This is where WiseSpend becomes genuinely addictive. Every purchase is translated into temporal goal impact:

• "$100 purchase = 2 weeks further from house deposit"
• "$6.50 latte = 15 minutes of retirement fund"

The math runs against your actual financial goals, monthly income, and historical savings rate. It's not generic advice—it's your trade-off, calculated in real-time.

🛡️ Safety & Moderation via Opik Evaluation

Financial advice is regulated for good reason. WiseSpend implements ModerationMetric (blocks high-risk investment suggestions), HallucinationMetric (validates balance data accuracy), and LLM-as-a-Judge evaluation. Over 50 edge-case financial scenarios run through regression testing.

📊 Financial Dashboard with Wise Score

A proprietary Wise Score (0-100) measures financial resilience across emergency funds, spending patterns, goal progress, and behavioral consistency. It's gamified without being infantilizing.

🎓 Gamified Financial Learning (Grow Page)

12 financial literacy topics—from 50/30/20 budgeting to compound growth, inflation dynamics, and tax-advantaged accounts—delivered through AI-generated, personalized quizzes. Click a topic, Gemini curates 5 multiple-choice questions, you get scored with explanations. New quiz anytime.

💼 Investment Opportunity Curation

Powered by Alpha Vantage real-time market data, the Grow page surfaces curated ETFs (SPY, VTI, VEA, BND, VNQ) and major stocks with risk classifications, minimum investment thresholds, and educational descriptions. Filtered by type, sorted by performance, guarded by safety metrics.

Use Cases: Where WiseSpend Actually Wins

Use Case 1: The Conscious Spender (Breaking Impulse Habits)

Problem: You make $85K, save nothing, and feel vaguely anxious about money. Budgeting apps make you feel worse.

WiseSpend solution: Scan your first coffee receipt. See "15 minutes of retirement." The Socratic question lands differently than a red bar chart. You start scanning before purchases, not after. The agent recognizes your Friday stress-spending pattern. You catch yourself mid-impulse. Behavior changes through awareness, not shame.

Use Case 2: The Goal-Oriented Saver (House Deposit Acceleration)

Problem: Vague goal ("save for house"), no connection between daily decisions and distant outcome.

WiseSpend solution: Set $10,000 house deposit goal. Every scanned receipt shows temporal impact. "$80 dinner = 3 days further from goal." The visualization creates immediate feedback loops that abstract savings targets never could. You start choosing differently because the trade-off is visceral, not theoretical.

Use Case 3: The Financially Curious (Knowledge Building)

Problem: You want to understand investing but don't trust random YouTube advice and find traditional finance courses boring.

WiseSpend solution: Open Grow page → Knowledge tab → click "Compound Growth" → generate AI-curated quiz. Answer 5 questions, get explanations, retake with new questions. Side AI panel answers follow-ups: "What's VTI vs. VOO?" All traced, all evaluated for accuracy. Learning becomes interactive, personalized, and verifiable.

Use Case 4: The Cautious Investor (First Steps)

Problem: You have $500 to invest, terrified of losing it, overwhelmed by options.

WiseSpend solution: Grow page → Investment Suggestions tab. Filter to low-risk ETFs. See VTI: "Total stock market, $1 minimum, medium risk, 7.2% historical return." Read description. Ask AI panel: "What happens if market drops 20%?" Get contextual, moderated response. Informed entry with guardrails.

Step-by-Step Installation & Setup Guide

WiseSpend is a full-stack TypeScript application with separate frontend and backend. Here's how to run it locally.

Prerequisites

• Node.js 18+ and npm
• MongoDB (local instance or MongoDB Atlas cluster)
• Google Cloud account (for Gemini API key)
• Comet account (for Opik observability)
• Cloudinary account (for receipt image storage)
• Alpha Vantage API key (for investment data)

Backend Setup

# Clone the repository
git clone https://github.com/olathedev/wise-spend.git
cd wise-spend/backend

# Install dependencies
npm install

# Configure environment variables
cp .env.example .env

Edit .env with your actual credentials:

# Required: Google Gemini 3 API access
GEMINI_API_KEY=your_google_cloud_api_key

# Required: Opik observability
OPIK_API_KEY=your_comet_opik_key
OPIK_PROJECT_NAME=WiseSpend-Evals
OPIK_WORKSPACE=your_comet_workspace

# Required: Database
MONGODB_URI=mongodb+srv://user:pass@cluster.mongodb.net/wisespend

# Required: Image storage
CLOUDINARY_CLOUD_NAME=your_cloud_name
CLOUDINARY_API_KEY=your_api_key
CLOUDINARY_API_SECRET=your_api_secret

# Optional: Investment data
ALPHA_VANTAGE_API_KEY=your_alpha_vantage_key

# Start development server
npm run dev

Backend runs at http://localhost:8000

Frontend Setup

cd ../frontend

# Install dependencies
npm install

# Configure environment
cp .env.example .env

Edit .env:

# Required: API connection
NEXT_PUBLIC_API_URL=http://localhost:8000/api/v1

# Required: NextAuth authentication
GOOGLE_CLIENT_ID=your_google_oauth_client_id
GOOGLE_CLIENT_SECRET=your_google_oauth_client_secret
NEXTAUTH_SECRET=your_random_secret  # Generate: openssl rand -base64 32
NEXTAUTH_URL=http://localhost:3000

# Start development server
npm run dev

Frontend runs at http://localhost:3000

Verify Opik Integration

After making API calls, visit https://www.comet.com/opik, navigate to your workspace, and open your configured project. You should see traces for chat-gemini-3 and analyze-receipt-image with nested spans showing the full reasoning chain.

REAL Code Examples from the Repository

Let's examine actual implementation patterns from WiseSpend's codebase, with detailed explanations of how the agentic architecture works.

Example 1: Receipt Analysis with Full Opik Tracing

This pattern from GoogleGenAIService.ts demonstrates how every AI operation is wrapped in observable, evaluable traces:

// backend/src/infrastructure/services/GoogleGenAIService.ts
// Core service handling all Gemini 3 interactions with full observability

import { GoogleGenerativeAI } from "@google/generative-ai";
import { OpikService } from "./OpikService";

export class GoogleGenAIService {
  private genAI: GoogleGenerativeAI;
  private opik: OpikService;

  constructor() {
    // Initialize with API key from environment
    this.genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY!);
    this.opik = new OpikService();
  }

  async analyzeReceiptImage(imageBuffer: Buffer, userContext: UserContext) {
    // Create parent trace for entire receipt analysis operation
    const trace = this.opik.createTrace("analyze-receipt-image", {
      input: { imageSize: imageBuffer.length, userId: userContext.id }
    });

    try {
      // SPAN 1: Preprocess image (resize, validate format)
      const preprocessSpan = trace.span("preprocess-receipt-image", "general");
      const processedImage = await this.preprocessImage(imageBuffer);
      preprocessSpan.end({ output: { format: "jpeg", dimensions: processedImage.dimensions } });

      // SPAN 2: Multimodal AI analysis with Gemini 3 Vision
      const analysisSpan = trace.span("gemini-multimodal-analysis", "llm");
      
      const model = this.genAI.getGenerativeModel({
        model: "gemini-1.5-flash",  // Gemini 3 multimodal model
        generationConfig: {
          temperature: 0.2,  // Low temperature for consistent extraction
          maxOutputTokens: 2048
        }
      });

      // Structured prompt for consistent JSON output
      const prompt = `
        Analyze this receipt image. Extract:
        1. Merchant name and date
        2. Line items with prices
        3. Categorize each item: necessity, luxury, or investment
        4. Identify any recurring purchase patterns
        
        User context: Monthly income $${userContext.monthlyIncome}, 
        Goal: ${userContext.primaryGoal?.description || "none"}
        
        Respond in valid JSON format.
      `;

      const result = await model.generateContent([
        { text: prompt },
        { inlineData: { data: processedImage.base64, mimeType: "image/jpeg" } }
      ]);

      const analysis = JSON.parse(result.response.text());
      
      // Log token usage for cost observability
      analysisSpan.end({
        input: { model: "gemini-1.5-flash", prompt, temperature: 0.2 },
        output: { analysis, fullAnalysis: result.response.text() },
        metadata: {
          usage: result.response.usageMetadata,  // promptTokens, completionTokens, totalTokens
          provider: "google-genai"
        }
      });

      // SPAN 3: Calculate personalized goal impact
      const impactSpan = trace.span("calculate-goal-impact", "general");
      const goalImpact = this.calculateGoalImpact(analysis, userContext);
      impactSpan.end({ input: { analysis }, output: { goalImpact } });

      // Complete trace with final output
      trace.end({
        output: { analysis, goalImpact },
        metadata: { processingTimeMs: Date.now() - trace.startTime }
      });

      return { analysis, goalImpact };

    } catch (error) {
      // All errors captured in trace with full context for debugging
      trace.end({ error: { type: error.name, message: error.message, stack: error.stack } });
      throw error;
    }
  }

  private calculateGoalImpact(analysis: ReceiptAnalysis, userContext: UserContext): GoalImpact {
    const totalAmount = analysis.items.reduce((sum, item) => sum + item.price, 0);
    
    if (!userContext.primaryGoal) {
      return { message: "Set a goal to see impact calculations" };
    }

    // Calculate temporal impact: how much does this delay the goal?
    const monthlySavingsNeeded = userContext.primaryGoal.targetAmount / 
      this.monthsUntil(userContext.primaryGoal.deadline);
    const daysDelayed = (totalAmount / monthlySavingsNeeded) * 30;

    return {
      amount: totalAmount,
      daysDelayed: Math.round(daysDelayed * 10) / 10,
      contextualMessage: this.generateImpactMessage(totalAmount, daysDelayed, analysis, userContext)
    };
  }

  private generateImpactMessage(amount: number, daysDelayed: number, 
    analysis: ReceiptAnalysis, userContext: UserContext): string {
    // Socratic question generation based on purchase context
    const luxuryItems = analysis.items.filter(i => i.category === "luxury");
    
    if (luxuryItems.length > 0 && userContext.primaryGoal?.type === "house_deposit") {
      return `That $${amount.toFixed(2)} purchase = ${daysDelayed} days further from your house deposit. Is ${luxuryItems[0].name} worth delaying your goal?`;
    }
    
    return `That $${amount.toFixed(2)} purchase = ${daysDelayed} days further from ${userContext.primaryGoal?.description || "your goal"}.`;
  }
}

What's happening here: This isn't a simple API call. It's a three-span reasoning chain where each step is independently observable. The preprocessing span catches image issues. The LLM span tracks token costs and model behavior. The goal impact span isolates business logic. If something breaks, you know exactly where. If costs spike, you see which model calls grew.

Example 2: Graceful Shutdown with Trace Flushing

From index.ts, production-grade process management:

// backend/src/index.ts
// Ensures no traces are lost during deployment restarts or crashes

import { OpikService } from "./infrastructure/services/OpikService";

const opikService = new OpikService();

// Graceful shutdown handlers
const shutdownSignals: NodeJS.Signals[] = ["SIGTERM", "SIGINT"];

shutdownSignals.forEach((signal) => {
  process.on(signal, async () => {
    console.log(`Received ${signal}, starting graceful shutdown...`);
    
    // CRITICAL: Flush all pending traces to Opik before exit
    // Without this, traces in batch buffer are lost
    await opikService.flush();
    
    console.log("All traces flushed to Opik. Exiting.");
    process.exit(0);
  });
});

Why this matters: In production, containers restart constantly. Without explicit flushing, you lose observability data exactly when you need it most—during incidents. This pattern ensures 100% trace capture.

Example 3: Quiz Generation with LLM-as-a-Judge Evaluation

From GenerateQuizzesUseCase.ts and QuizEvaluator.ts:

// backend/src/application/use-cases/GenerateQuizzesUseCase.ts
// AI-curated financial literacy quizzes with full evaluation

export class GenerateQuizzesUseCase {
  private quizAgent: QuizCuratorAgent;
  private quizEvaluator: QuizEvaluator;
  private opik: OpikService;

  async execute(concept: FinancialConcept, userProfile: UserProfile) {
    const trace = this.opik.createTrace("generate-personalized-quizzes", {
      input: { concept: concept.id, userLevel: userProfile.financialLiteracyLevel }
    });

    // Generate quiz using specialized agent
    const quizSpan = trace.span("generate-quiz-for-concept", "general");
    const quiz = await this.quizAgent.generate(concept, userProfile);
    quizSpan.end({ input: { concept }, output: { questionCount: quiz.questions.length } });

    // Evaluate quiz quality if evaluation mode enabled
    if (process.env.EVALUATE_QUIZZES === "true") {
      const evalSpan = trace.span("quiz-llm-judge-evaluation", "general");
      const evaluation = await this.quizEvaluator.evaluate(quiz, concept);
      evalSpan.end({ 
        output: { 
          accuracyScore: evaluation.accuracyScore,
          difficultyMatch: evaluation.difficultyMatch,
          passed: evaluation.passed 
        } 
      });

      // Auto-reject poor quality quizzes
      if (!evaluation.passed) {
        trace.end({ output: { quiz, evaluation, status: "rejected" } });
        throw new Error(`Quiz failed evaluation: ${evaluation.failureReason}`);
      }
    }

    trace.end({ output: { quiz, status: "approved" } });
    return quiz;
  }
}

// backend/src/infrastructure/services/QuizEvaluator.ts
// LLM-as-a-Judge: uses separate model to evaluate quiz quality

export class QuizEvaluator {
  async evaluate(quiz: Quiz, concept: FinancialConcept): Promise<QuizEvaluation> {
    const judgePrompt = `
      You are a financial education expert evaluating a quiz about "${concept.name}".
      
      Evaluate this quiz on:
      1. FACTUAL ACCURACY: Are all answers correct? (0-100)
      2. DIFFICULTY MATCH: Appropriate for stated level? (0-100)
      3. CONCEPT ALIGNMENT: Does it test the actual concept? (0-100)
      4. DISTRACTOR QUALITY: Are wrong answers plausible? (0-100)
      
      Quiz: ${JSON.stringify(quiz.questions)}
      
      Respond with JSON: { "accuracyScore": number, "difficultyMatch": number, 
        "conceptAlignment": number, "distractorQuality": number, 
        "passed": boolean, "failureReason": string|null }
    `;

    // Use separate model instance for evaluation to avoid contamination
    const judgeResult = await this.judgeModel.generateContent(judgePrompt);
    return JSON.parse(judgeResult.response.text());
  }
}

The evaluation pattern: This is agentic quality control. The system doesn't just generate content—it validates it against expert criteria using a separate "judge" model. Failed quizzes are rejected and regenerated. This is how you build trustworthy AI systems at scale.

Advanced Usage & Best Practices

Optimizing for Production

1. Batch receipt processing: The analyze-receipt-image trace supports bulk operations. Queue multiple receipts and process with shared user context to reduce API costs.

2. Custom evaluation metrics: Extend Opik's built-in evaluators with domain-specific checks. Add a SpendingVelocityMetric that flags users whose scan frequency drops below weekly—indicating abandonment risk.

3. Prompt A/B testing via Opik Agent Optimizer: Run parallel prompt variants for Socratic questions. Measure which phrasing drives higher goal-completion rates. The "supportive but firm" baseline is just the starting point.

4. Cost governance: Token usage is tracked per-trace. Set alerts when totalTokens exceeds thresholds. Gemini 3 Flash is cost-effective, but unbounded image uploads can surprise you.

Security Considerations

• Receipt images contain PII. Cloudinary uploads should use signed URLs with expiration.
• Financial data in MongoDB requires field-level encryption for account balances.
• The ModerationMetric is a safety net, not a guarantee. Layer additional checks for investment advice endpoints.

Comparison with Alternatives

Why choose WiseSpend? If you want transparent, improvable AI that actually changes spending behavior—not just tracks it. If you're a developer who needs observability for financial AI compliance. If you believe your data shouldn't be trapped in someone else's SaaS.

FAQ

What makes WiseSpend different from other receipt scanners?

WiseSpend uses multimodal AI to understand intent, not just text. Gemini 3 Vision distinguishes "necessity" from "luxury," then calculates personalized goal impact. Most scanners perform OCR and stop. WiseSpend starts there.

Is my financial data safe?

You control the infrastructure. WiseSpend is open-source and self-hostable. Your data lives in your MongoDB instance, your Cloudinary account, your Opik workspace. No third-party SaaS has blanket access.

What does "agentic" actually mean here?

Autonomous reasoning chains. The system doesn't just respond to prompts—it plans steps (preprocess → analyze → calculate → coach), uses tools (vision API, calculators), maintains memory (your goals, history), and improves through evaluation. This is agent architecture, not chatbot wrapping.

How accurate is the goal impact calculation?

As accurate as your inputs. It uses your stated monthly income, savings rate, and goal deadline. The math is deterministic; the behavioral insight comes from making trade-offs visceral rather than abstract.

Can I use WiseSpend without Opik?

Technically yes, practically no. The code has Opik dependencies for tracing. You'd need to mock or replace OpikService. For production use, Opik's free tier covers substantial volume, and the observability is worth the integration.

What's required to run the investment features?

Alpha Vantage API key (free tier available). The Grow page's investment tab fetches real-time prices for curated ETFs and stocks. Without it, the knowledge quizzes still function fully.

How do I contribute or customize the coaching tone?

Fork and modify ChatUseCase.ts and system prompts. The Opik Agent Optimizer can then A/B test your variants. The project welcomes PRs for additional financial literacy topics and localization.

Conclusion: Why WiseSpend Matters

WiseSpend isn't perfect. It's a hackathon project with room to grow—more bank integrations, richer visualizations, deeper investment analysis. But its architectural decisions are prescient.

The combination of multimodal AI for unstructured data, agentic reasoning for complex decisions, and production observability for trustworthiness represents where fintech is heading. Not chatbots that answer questions, but agents that change behavior.

The Socratic coaching approach—asking rather than telling—deserves particular attention. In a field saturated with judgmental budgeting apps, curiosity is the underrated growth hack. People engage with questions they answer themselves. They ignore commands from algorithms.

For developers, WiseSpend is a reference implementation for building observable, evaluable AI agents. The Opik integration patterns—nested spans, LLM-as-a-Judge, graceful shutdown flushing—are directly transferable to any agentic system.

For users, it's a genuinely different financial tool—one that respects your intelligence while nudging your behavior.

The receipt scan isn't about that ONE purchase. It's about building awareness so your NEXT 100 purchases are different.

Ready to transform financial fog into actionable resilience? Clone the repository, scan your first receipt, and watch your relationship with money change.

👉 Star WiseSpend on GitHub — and start building financial awareness that actually sticks.

Built with 💰 for the Comet AI Agents Hackathon. Powered by Google Gemini 3 and Opik observability.