Fiori Test Automation System - Browser Extension

🎯 The First Step to Killing Fiori: Capture Everything, Then Eliminate the UI

"The ideal system is one that doesn't exist but its function is performed." - TRIZ

🚀 The Vision: From 23 Clicks to Zero

Article:

I killed Fiori (ext)
I killed Fiori

This extension is the first phase in eliminating the Fiori UI layer entirely. By capturing every interaction, UI5 context, and OData request, we're building the foundation for direct AI-to-OData communication.

Current Reality: User → Clicks → UI → OData → Backend
Our Future: User → Natural Language → OData → Backend

The extension proves that 80% of Fiori interactions are pure UI overhead. Once we capture the patterns, we can bypass the UI entirely.

✅ Current Status: Fully Functional

This extension is production-ready with all major issues resolved:

✅ Service worker installation and manifest compliance
✅ Event capture with proper synchronization
✅ Screenshot system with semantic filenames and rate limiting
✅ Bundle export (JSON + Markdown + Screenshots in ZIP)
✅ Clean semantic filename format across all exports
✅ Enhanced SAPUI5/Fiori app detection
✅ Request filtering for business-focused captures
✅ Audio recording with voice narration support
✅ Single source of truth for filename generation

🎯 Why This Extension Exists

After analyzing hundreds of Fiori sessions, we discovered:

80% waste: Most interactions are just translating clicks to OData calls
10 clicks, 27 seconds for simple updates that should take 1 second
23 @UI annotations just to display fields that already exist in the backend

This extension captures the patterns needed to build the ultimate solution: an MCP-OData bridge that lets AI talk directly to SAP backends, eliminating the UI entirely.

The Master Plan

Phase 1 (This Extension): Capture all interactions, understand the patterns
Phase 2 (Coming Soon): Build parametrized automations from captured sessions
Phase 3 (The Kill Shot): MCP-OData Bridge - Natural language directly to backend

🚀 Quick Start

Installation

Method 1: Load from Folder (Chrome & Edge)

Chrome:

Download or clone this repository to a local folder
Open Chrome and navigate to chrome://extensions/
Enable "Developer mode" toggle in the top right corner
Click "Load unpacked" button
Browse and select the fiori_automator folder
The extension will appear in your extensions list
Click the puzzle piece icon in toolbar and pin the extension

Edge:

Download or clone this repository to a local folder
Open Edge and navigate to edge://extensions/
Enable "Developer mode" toggle in the left sidebar
Click "Load unpacked" button
Browse and select the fiori_automator folder
The extension will appear in your extensions list
Click the extensions icon in toolbar and pin the extension

Method 2: Pack as CRX (For Distribution)

Create CRX file:

In Chrome, go to chrome://extensions/
Enable "Developer mode"
Click "Pack extension" button
Browse to the extension directory
Leave private key field empty (first time) or use existing key
Click "Pack Extension" to generate .crx file

Install CRX file:

Chrome: Drag and drop the .crx file onto chrome://extensions/ page
Edge: Drag and drop the .crx file onto edge://extensions/ page
Note: Recent versions may require additional permissions for CRX installation

Method 3: From ZIP Archive

Download the extension as a ZIP file
Extract the ZIP to a folder on your computer
Follow Method 1 instructions to load the unpacked extension

Method 4: Firefox (Requires Manifest V2 Conversion)

This extension uses Chrome's Manifest V3 format. To use in Firefox:

Convert manifest.json to Manifest V2 format
Load as temporary extension via about:debugging
Note: Some features may require adaptation for Firefox APIs

Browser Compatibility

Browser	Support	Notes
Chrome	✅ Full	Version 88+ required for Manifest V3
Edge	✅ Full	Version 88+ required for Manifest V3
Brave	✅ Full	Same as Chrome
Opera	✅ Full	Same as Chrome
Firefox	⚠️ Partial	Requires Manifest V2 conversion
Safari	❌ No	Different extension format

Post-Installation Steps

Verify Installation: Check that the extension icon appears in your toolbar
Grant Permissions: The extension may request permissions when first used
Pin Extension: For easy access, pin the extension to your toolbar
Test Recording: Navigate to a Fiori app and test the recording feature

Basic Usage

Navigate to any SAP Fiori application
Click the extension icon and select "Start Recording"
Interact with the Fiori app (clicks, forms, navigation)
Stop recording when finished
Export your session as JSON or Bundle (recommended)

What to Capture (For Maximum Impact)

Focus on capturing sessions that show:

Repetitive tasks you do daily (these are prime automation candidates)
Multi-step processes that should be simple (expose the UI complexity)
Data updates that require navigating through multiple screens
Search and filter operations that could be natural language queries
Any workflow where you think "there must be a better way"

Each captured session helps map the OData services behind the UI curtain.

📋 Features

Core Capabilities (Building Blocks for UI Elimination)

Universal Capture: Records all user interactions to understand how humans navigate the UI maze
OData Correlation: The key insight - every UI action eventually triggers OData calls (this is what we'll use directly)
SAPUI5 Context: Extracts the business logic hidden behind @UI annotations and control hierarchies
Screenshot Capture: Documents the visual overhead we're about to eliminate
Real-time Analysis: Shows exactly how much time is wasted on UI navigation
Session Management: Build a library of patterns for direct OData access

Advanced Features

Network Request Interception: Captures all HTTP requests with intelligent filtering
Business Process Flow: Clean mermaid diagrams showing meaningful operations only
Visual Documentation: Screenshots automatically embedded in markdown exports
Request Filtering: Smart filtering of JS/CSS/assets to focus on business operations
Semantic Filenames: Clean, sortable filenames for all exports
Audio Recording: Optional voice narration during sessions
Bundle Export: Single ZIP containing JSON, Markdown, and all screenshots
Enhanced App Detection: Uses UI5 models, About Dialog, and Fiori Apps Library integration

Export Options

1. Save JSON - Session Data Only

File: fs-YYYY-MM-DD-HHMM-app-name.json
Complete session data with metadata
Application intelligence summary
Screenshot references
OData correlation analysis

2. Save Bundle - Complete Package (Recommended)

File: fs-YYYY-MM-DD-HHMM-app-name.zip
Contains: JSON + Markdown + All Screenshots
Ready for sharing and documentation
All files use consistent naming scheme

3. Export Audio (if recorded)

File: fs-YYYY-MM-DD-HHMM-app-name-audio.webm
Voice narration recorded during session
Normal quality, opus codec

🛠️ Technical Details

Architecture

Manifest V3 Chrome extension with proper service worker
Content Script: Captures DOM events and UI5 context
Background Script: Manages sessions, screenshots, and exports
Injected Scripts: Deep access to page context and UI5 internals

Filename Format

All exports use a consistent semantic naming pattern:

fs-YYYY-MM-DD-HHMM-app-name.{extension}

Examples:

fs-2025-06-10-1430-manage-detection-methods.json
fs-2025-06-10-1430-manage-detection-methods.zip
fs-2025-06-10-1430-manage-detection-methods-0001-click.png

Screenshot System

Rate Limited: Maximum 2 screenshots per second
Semantic IDs: Sequential numbering (0001, 0002, etc.)
Event Correlation: Each screenshot linked to specific user action
Quality: Full viewport capture with element highlighting

SAPUI5 Detection Methods

UI5 Model Data (AppInfo, SysInfo JSONModels)
About Dialog pattern detection
Direct API access to sap.ui.getCore()
DOM Heuristics for Fiori patterns
URL Analysis for app component patterns
Fiori Apps Library correlation

🧪 Testing & Debugging

Installation Verification

Check extension loads without errors in chrome://extensions/
Verify all files present: manifest.json, background.js, content.js, etc.
Test popup interface opens correctly

Recording Test

Navigate to any Fiori application
Open browser DevTools Console

Start recording and look for debug messages:

[Fiori] Recording started - isRecording = true
[Fiori] Click captured and sent to background

Export Test

Record a short session (2-3 clicks)
Stop recording
Open Sessions page and test "Save Bundle"
Verify ZIP contains JSON + MD + PNG files

Debug Mode

Enable debug logging:

Open browser console
Run: localStorage.setItem('fiori-debug', 'true')
Reload page - you'll see detailed logging

📁 File Structure

fiori_automator/
├── manifest.json           # Extension configuration
├── popup.html/js/css       # Extension popup interface
├── background.js           # Service worker (main logic)
├── content.js             # Page interaction capture
├── sessions.html/js/css    # Session management UI
├── injected.js            # Deep page context access
├── ui5-detector.js        # SAPUI5 detection engine
├── zip-utils.js           # ZIP file creation utility
├── icons/                 # Extension icons (16,32,48,128px)
└── README.md              # This file

🐛 Troubleshooting

Common Issues

Extension won't install

Check Chrome version (requires Manifest V3 support)
Ensure all files present in directory
Check for syntax errors in manifest.json

No events captured

Verify content script loaded (check console)
Check if page has CSP restrictions
Enable debug mode for detailed logging

Screenshots not working

Check chrome.tabs permission in manifest
Verify tab is active and visible
Check rate limiting (max 2/second)

Bundle export empty

Check if session has events
Verify screenshots were captured during recording
Enable debug logging to see screenshot collection

Support

For issues or questions:

Enable debug mode and check console logs
Check the Sessions page for detailed session data
Verify all extension files are present and loaded

🔧 Development

Building from Source

Clone repository
No build step required - load directly as unpacked extension
Modify files and reload extension to test changes

Key Files to Modify

background.js - Core logic, session management, exports
content.js - Event capture, UI5 detection
popup.js - User interface logic
sessions.js - Session viewer and export interface

Testing Changes

Make modifications
Go to chrome://extensions/
Click reload button for the extension
Test functionality in Fiori applications

🔮 The Endgame: Life After Fiori

When this project succeeds, here's what changes:

Before (Fiori Way)

User: "Update purchase order 4500123456 delivery date to next Friday"
Steps: Open Fiori → Find App → Click Tile → Wait → Search PO → Click Edit → 
       Change Date → Save → Confirm → Close
Time: 4 minutes, 23 clicks

After (Direct OData)

User: "Update purchase order 4500123456 delivery date to next Friday"
AI: Done. Delivery date updated to July 3, 2025.
Time: 2 seconds, 0 clicks

The extension you're using today is capturing the data to make this future possible. Every session recorded brings us closer to eliminating the 80% waste in enterprise software.

Version: 1.1 (The UI Killer)
Last Updated: June 2025
Chrome Manifest: V3 Compatible
Next Phase: MCP-OData Bridge (Coming Soon)

oisee/fiori_automator