Fate/Grand Automata (FGA) is an open-source Android automation app for the mobile game Fate/Grand Order (FGO). It automates repetitive farming battles by watching the screen and tapping things just like a human player would — no root access required. If you’re an Android developer curious about how to build something like this, read on.

What does it do?

FGA watches the game screen using Android’s MediaProjection API for screenshots and the Accessibility Service API for performing taps and swipes. It uses OpenCV for image recognition to figure out what’s on the screen at any given moment and decide what to do next.

The app does not modify the game or inject code into it. It operates entirely by observing pixels and issuing gestures, making it fundamentally different from traditional cheat tools.

Project Structure

The repository is split into four Gradle modules:

• app/ — The Android application: UI (Jetpack Compose), the Accessibility Service, the MediaProjection integration, and Dagger Hilt dependency injection wiring.
• libautomata/ — A platform-agnostic automation library. Defines the core primitives (coordinates, regions, image patterns) and the scripting API.
• scripts/ — FGO-specific game logic. Contains all the scripts (battle, lottery, gacha, etc.) built on top of libautomata.
• prefs/ — User preferences shared across modules.

This separation is intentional: libautomata has no knowledge of FGO. In theory you could reuse it for any Android automation task. The FGO-specific knowledge lives entirely in scripts/.

Core Primitives: Location and Region

Everything in FGA operates on 1440p virtual coordinates, regardless of the device’s actual resolution. The two fundamental types are Location (a 2D point) and Region (a rectangle).

data class Location(val x: Int = 0, val y: Int = 0) : Comparable<Location> {
    operator fun plus(other: Location) = Location(x + other.x, y + other.y)
    operator fun minus(other: Location) = Location(x - other.x, y - other.y)
    operator fun times(scale: Double) = Location((x * scale).roundToInt(), (y * scale).roundToInt())
}

data class Region(val x: Int, val y: Int, val width: Int, val height: Int) {
    val center get() = Location(x + width / 2, y + height / 2)
    val right get() = x + width
    val bottom get() = y + height
}

Coordinates are always in 1440p, but images are stored and matched in 720p (because screenshots are scaled down to 720p before comparison). The Transformer class handles scaling between these two coordinate spaces.

For screen elements that sit at a fixed offset from the center of the screen (like most FGO menus), helper methods such as xFromCenter() and xFromRight() are used so that the layout adapts across different screen aspect ratios (16:9, 18:9, etc.).

The AutomataApi Interface

The scripting layer communicates with the Android platform through a single interface, AutomataApi. This is the key to why libautomata stays platform-agnostic.

interface AutomataApi {
    fun Region.getPattern(tag: String = ""): Pattern
    fun <T> useSameSnapIn(block: () -> T): T
    fun Duration.wait(applyMultiplier: Boolean = true)
    fun Location.click(times: Int = 1)
    fun Region.exists(image: Pattern, timeout: Duration = Duration.ZERO, similarity: Double? = null): Boolean
    fun Region.waitVanish(image: Pattern, timeout: Duration, similarity: Double? = null): Boolean
    fun Region.findAll(pattern: Pattern, similarity: Double? = null): Sequence<Match>
    fun Region.isWhite(): Boolean
    fun Region.isBlack(): Boolean
    fun Region.detectText(outlinedText: Boolean = false): String
    // ... and more
}

All functions are defined as extension functions on Region or Location. This design (which I described in detail in a previous post) makes scripts read like natural language:

// Check if the battle screen is showing
if (images[Images.BattleScreen] in locations.battle.screenCheckRegion) {
    locations.battle.attackClick.click()
}

The concrete implementation, StandardAutomataApi, wires up the ScreenshotManager, Clicker, ImageMatcher, and other platform services via Dagger Hilt constructor injection.

How Screenshots Work

FGA uses Android’s MediaProjection API to take screenshots. The process is:

1. A screenshot is taken at the device’s native resolution.
2. It is converted to grayscale and scaled down to 720p.
3. Any black/blue letterbox bars (common on tablets or in Custom Game Area mode) are cropped away.
4. The resulting 720p grayscale image is stored in memory.

When a script calls region.exists(image), OpenCV’s template matching (matchTemplate) runs against the cropped region of the in-memory screenshot. The default minimum similarity threshold is 80%. Values below 65% generally mean the reference image simply doesn’t match the screen at all.

Because all comparison images in the assets folder are in 720p, newly contributed images must also be in 720p to ensure correct matching behavior.

The useSameSnapIn { } block is important for performance: it tells the framework to reuse the same screenshot for all image checks inside the block rather than taking a fresh screenshot for each call.

Screen Gestures via Accessibility Service

Tapping and swiping are performed through Android’s AccessibilityService. FGA registers a service that the user grants permission to, and it dispatches GestureDescription objects to simulate touches. No root is needed because Accessibility Services are a standard Android feature.

The GestureService interface abstracts over the actual service so that libautomata doesn’t import any Android framework classes:

interface GestureService {
    fun click(location: Location)
    fun swipe(start: Location, end: Location, duration: Duration)
}

EntryPoint: The Base Class for Scripts

Every runnable script extends EntryPoint:

abstract class EntryPoint(val exitManager: ExitManager) {
    fun run() {
        thread(start = true) { scriptRunner() }
    }

    fun stop() = exitManager.exit()

    abstract fun script(): Nothing

    var scriptExitListener: (Exception) -> Unit = { }
}

The script() method returns Nothing because it is designed to loop forever (using while(true)) until either an ExitManager.exit() call throws a ScriptAbortException, or the script itself decides to stop by throwing a BattleExitException. The exit listener is how the UI learns that a script has finished and what the reason was.

The AutoBattle Script

AutoBattle is the most complex entrypoint. It coordinates support selection, party selection, combat, AP refilling, drop tracking, and story skipping.

The core loop uses a clean pattern: a Map of validator lambdas to action lambdas. On every iteration, the app takes a snapshot, evaluates all validators in order, and runs the first action whose validator returns true.

val screens: Map<() -> Boolean, () -> Unit> = mapOf(
    { connectionRetry.needsToRetry() } to { connectionRetry.retry() },
    { battle.isIdle() } to { battle.performBattle() },
    { isInMenu() } to { menu() },
    { isInResult() } to { result() },
    { isInSupport() } to { support() },
    { isRepeatScreen() } to { repeatQuest() },
    { withdraw.needsToWithdraw() } to { withdraw.withdraw() },
    { needsToStorySkip() } to { skipStory() },
    // ...
)

while (true) {
    val actor = useSameSnapIn {
        screens.asSequence()
            .filter { (validator, _) -> validator() }
            .map { (_, actor) -> actor }
            .firstOrNull()
    }
    actor?.invoke()
    0.5.seconds.wait()
}

This approach keeps the loop logic easy to read and easy to extend: adding a new screen state is just adding one more entry to the map.

Exit conditions are modelled as a sealed class:

sealed class ExitReason(val cause: Exception? = null) {
    data object Abort : ExitReason()
    data object APRanOut : ExitReason()
    data object InventoryFull : ExitReason()
    class LimitRuns(val count: Int) : ExitReason()
    class LimitMaterials(val count: Int) : ExitReason()
    data object CEGet : ExitReason()
    // ...
}

When the UI shows the “battle finished” dialog, it deconstructs this sealed class to show the appropriate message to the user.

Battle Modules

The Battle class orchestrates each turn of combat by composing several smaller, injected modules:

• AutoSkill — Parses and executes the user-configured skill command string (e.g. "1a,2b,3c" meaning “use skill 1 on slot A, skill 2 on slot B, skill 3 on slot C”).
• Card — Reads the five face cards dealt after pressing the attack button using image recognition, then selects the optimal cards based on the configured priority.
• CardParser — Identifies each command card as Buster, Arts, or Quick, and determines which Servant it belongs to.
• ApplyBraveChains — Re-orders selected cards to form Brave Chains (three cards from the same Servant) when possible.
• SkillSpam — Optionally uses skills automatically every turn without a specific sequence.
• ShuffleChecker — Triggers the Master Skill shuffle mechanic when the dealt cards don’t match the desired wave composition.
• StageTracker — Tracks which wave of the quest the script is currently on.
• ServantTracker — Monitors which Servants are alive in the party.

Each module is a separate class with a single focused responsibility, making them easy to test and modify independently.

Support Selection

The support selection flow is handled by SupportSelectionLoop, which tries a configured strategy (preferred Servant/CE image match) and falls back gracefully:

fun selectSupport(selectionMode: SupportSelectionModeEnum = supportPrefs.selectionMode) {
    val provider = decider.decide(selectionMode)
    if (!loop.select(provider)) {
        val fallback = decider.decide(supportPrefs.fallbackTo)
        if (!loop.select(fallback)) {
            ManualSupportSelection.select() // Notify user to pick manually
        }
    }
}

Image matching for Servant and CE portraits uses the user-supplied images in the app’s “support images” folder. The SupportImageMaker script automates the creation of these reference images by cropping them from a live screenshot.

Dependency Injection with Hilt

FGA uses Dagger Hilt throughout. A custom @ScriptScope annotation limits the lifetime of script-related objects to a single script run (they are destroyed when the script exits and recreated for the next run):

@ScriptScope
class AutoBattle @Inject constructor(
    exitManager: ExitManager,
    api: IFgoAutomataApi,
    private val battle: Battle,
    private val support: Support,
    private val refill: Refill,
    // ...
) : EntryPoint(exitManager), IFgoAutomataApi by api { ... }

The IFgoAutomataApi by api Kotlin delegation pattern is the same technique described in my extension functions DI post: all extension functions from AutomataApi and IFgoAutomataApi become available in AutoBattle without any boilerplate forwarding.

Adding New Screen Coordinates

Because FGA maps everything to a 1440p virtual canvas, adding support for a new screen element involves these steps:

1. Take a screenshot on the device you want to support.
2. Open it in Photopea and remove any letterbox bars.
3. Scale the image to 720p height.
4. Use the ruler tool to find the X and Y positions of the element.
5. Double the values (720p → 1440p).
6. If the element sticks to the center of the screen, subtract half the width from X and use xFromCenter().

// Example: a button 200px to the right of center
val myButtonClick = locations.scriptArea.center + Location(400, 0)
// or more explicitly:
val myButtonRegion = Region(Location(0, 600).xFromCenter(), 580, 400, 60)

Contributing

The project welcomes contributions. The workflow is:

1. Fork the repository on GitHub.
2. Clone and open in Android Studio.
3. Set versionCode in app/build.gradle.kts to your installed FGA version (Android won’t downgrade).
4. Test with a physical device or emulator (ADB: adb connect localhost:5555 for BlueStacks).
5. Submit a pull request against the master branch.

Translations live on POEditor and don’t require code changes.

Wrapping Up

FGA is a fascinating example of combining several Android platform features — MediaProjection, Accessibility Services, and OpenCV — into a coherent automation framework. The clean separation between libautomata (platform abstractions), scripts (game logic), and app (Android glue) makes the codebase easy to navigate even as it handles dozens of different game screens and edge cases.

The use of Kotlin extension functions in interfaces, dependency injection with Hilt, and sealed classes for exit reasons are patterns that transfer directly to other Android projects. If you’re building anything that needs to observe and interact with the screen, FGA’s source code is well worth reading.

Check out the full source on GitHub.