# Autogedal_FeedMonitor

`Autogedal_FeedMonitor` records every run of the stock / price feed jobs and emails the team when a feed cannot be
read, is invalid or much smaller than usual, when a job stops running or never finishes, and when the Magento cron
itself is down.

## Architecture

```
Firebear jobs ---------┐
Custom crons ----------┤
Sub-feeds -------------┼─> source adapters ─> FeedMonitor (core) ─> run history
Magento heartbeat -----┘    (Model\Adapter)                     ─> alert state ─> email (TransportBuilder / SMTP)
```

- **`Model\FeedMonitor` (core)**: run history (`autogedal_feed_monitor_run`), alert state per job / sub-feed
  (`autogedal_feed_monitor_state`): one email when a problem starts or changes, at most one reminder every N hours,
  one email when it is gone (the alert closes only once that email left, else it stays "recovery pending" and is
  retried), each change under a lock per key (`LockManagerInterface`) so overlapping checks never
  send the same email twice; the guard before updating (`acceptItemCount()`, fail closed: when the check itself cannot
  be done, the update does not start); jobs that stopped running; runs that
  never finished (limit per job). It never stops a job: its own errors are only logged.
- **Source adapters (`Model\Adapter`)**, listed in `etc/di.xml` (argument `sources` of `Model\Checker`):
  - `CustomCronMonitor`: what the custom crons call (`begin`, `readFeed`, `acceptItemCount`, `complete`, `fail`,
    `failException`) and the "not running" check of those jobs. `complete()` ends the run with the result of the
    update: products that could not be saved (and prices refused by `BasePriceStorage` / `SpecialPriceStorage`) make
    the run fail with problem "update" and the number actually saved;
  - `CustomSubfeedMonitor`: the same per sub-feed (`autogedal_feed_oos_products`: one run and one alert per feed),
    plus the saved reference copy (`readReference`) and the "too many products missing" guard (`acceptMissingCount`);
  - `MagentoHeartbeatMonitor`: the Magento cron itself, checked only from the server crontab;
  - `FirebearJobMonitor`: the Firebear import jobs that are active and scheduled (read through Firebear's
    `JobRepository`, monitored as `firebear_job_<id>`):
    - the result of each import, recorded by a plugin on Firebear's `SummaryHandler` (the handler that dispatches
      `firebear_import_success` / `firebear_import_failure`), which gets the file of the run that just ended: the
      history row and the log (`var/log/firebear/<file>.log`) of that run are read, never the job's latest one;
    - judged after the import (Firebear has no guard of its own) with the rules of the custom feeds: failed import
      (problem "import"), no row checked ("empty"), more than 10% invalid rows or a log without the rows checked
      ("invalid"), far fewer rows than the last successful run ("incomplete", accepted once with
      `allow-incomplete firebear_job_<id>`); a few invalid rows are only noted with the success;
    - imports started and never finished (Firebear history without an end time, up to 48 h back);
    - jobs that have not started within the interval of their cron expression + 2 h (jobs whose expression restricts
      days of the month or months, e.g. `* * 30 2 *`, are not checked);
    - monitored: imports run by cron, CLI (`import:job:run`) and API (Firebear's `RunById` and its handlers). Not
      monitored, on purpose: imports started by hand from the admin (System > Import Jobs > Run), which go through
      `Helper::runImport()` and skip the handlers;
    - the feed is not downloaded a second time: what counts is the result of Firebear's own import;
    - Firebear's own email notifications (per job, currently "None") are left as they are; enabling them would
      send a second email for the same failed import.
- **Common feed reading (`Model\Fetch\FeedReader`)**: download (`FeedFetcher`: connect / per-attempt timeouts, up to
  N attempts within a limit for the whole read, attempts and pauses included; only temporary failures retried; only the
  host of the feed in messages) and format validation (`CsvFormat`, `XmlFormat`). The
  result is explicit: OK with the rows, or empty / invalid / fetch error with the reason. Rows missing a required
  field, with a non numeric value where a number is expected or with a value outside its pattern / accepted values are
  rejected and reported with the run; when more than 10% of the rows are rejected the whole feed is invalid and nothing
  is updated from it. `readFeed()` then refuses a
  feed with no products or below the configured share of the last successful run.
- **`Model\Checker`** (cron `autogedal_feed_monitor_check`, every 15 minutes, and `autogedal:feed-monitor:check`):
  closes runs that never finished, runs the check of every adapter, sends reminders.

Monitored jobs (declared in the `di.xml` of the module that owns them, argument `jobs` of `Model\JobRegistry`):

| Job | Module | Adapter |
|---|---|---|
| `smartbill_stock_update`, `smartbill_um_update` | Leadlion_StockUpdates | CustomCronMonitor (SmartBill API, no HTTP feed) |
| `ptc_update_stock`, `ptc_custom_product_prices` | Leadlion_StockUpdates | CustomCronMonitor, CSV without header |
| `ptc_la_comanda_stock` | Leadlion_StockUpdates | CustomCronMonitor (list saved by `ptc_update_stock`) |
| `carsupreme_paravanturi_stock`, `carsupreme_paravanturi_prices` | Leadlion_StockUpdates | CustomCronMonitor, CSV with header |
| `alfaromtrans_paravanturi_stock` | Leadlion_StockUpdates | CustomCronMonitor, XML |
| `autogedal_feed_oos_products` (5 sub-feeds) | Leadlion_Autogedal | CustomSubfeedMonitor, CSV |
| `firebear_job_<id>`: every active, scheduled Firebear import job | Firebear_ImportExport | FirebearJobMonitor |

## Setup

```bash
bin/magento module:enable Autogedal_FeedMonitor
bin/magento setup:upgrade
bin/magento cache:flush
```

Admin: Stores > Configuration > Autogedal > Feed Monitor: recipients, reminder interval, incomplete thresholds
(default and per feed), maximum interval without a run per job, timeouts and attempts.

Server crontab, apart from `cron:run`:

```
0,30 * * * * php /path/to/magento/bin/magento autogedal:feed-monitor:check >> /path/to/magento/var/log/feed_monitor_check.log 2>&1
```

## Commands

```bash
bin/magento autogedal:feed-monitor:status                       # last run and open alert per job / sub-feed,
                                                                # jobs missing from crontab.xml
bin/magento autogedal:feed-monitor:run alfaromtrans_paravanturi_stock   # run a monitored job now
bin/magento autogedal:feed-monitor:allow-incomplete ptc_update_stock    # next run may update from a smaller feed (24 h)
bin/magento autogedal:feed-monitor:allow-incomplete autogedal_feed_oos_products --feed=schiuri
bin/magento autogedal:feed-monitor:check                        # checks + Magento cron (what the server crontab runs)
bin/magento autogedal:feed-monitor:test-email                   # send a test alert
```

Testing a broken feed (only in the process of the command, through `SimulatedHttpClient`; the cron is not affected):

```bash
bin/magento autogedal:feed-monitor:run alfaromtrans_paravanturi_stock --feed-file=var/feed_monitor_test/eroare_html.html
bin/magento autogedal:feed-monitor:run ptc_custom_product_prices --http-code=503
bin/magento autogedal:feed-monitor:run alfaromtrans_paravanturi_stock --connection-error
bin/magento autogedal:feed-monitor:run autogedal_feed_oos_products --feed-file=... --only-url=061ca76d   # one sub-feed
```

## Adding a job

1. Declare it in the owning module's `di.xml`:
   ```xml
   <type name="Autogedal\FeedMonitor\Model\JobRegistry">
       <arguments>
           <argument name="jobs" xsi:type="array">
               <item name="my_feed_job" xsi:type="array">
                   <item name="label" xsi:type="string">Stoc furnizor X</item>
                   <item name="max_age_minutes" xsi:type="number">1560</item>
                   <item name="log_file" xsi:type="string">var/log/my_feed_job.log</item>
               </item>
           </argument>
       </arguments>
   </type>
   ```
2. In the job (inject `CustomCronMonitor`):
   ```php
   $run = $this->cronMonitor->begin('my_feed_job');
   try {
       $feed = $this->cronMonitor->readFeed($run, $url, new CsvFormat(',', true, 'id', ['qty'], ['qty']), [], $logger);
       if ($feed === null) {
           return; // unreadable / invalid / empty / far smaller than usual: already reported, nothing updated
       }
       // ... update from $feed->getRows(), collecting $failures[$sku] = $error for products not saved ...
       $this->cronMonitor->complete($run, $feed->getCount(), $saved, $failures);
   } catch (\Throwable $e) {
       $this->cronMonitor->failException($run, $e);
       throw $e;
   }
   ```