Unify remaining crawler URL handoffs
This commit is contained in:
@@ -1,597 +0,0 @@
|
||||
# Peepo Loader crawler and extractor map
|
||||
|
||||
**Status:** repository map, written 2026-09-28
|
||||
**Repository:** `browser-extension__peepo-loader`
|
||||
**Purpose:** explain what each crawler/extractor actually does, which route invokes it, what it returns, and where the current logs can give a false impression of coverage.
|
||||
|
||||
This is a source map, not a promise that every path is correct. Where the implementation behaves differently from the apparent product intent, this document says so under **Observed gap / verification risk**.
|
||||
|
||||
## 1. The vocabulary that matters
|
||||
|
||||
There are four different things that are easy to call “the crawler”:
|
||||
|
||||
1. **A collector** asks one already-open page for links, images, videos, or other browser data.
|
||||
2. **A crawl plugin** repeatedly collects a page, scrolls it, waits, and collects again. The current plugins are `site-scroll` and `redgifs-suggested`.
|
||||
3. **An extractor** is an enabled operation such as `reddit-gallery`, `youtube-video`, or `direct-media`. In the shared background graph, each enabled extractor is attempted according to the selected routing mode, regardless of which transport produced the seed.
|
||||
4. **The tab pipeline** is the background service-worker workflow that first harvests source tabs, then traverses a graph of discovered URLs one extractor operation at a time.
|
||||
|
||||
The most important mental model is therefore:
|
||||
|
||||
```text
|
||||
input URL / selected tab
|
||||
│
|
||||
├─ direct drop / paste ──────────────► queue URL
|
||||
│ (routing label only)
|
||||
│
|
||||
├─ forced site-scroll ───────────────► crawl page repeatedly
|
||||
│ ► general-extraction labeling
|
||||
│ ► queue source + discovered URLs
|
||||
│
|
||||
├─ forced redgifs-suggested ──────────► RedGIFs crawl repeatedly
|
||||
│ ► queue crawled RedGIFs URLs
|
||||
│
|
||||
├─ manual Tab extraction pipeline ────► collect source tabs
|
||||
│ ► graph traversal
|
||||
│ ► real extractor-specific tabs
|
||||
│ ► queue graph results
|
||||
│
|
||||
└─ standalone collector ─────────────► one active-tab collector,
|
||||
domain-tab collector,
|
||||
M3U8 listener, or Reddit sequence
|
||||
```
|
||||
|
||||
The actual downloader workers are downstream of this extension. These paths discover and queue URLs; they do not download the media themselves.
|
||||
|
||||
## 2. Source index
|
||||
|
||||
The behavior is spread across these files:
|
||||
|
||||
| Concern | Source |
|
||||
|---|---|
|
||||
| Extractor IDs, default enablement, URL-to-plan routing, pipeline controls | [`src/extractor-pipeline.ts`](../src/extractor-pipeline.ts) |
|
||||
| Drop/paste/manual route, panel-side general extraction, settings, queue persistence | [`src/App.tsx`](../src/App.tsx) |
|
||||
| Service-worker entrypoints, temporary tabs, crawl plugins, graph traversal, recovery | [`src/background.ts`](../src/background.ts) |
|
||||
| In-page collectors, scroll-container selection, RedGIFs Share → Copy Link automation | [`content-script/src/main.tsx`](../content-script/src/main.tsx) |
|
||||
| Browser-side collector/security behavior tests | [`tests/`](../tests/) |
|
||||
|
||||
Useful anchors:
|
||||
|
||||
- Extractor type and defaults: [`src/extractor-pipeline.ts:1-102`](../src/extractor-pipeline.ts#L1-L102)
|
||||
- URL routing: [`src/extractor-pipeline.ts:128-170`](../src/extractor-pipeline.ts#L128-L170)
|
||||
- Panel crawl bridge: [`src/App.tsx:620-653`](../src/App.tsx#L620-L653)
|
||||
- Panel-side graph: [`src/App.tsx:2790-2936`](../src/App.tsx#L2790-L2936)
|
||||
- Drop/manual dispatch: [`src/App.tsx:2938-3015`](../src/App.tsx#L2938-L3015)
|
||||
- Paste/forced-extractor dispatch: [`src/App.tsx:3141-3186`](../src/App.tsx#L3141-L3186)
|
||||
- Generic background crawl loop: [`src/background.ts:578-745`](../src/background.ts#L578-L745)
|
||||
- Background extractor filtering: [`src/background.ts:936-1041`](../src/background.ts#L936-L1041)
|
||||
- Background tab pipeline: [`src/background.ts:1043-1604`](../src/background.ts#L1043-L1604)
|
||||
- Content-script collectors: [`content-script/src/main.tsx:236-295`](../content-script/src/main.tsx#L236-L295)
|
||||
- Content-script crawl bridge: [`content-script/src/main.tsx:398-484`](../content-script/src/main.tsx#L398-L484)
|
||||
|
||||
## 3. URL routing: how a URL gets its initial extractor plan
|
||||
|
||||
`extractorPlanForUrl()` parses the URL and returns an ordered list. It never returns `site-scroll` automatically.
|
||||
|
||||
```text
|
||||
parse URL
|
||||
├─ invalid URL ─────────────────────────────► [direct-media]
|
||||
│
|
||||
├─ redgifs.com or subdomain
|
||||
│ ├─ pathname matches /user/, /users/, /profile/
|
||||
│ │ ► [redgifs-uploader]
|
||||
│ └─ otherwise ► [redgifs-suggested, redgifs-post]
|
||||
│
|
||||
├─ reddit.com or subdomain ► [reddit-post,
|
||||
│ reddit-gallery,
|
||||
│ external-source]
|
||||
│
|
||||
├─ youtube.com /watch, /shorts/, /live/
|
||||
│ or youtu.be ► [youtube-video]
|
||||
│
|
||||
└─ every valid branch ► append direct-media
|
||||
```
|
||||
|
||||
The final plan is de-duplicated. Therefore:
|
||||
|
||||
| URL shape | Ordered plan |
|
||||
|---|---|
|
||||
| RedGIFs post-like URL | `redgifs-suggested`, `redgifs-post`, `direct-media` |
|
||||
| RedGIFs user/profile URL | `redgifs-uploader`, `direct-media` |
|
||||
| Reddit URL | `reddit-post`, `reddit-gallery`, `external-source`, `direct-media` |
|
||||
| YouTube watch/short/live or `youtu.be` | `youtube-video`, `direct-media` |
|
||||
| Other valid web URL | `direct-media` |
|
||||
| Invalid URL passed to the planner | `direct-media` |
|
||||
|
||||
Settings filter this plan through `enabledExtractorPlan()`. A forced extractor can be appended/selected independently of the URL plan in panel-side flows.
|
||||
|
||||
### 3.1 Important consequence: `site-scroll` is not automatic
|
||||
|
||||
`site-scroll` is a manually forced crawler. It is enabled by default in extractor settings, but normal URL routing never selects it. The visible “Scroll & collect page” control is therefore a deliberate user choice, not a fallback for an ordinary page.
|
||||
|
||||
### 3.2 RedGIFs suggested ordering is intentional
|
||||
|
||||
RedGIFs post-like URLs route to `redgifs-suggested` before `redgifs-post`. The source comment explains why: the suggested crawler needs the first usable RedGIFs page and the generic post collector can lease/close a separate tab too early. See [`src/extractor-pipeline.ts:143-159`](../src/extractor-pipeline.ts#L143-L159).
|
||||
|
||||
## 4. Entry paths into the system
|
||||
|
||||
### 4.1 Dropzone: browser drag/drop
|
||||
|
||||
Entry: `queueDroppedContent()` in [`src/App.tsx:2938`](../src/App.tsx#L2938).
|
||||
|
||||
The panel tries these payloads in order:
|
||||
|
||||
1. Peepo’s custom `application/x-peepo-gallery-urls` payload.
|
||||
2. Browser `text/uri-list` and `text/plain` lines.
|
||||
3. Parsed `text/html` attributes from `img[src]`, `a[href]`, `video[src]`, and `source[src]`.
|
||||
|
||||
Every candidate must be an `http:`, `https:`, or `magnet:` URL and is de-duplicated. A forced extractor replaces normal routing. Without a force choice, `extractorForDroppedUrl()` takes the first item in `extractorPlanForUrl()`.
|
||||
|
||||
After the input is parsed:
|
||||
|
||||
```text
|
||||
drop
|
||||
│
|
||||
├─ no valid candidates ─────────────────────► warning; stop
|
||||
│
|
||||
├─ forced redgifs-suggested ────────────────► dedicated RedGIFs crawl
|
||||
│
|
||||
├─ forced site-scroll ─────────────────────► dedicated generic crawl
|
||||
│ ► panel general extraction
|
||||
│
|
||||
├─ no forced extractor ─────────────────────► panel general extraction
|
||||
│ (usually labels/queues URLs)
|
||||
│
|
||||
└─ other forced extractor ─────────────────► queue the dropped URL
|
||||
with that label
|
||||
```
|
||||
|
||||
The queue layer then applies URL normalization, URL filters, routing rules, duplicate suppression, and queue capacity. See [`src/App.tsx:2725-2787`](../src/App.tsx#L2725-L2787).
|
||||
|
||||
### 4.2 Paste/current-tab input
|
||||
|
||||
`queueInputWithExtractor()` follows the same split as drag/drop:
|
||||
|
||||
- `redgifs-suggested`: run the RedGIFs crawl, then queue the returned URLs.
|
||||
- `site-scroll`: run the generic crawl, then send the source and discovered URLs into the shared extraction graph.
|
||||
- Anything else: queue the supplied URL directly with the chosen extractor label.
|
||||
|
||||
See [`src/App.tsx:3141-3186`](../src/App.tsx#L3141-L3186).
|
||||
|
||||
### 4.3 Manual Tab extraction pipeline
|
||||
|
||||
The Collector UI sends `run-tab-pipeline` with:
|
||||
|
||||
- a domain list in `options.domains`;
|
||||
- tab-pipeline extractor enablement;
|
||||
- graph controls such as recursion, depth, node count, invocation count, pacing, rate-limit handling, and load timeout;
|
||||
- crawler options such as scroll passes and scroll-stall handling.
|
||||
|
||||
The service worker rejects a second pipeline while one is active. See [`src/background.ts:1043-1054`](../src/background.ts#L1043-L1054) and the panel dispatch at [`src/App.tsx:3396-3420`](../src/App.tsx#L3396-L3420).
|
||||
|
||||
### 4.4 Standalone collectors
|
||||
|
||||
The Collector UI can also send `run-collector` without starting the graph pipeline:
|
||||
|
||||
- `links`, `videos`, `images`: ask the active tab’s content script to collect one type.
|
||||
- `domainTabs`: inspect matching tabs as a separate collector.
|
||||
- `m3u8-start` / `m3u8-stop`: start/stop a network listener.
|
||||
- `reddit-sequence-start` / `reddit-sequence-stop`: run/stop the sequential Reddit post worker.
|
||||
|
||||
The dispatch is in [`src/background.ts:2214-2269`](../src/background.ts#L2214-L2269).
|
||||
|
||||
## 5. The two crawl plugins
|
||||
|
||||
Both plugins use the same `runCrawlPlugin()` loop. Their defaults are defined in [`src/background.ts:266-289`](../src/background.ts#L266-L289).
|
||||
|
||||
| Plugin | Host restriction | Default passes | Scroll | Wait after load | Wait after scroll | Special behavior |
|
||||
|---|---:|---:|---:|---:|---:|---|
|
||||
| `redgifs-suggested` | `redgifs.com` and subdomains | 5 | 850 px | 1500 ms | 1250 ms | temporary focus activation; optional Share → Copy Link |
|
||||
| `site-scroll` | any host | 8 | 850 px | 1000 ms | 1000 ms | ordinary links/videos/images only |
|
||||
|
||||
The panel settings override passes, distance, waits, progress polling, share-link behavior, minimum page-height growth, and stall action. The settings are sent by [`src/App.tsx:620-640`](../src/App.tsx#L620-L640).
|
||||
|
||||
### 5.1 Temporary-tab lifecycle
|
||||
|
||||
For a direct crawl:
|
||||
|
||||
```text
|
||||
find existing tab with same normalized URL
|
||||
├─ usable existing tab found ──────────────► reuse it
|
||||
└─ otherwise ──────────────────────────────► create background crawl window/tab
|
||||
|
||||
move tab into “Peepo crawl” group
|
||||
wait for document complete
|
||||
wait for plugin load delay
|
||||
if RedGIFs:
|
||||
focus tab once, wait 100 ms, optionally install page-world clipboard hook
|
||||
|
||||
run collect/scroll passes
|
||||
|
||||
success ───────────────────────────────────► close leased tab
|
||||
failure / abort / user stop ───────────────► restore original group/index
|
||||
```
|
||||
|
||||
The tab placement and restoration logic is in [`src/background.ts:805-893`](../src/background.ts#L805-L893). A successful crawl closes the leased tab; a failed crawl restores it instead. A RedGIFs crawl also restores the user’s previously active tab after completion when possible.
|
||||
|
||||
### 5.2 One collect/scroll pass
|
||||
|
||||
The loop is the following, in this exact order:
|
||||
|
||||
```text
|
||||
for step = 0 .. steps - 1
|
||||
│
|
||||
├─ check stop flag / wait while paused
|
||||
├─ confirm crawl tab still exists
|
||||
├─ optional plugin.beforeStep
|
||||
├─ crawl-collect links + videos + images
|
||||
├─ optional RedGIFs Share → Copy Link collector
|
||||
├─ de-duplicate URLs against URLs already seen by this crawl
|
||||
├─ enforce maxCrawlUrls
|
||||
├─ emit “elements collected” and checkpoint all URLs
|
||||
├─ if URL limit reached: finish successfully
|
||||
├─ if queue is full: pause until capacity is available
|
||||
├─ optional plugin.afterCollect
|
||||
├─ if another pass remains:
|
||||
│ ├─ send crawl-scroll
|
||||
│ ├─ wait after scroll
|
||||
│ ├─ query settled scroll state
|
||||
│ ├─ emit “scroll complete”
|
||||
│ ├─ compare page height before/after
|
||||
│ └─ abort or reload if growth is below threshold
|
||||
└─ continue
|
||||
```
|
||||
|
||||
See [`src/background.ts:668-735`](../src/background.ts#L668-L735).
|
||||
|
||||
The final pass collects but does not scroll. This means a crawl with `N` passes performs `N` collections and at most `N-1` scrolls.
|
||||
|
||||
### 5.3 Scroll-stall detection
|
||||
|
||||
The content script chooses a primary scroll container by looking at the document scrolling element and up to the first 800 DOM elements. It selects the eligible element with the largest scrollable range. See [`content-script/src/main.tsx:31-41`](../content-script/src/main.tsx#L31-L41).
|
||||
|
||||
After `crawl-scroll`, the background worker waits the configured post-scroll delay and sends `crawl-scroll-state`. If both heights are available:
|
||||
|
||||
```text
|
||||
growth = settledScrollHeight - scrollHeightBefore
|
||||
if scroll position did not advance AND growth < crawlerScrollHeightGrowthThresholdPx:
|
||||
increment consecutive stall count
|
||||
if count reaches 3:
|
||||
scroll is considered stalled
|
||||
else:
|
||||
reset consecutive stall count
|
||||
```
|
||||
|
||||
Default threshold: **100 px**. The setting is shared by both crawl plugins.
|
||||
|
||||
- A scroll is only considered stalled when neither the scroll position nor the page height makes progress. This avoids treating ordinary fixed-height pages (including Reddit views) as stalled merely because their scrollable content did not expand.
|
||||
- The first two no-progress checks emit `scroll progress delayed` and do not take corrective action. Any position or height progress resets the consecutive counter.
|
||||
- `abort`: after 3 consecutive below-threshold checks, emit `scroll stalled`, throw an error, preserve checkpointed URLs, and finish as an error/partial crawl.
|
||||
- `refresh`: after 3 consecutive below-threshold checks, emit `scroll stalled`, reload the crawl tab, wait for completion and the normal load delay, reset the counter, and continue the loop.
|
||||
|
||||
The current rule only evaluates when both height readings are available. A missing bridge response does not itself count as a height stall; other collection/bridge error handling may still fail the crawl.
|
||||
|
||||
## 6. RedGIFs suggested crawler branch
|
||||
|
||||
This branch is more than a normal DOM snapshot.
|
||||
|
||||
### 6.1 Standard collection
|
||||
|
||||
Each pass sends `crawl-collect` for `links`, `videos`, and `images`. The content script returns a de-duplicated URL list, a DOM-element count, and page signals for rate limits/challenges. See [`content-script/src/main.tsx:460-474`](../content-script/src/main.tsx#L460-L474).
|
||||
|
||||
### 6.2 Share → Copy Link collection
|
||||
|
||||
When enabled, the crawler additionally:
|
||||
|
||||
1. Finds visible RedGIFs “more actions” buttons.
|
||||
2. Skips post roots already processed in this content-script lifetime.
|
||||
3. Clicks the menu.
|
||||
4. Clicks `Share`.
|
||||
5. Clicks `Copy Link`.
|
||||
6. Waits, then reads either:
|
||||
- the page-world clipboard-write hook (`page-hook`, default), or
|
||||
- `navigator.clipboard.readText()` (`clipboard` mode).
|
||||
7. Accepts only valid `http(s)` RedGIFs URLs.
|
||||
8. Closes the popup and marks the post processed.
|
||||
|
||||
See [`content-script/src/main.tsx:337-377`](../content-script/src/main.tsx#L337-L377). Events are emitted as “reading link from clipboard”; the event wording remains true even in page-hook mode because the hook captures the page’s clipboard write rather than reading browser clipboard permission.
|
||||
|
||||
### 6.3 RedGIFs result path
|
||||
|
||||
```text
|
||||
RedGIFs source URL
|
||||
► collect visible DOM links/media
|
||||
► optionally collect copied post links
|
||||
► queue newly seen URLs in crawl context
|
||||
► after crawl completes:
|
||||
direct forced/manual route: queue returned URLs as redgifs-suggested
|
||||
tab-pipeline route: enqueue returned URLs as graph children
|
||||
with parent = current RedGIFs URL
|
||||
and path += redgifs-suggested
|
||||
```
|
||||
|
||||
The direct panel route is [`src/App.tsx:2945-2970`](../src/App.tsx#L2945-L2970). The pipeline route is [`src/background.ts:1477-1521`](../src/background.ts#L1477-L1521).
|
||||
|
||||
## 7. Generic `site-scroll` branch
|
||||
|
||||
The generic crawler uses the same collect/scroll loop but no interactive RedGIFs collector.
|
||||
|
||||
For a forced panel crawl:
|
||||
|
||||
```text
|
||||
source URL
|
||||
► site-scroll passes
|
||||
► return every collected link/video/image URL
|
||||
► send returned URLs into the shared extraction graph
|
||||
► include original source URL
|
||||
► de-duplicate
|
||||
► queue through normal URL validation/filter/routing rules
|
||||
```
|
||||
|
||||
See [`src/App.tsx:2972-3005`](../src/App.tsx#L2972-L3005) and [`src/App.tsx:3155-3176`](../src/App.tsx#L3155-L3176).
|
||||
|
||||
### Normalized seeded extraction
|
||||
|
||||
The panel no longer has a second, label-only extraction implementation. A forced scroll, RedGIFs crawl, drag/drop seed, paste seed, or current-tab seed is converted into `seedRecords` and sent to the background tab pipeline. The same graph runner then applies the enabled extractor settings, opens/reuses extractor tabs, records execution, and queues discovered children.
|
||||
|
||||
Transport-specific work is limited to finding seeds. This includes standalone page collectors and content-script URL handoffs; they no longer bypass the extraction graph by forwarding directly to the queue.
|
||||
|
||||
```text
|
||||
transport criteria
|
||||
► seed URLs
|
||||
► shared background extraction graph
|
||||
► enabled extractor attempts
|
||||
► recursive child seeds
|
||||
► normal queue validation/deduplication
|
||||
```
|
||||
|
||||
Seeded transports force `all-enabled` routing: an enabled extractor is attempted for every seed regardless of the URL’s origin or which transport found it. An extractor may legitimately return zero URLs. `site-scroll` remains a transport and is excluded from per-URL extractor plans so it cannot recursively scroll every child URL.
|
||||
|
||||
## 8. Background Tab Pipeline: complete path
|
||||
|
||||
The background pipeline is a separate implementation and does perform real extractor-specific page inspections.
|
||||
|
||||
### Phase A: select source tabs
|
||||
|
||||
1. Read `options.domains` and normalize it.
|
||||
2. Query all browser tabs.
|
||||
3. Build a reusable-tab map keyed by `graphUrlKey()`.
|
||||
4. Select tabs whose hostname equals or is a subdomain of one of the requested domains.
|
||||
5. Limit selected tabs by `maxDomainTabs`.
|
||||
6. Lease each selected tab into the `Peepo crawl` group.
|
||||
7. Focus it temporarily so content bridges/lazy DOM work reliably.
|
||||
8. Wait for page completion; RedGIFs gets the extra SPA settle/focus sequence.
|
||||
|
||||
See [`src/background.ts:1043-1078`](../src/background.ts#L1043-L1078) and [`src/background.ts:1205-1240`](../src/background.ts#L1205-L1240).
|
||||
|
||||
### Phase A collection branches
|
||||
|
||||
For each source tab:
|
||||
|
||||
```text
|
||||
collectTabUrl enabled?
|
||||
├─ yes ──► materialize the tab URL as a depth-0 seed
|
||||
└─ no ──► log “extractor skipped” for tab URL
|
||||
|
||||
collectPageMedia enabled?
|
||||
├─ yes ──► content bridge collects links/videos/images
|
||||
│ ├─ response ──► materialize returned URLs as depth-0 seeds
|
||||
│ └─ no response ► fail this source tab and restore it
|
||||
└─ no ──► use empty page collection
|
||||
|
||||
source tab succeeded?
|
||||
├─ yes ──► close leased source tab
|
||||
└─ no ──► restore its original group/index
|
||||
```
|
||||
|
||||
The source URL is committed before page-media bridge work so a later page-collector timeout does not erase it. See [`src/background.ts:1241-1285`](../src/background.ts#L1241-L1285).
|
||||
|
||||
### Phase B: materialize and traverse graph
|
||||
|
||||
Seeds are de-duplicated by URL/source pair, bounded by `maxCrawlUrls`, and materialized into `graphResults` before traversal. This deliberately preserves seed URLs even if the extractor invocation budget is exhausted.
|
||||
|
||||
The traversal is breadth-first because it shifts from the front of `graphPending` and appends children to the end.
|
||||
|
||||
```text
|
||||
graphPending
|
||||
► shift current node
|
||||
► skip if URL already visited (when enabled)
|
||||
► mark URL visited
|
||||
► skip if depth > maxDepth
|
||||
► compute matched extractor plan
|
||||
► materialize URL result if not already present
|
||||
► emit router decision
|
||||
► emit disabled-extractor records
|
||||
► for each selected extractor:
|
||||
check recursion/depth/compatibility/budget
|
||||
pace and count invocation
|
||||
run extractor operation
|
||||
record visited/failed/skipped graph record
|
||||
enqueue new child URLs with parent/path/depth
|
||||
► repeat until pending empty, invocation budget exhausted,
|
||||
or graph node budget exhausted
|
||||
```
|
||||
|
||||
The queue and child-edge logic is [`src/background.ts:1352-1405`](../src/background.ts#L1352-L1405). The traversal and router decision are [`src/background.ts:1410-1461`](../src/background.ts#L1410-L1461).
|
||||
|
||||
Every child gets:
|
||||
|
||||
- `sourceUrl`: original source-tab URL;
|
||||
- `parentUrl`: URL whose extractor returned it;
|
||||
- `depth`: parent depth + 1;
|
||||
- `path`: parent path plus the extractor that produced the child.
|
||||
|
||||
The graph uses `graphUrlKey()`, which removes hashes, trailing slashes, and common tracking parameters (`utm_*`, `fbclid`, `gclid`) for identity/deduplication. The stored/displayed URL is not necessarily rewritten to that key.
|
||||
|
||||
### Phase B extractor execution matrix
|
||||
|
||||
| Extractor | Background operation | Returned URLs |
|
||||
|---|---|---|
|
||||
| `direct-media` | No page operation | Empty list; the current URL itself remains a graph result/queue candidate |
|
||||
| `reddit-post` | Open/reuse page, collect all links/media, then filter | Links whose hostname differs from source hostname |
|
||||
| `external-source` | Same page collection/filter as `reddit-post` | External-host links |
|
||||
| `reddit-gallery` | Open/reuse page, collect all links/media, then filter | URLs matching media extensions |
|
||||
| `redgifs-post` | Same page collection/filter as media branch | URLs matching media extensions |
|
||||
| `youtube-video` | Same page collection/filter as media branch | URLs matching media extensions |
|
||||
| `redgifs-uploader` | Open/reuse page and filter collected candidates | URLs matching `redgifs.com/(watch|users)/` |
|
||||
| `redgifs-suggested` | Repeated RedGIFs crawl plugin | All URLs returned by its DOM + optional share-link passes |
|
||||
| `site-scroll` | Repeated generic crawl plugin | All URLs returned by its DOM passes |
|
||||
|
||||
The exact filtering is [`src/background.ts:936-948`](../src/background.ts#L936-L948). “Media” means a URL ending in `jpg`, `jpeg`, `png`, `gif`, `webp`, `avif`, `mp4`, `webm`, or `mov`, optionally followed by query/hash text. A URL that is media in content but does not expose one of these extensions is not returned by those extractor branches.
|
||||
|
||||
### Page/tab reuse and cache
|
||||
|
||||
For non-crawler extractors, `collectGraphUrls()`:
|
||||
|
||||
1. Returns immediately for `direct-media`.
|
||||
2. Looks in the per-run page cache by normalized graph URL.
|
||||
3. Reuses an existing matching tab when possible.
|
||||
4. Otherwise creates a temporary background tab.
|
||||
5. Waits for normal completion, or the longer RedGIFs SPA readiness sequence.
|
||||
6. Sends one `crawl-collect` request for links/videos/images.
|
||||
7. If a reused tab does not answer, closes/restores it and retries in a new temporary tab.
|
||||
8. Records rate-limit/challenge signals and caches the raw collection.
|
||||
9. Applies extractor-specific filtering to the cached raw collection.
|
||||
10. Closes successful temporary/reused leased tabs; restores failed tabs.
|
||||
|
||||
See [`src/background.ts:970-1040`](../src/background.ts#L970-L1040).
|
||||
|
||||
The cache is shared across extractor branches in one pipeline run. Thus two extractors applied to the same normalized URL can reuse one raw page snapshot but receive different filtered result sets.
|
||||
|
||||
## 9. Pipeline routing modes and their sharp edges
|
||||
|
||||
### `matched` mode
|
||||
|
||||
The background pipeline selects only the extractor IDs returned by `extractorPlanForUrl()` and enabled in pipeline settings. Since that planner never returns `site-scroll`, `site-scroll` is not run for normal matched routing.
|
||||
|
||||
### `all-enabled` mode
|
||||
|
||||
The background pipeline selects every enabled extractor except the transport-only `site-scroll`. Each selected extractor is attempted even when the URL is not a URL shape it normally recognizes; its extractor-specific collector may simply return no children.
|
||||
|
||||
### Transport seeds use the shared runner
|
||||
|
||||
Panel transport paths use the same background graph as the tab pipeline. Their extractor settings are passed as the pipeline’s enabled extractor set and their controls are normalized with `extractionMode: 'all-enabled'`.
|
||||
|
||||
### Observed gap: depth behavior differs
|
||||
|
||||
Panel-side `runGeneralExtraction()` refuses to run `redgifs-suggested` when `depth >= maxDepth` before invoking the crawl. The background graph traversal allows the current node when `depth <= maxDepth`, and only blocks newly enqueued children once the current node is at the depth boundary. The two routes can therefore produce different RedGIFs crawl coverage for the same settings.
|
||||
|
||||
### Observed gap: “visited” does not mean “completed”
|
||||
|
||||
Both graph implementations add a URL/extractor visit marker before all work completes. A later failure is represented separately as a failed graph record, but a retry must use the explicit retry path; simply seeing a URL in a visited set does not prove successful extraction.
|
||||
|
||||
## 10. Failure, pause, retry, and recovery paths
|
||||
|
||||
### Direct crawl failure
|
||||
|
||||
The crawl checkpoints URLs after each collection pass. If a later operation fails and some URLs exist, `runCrawlPlugin()` wraps the failure as `CrawlPartialError` and returns the checkpointed URL set to the error handler. The persisted crawl remains retryable with `retryFromStep`. See [`src/background.ts:715-745`](../src/background.ts#L715-L745) and [`src/background.ts:2141-2160`](../src/background.ts#L2141-L2160).
|
||||
|
||||
### User stop/pause
|
||||
|
||||
- Stop sets a control flag; the active loop notices it at its next guard and lets `finally` restore/close the tab safely.
|
||||
- Pause blocks at `waitIfPaused()`; queue-full pauses use the same mechanism.
|
||||
- A direct crawl and a tab pipeline have separate control maps and control messages.
|
||||
|
||||
Direct control handling: [`src/background.ts:2165-2184`](../src/background.ts#L2165-L2184).
|
||||
Pipeline control handling: [`src/background.ts:2192-2211`](../src/background.ts#L2192-L2211).
|
||||
|
||||
### Tab-pipeline failure
|
||||
|
||||
An individual graph extractor failure is recorded and traversal continues to the next graph item. A rate-limit-like error can pause the host with exponential backoff. A source-tab failure skips that source tab and continues with later source tabs.
|
||||
|
||||
At pipeline end, graph results are sent to the panel. On error, already captured graph results/seeds are flushed and a persisted error snapshot is written. See [`src/background.ts:1559-1603`](../src/background.ts#L1559-L1603).
|
||||
|
||||
### Service-worker recovery
|
||||
|
||||
Startup scans persisted running direct crawls and tab pipelines. Direct running crawls are converted to interrupted/error records with checkpointed URLs preserved. Running tab pipelines are resumed from stored seeds, pending nodes, active node, visited set, graph records, and results. See [`src/background.ts:1606-1653`](../src/background.ts#L1606-L1653).
|
||||
|
||||
## 11. Standalone collectors that are crawler-adjacent
|
||||
|
||||
### Links, videos, images
|
||||
|
||||
The service worker forwards these to the active tab’s content script. The content script implements:
|
||||
|
||||
- `links`: every anchor `href`;
|
||||
- `videos`: `video source` and `video` `src`;
|
||||
- `images`: `img.src`, all `srcset` candidates, lazy attributes (`data-src`, `data-lazy-src`, `data-original`), and `picture source` candidates.
|
||||
|
||||
Results are bounded and sent back in batches of 500. See [`content-script/src/main.tsx:236-295`](../content-script/src/main.tsx#L236-L295).
|
||||
|
||||
### Domain-tabs collector
|
||||
|
||||
This is separate from the tab pipeline. It operates over browser tabs matching supplied domain options; it does not create a graph of extractor children. Its implementation begins at [`src/background.ts:1836`](../src/background.ts#L1836).
|
||||
|
||||
### M3U8 / DASH listener
|
||||
|
||||
`m3u8-start` installs a `webRequest.onBeforeRequest` listener for `.m3u8` and `.mpd` URLs. M3U8 manifests are fetched/parsed, variant URLs are discovered, and results are forwarded to the panel. The listener is global or tab-scoped depending on the supplied `tabId`; `m3u8-stop` removes it. See [`src/background.ts:1874-1959`](../src/background.ts#L1874-L1959).
|
||||
|
||||
### Reddit sequence
|
||||
|
||||
This is a recursive sequential worker, not the normal extractor graph:
|
||||
|
||||
```text
|
||||
start
|
||||
► find a Reddit tab in group “Reddit Queue”
|
||||
► ask content script for visible /comments/ links
|
||||
► choose first URL not in visitedPosts
|
||||
► open or reuse worker tab
|
||||
► wait for load + 5 seconds
|
||||
► classify page as gallery/video/image/unknown
|
||||
► queue the page URL
|
||||
► bookmark it using content type as title
|
||||
► close successful worker or restore failed worker
|
||||
► repeat until no unvisited post or maxRedditSequencePosts
|
||||
```
|
||||
|
||||
See [`src/background.ts:2343-2495`](../src/background.ts#L2343-L2495). It does not run the extractor plan for each post; it queues/bookmarks the classified page URL.
|
||||
|
||||
## 12. What the logs prove—and what they do not
|
||||
|
||||
### Strong evidence currently present
|
||||
|
||||
These events are meaningful when they appear with the same URL/extractor/depth/path:
|
||||
|
||||
- `extractor plan`: the background router computed matched/enabled/selected/disabled sets.
|
||||
- `graph tab started` / `nested graph tab started`: a real background extractor operation was about to run.
|
||||
- `elements collected`: a real content bridge returned a collection, or a crawl pass collected page URLs.
|
||||
- `URL collected`: a concrete URL entered the returned set.
|
||||
- `checkpoint saved`: a crawl result was persisted and can be resumed/reviewed.
|
||||
- `extractor complete` / `crawler complete`: the operation returned without throwing.
|
||||
- `extractor skipped`: the log includes the reason, such as disabled, incompatible, recursion disabled, host budget, or already visited.
|
||||
|
||||
### Weak or ambiguous evidence
|
||||
|
||||
- Panel-side `Applying X` / visited graph entries prove that the panel considered an extractor label, not that a page was opened for that extractor.
|
||||
- A URL shown as `URL materialized` proves it was retained as a graph result, not that it was inspected.
|
||||
- `crawler started` alone proves no collection yet.
|
||||
- `scroll complete` proves the scroll command/state query returned; it does not prove new content loaded.
|
||||
- `elements collected` with zero URLs can be a legitimate empty page, a page with unsupported URL shapes, or a bridge that returned no candidates.
|
||||
- A final queue count can be lower than collected count because duplicate, filtered, invalid, or capacity-limited URLs are rejected by the queue layer.
|
||||
- Live feeds are bounded by `maxLiveCrawlLogEntries`; absence from the visible feed is not proof that an event never happened.
|
||||
|
||||
### Current observability gaps worth fixing
|
||||
|
||||
1. Panel-side general extraction should distinguish `classified` from `inspected`; currently ordinary extractor labels can look like completed extractor work.
|
||||
2. A crawl event should include `heightBefore`, `heightAfter`, `heightGrowth`, and `stallAction`, not only a prose detail string.
|
||||
3. Queue persistence should expose per-reason rejection counts separately: duplicate, filter, invalid, and capacity.
|
||||
4. Graph records should carry an explicit operation phase (`planned`, `opened-tab`, `collected`, `completed`, `failed`) instead of overloading `visited`.
|
||||
5. All transports should continue to feed the shared graph rather than reimplementing extractor behavior.
|
||||
6. Keep transport-only crawlers such as `site-scroll` out of recursive per-URL extractor plans.
|
||||
|
||||
## 13. Verification checklist for a real crawl
|
||||
|
||||
For a source URL `S`, a reviewer can reconstruct expected behavior with this checklist:
|
||||
|
||||
1. Record the URL planner output for `S`.
|
||||
2. Record which extractors are enabled and which pipeline mode is active.
|
||||
3. Confirm whether this is a forced panel route, panel general extraction, or background tab pipeline.
|
||||
4. For a background pipeline, find the `extractor plan` event for `S`.
|
||||
5. For every selected non-direct extractor, require a matching `graph tab started` event followed by `elements collected` and `extractor complete` or `crawler failed`.
|
||||
6. For every crawler, count collection events and compare with configured passes; expect one collection per pass and at most one scroll between passes.
|
||||
7. For each scroll, inspect height before/after and stall action; do not infer lazy-load success from scroll position alone.
|
||||
8. Follow every `URL collected` child through its `parentUrl`, depth, and path.
|
||||
9. Compare graph result count with queue persistence count and explain every difference as duplicate, filter, invalid, or capacity.
|
||||
10. If a run stopped, inspect checkpointed URLs and `retryFromStep`/pending graph state rather than treating the last visible live-feed entry as the whole run.
|
||||
|
||||
## 14. Short diagnosis of the current “it feels like they do not” symptom
|
||||
|
||||
The panel’s former label-only extraction path has been removed. All transport origins now converge on the background graph, so the graph’s `started`, `elements collected`, `complete`, `failed`, and `skipped` records describe the same execution mechanism for tab, scroll, drag/drop, paste, and crawler-produced URLs. Queue deduplication/filtering still happens after discovery, so “collected” and “queued” remain different populations.
|
||||
@@ -1988,9 +1988,9 @@ function startM3U8Collector(message: CollectorMessage) {
|
||||
void (async () => {
|
||||
try {
|
||||
const variants = await parseM3U8Variants(details.url);
|
||||
safeSendMessage({ type: 'panel-add-urls', urls: variants });
|
||||
forwardUrls(variants);
|
||||
} catch {
|
||||
safeSendMessage({ type: 'panel-add-urls', urls: [details.url] });
|
||||
forwardUrls([details.url]);
|
||||
}
|
||||
})();
|
||||
};
|
||||
@@ -2168,9 +2168,6 @@ chrome.runtime.onMessage.addListener((rawMessage: unknown, _sender, sendResponse
|
||||
: event.detail || (event.action === 'scrolling started' ? 'Beginning page scroll' : event.action === 'scroll complete' ? 'Scroll finished' : 'Page and initial DOM ready');
|
||||
appendFeed({ action: event.action, detail, timestamp: new Date().toISOString() });
|
||||
appendFeed(...event.urls.map((url) => ({ action: 'URL collected', detail: url, timestamp: new Date().toISOString() })));
|
||||
for (let offset = 0; offset < event.urls.length; offset += 500) {
|
||||
safeSendMessage({ type: 'panel-add-urls', urls: event.urls.slice(offset, offset + 500), crawlJobId: jobId });
|
||||
}
|
||||
currentUrlsCollected = event.totalUrls ?? currentUrlsCollected;
|
||||
emit('running', event.step, event.action, rawMessage.url, currentUrlsCollected);
|
||||
}, control, interactionOptions, enableInteractiveCollector, undefined, async (checkpoint) => {
|
||||
@@ -2383,10 +2380,7 @@ async function bookmarkUrl(url: string, title: string) {
|
||||
}
|
||||
|
||||
function queueDiscoveredUrl(url: string) {
|
||||
safeSendMessage({
|
||||
type: 'panel-add-urls',
|
||||
urls: [url],
|
||||
});
|
||||
forwardUrls([url]);
|
||||
}
|
||||
|
||||
async function runRedditSequenceIteration() {
|
||||
|
||||
Reference in New Issue
Block a user