Context-aware Visual Attention-based (CoVA) webpage object detection pipeline

Context-Aware Visual Attention-based end-to-end pipeline for Webpage Object Detection (CoVA) aims to learn function f to predict labels y = [y₁, y₂, ..., y_N] for a webpage containing N elements. The input to CoVA consists of: 1. a screenshot of a webpage, 2. list of bounding boxes [x, y, w, h] of the web elements, and 3. neighborhood information for each element obtained from the DOM tree.

This information is processed in four stages: 1. the graph representation extraction for the webpage, 2. the Representation Network (RN), 3. the Graph Attention Network (GAT), and 4. a fully connected (FC) layer.

The graph representation extraction computes for every web element i its set of K neighboring web elements $N_i_. The _RN_consists of a Convolutional Neural Net (_CNN_) and a positional encoder aimed to learn a visual representation _v_i_for each web element ᵢ isin; 1, ..., N_. The _GAT_combines the visual representation _v_iₒf the web element ᵢₜo be classified and those of its neighbors, i.e., _v_k$ ∀k ∈ $N_iₜo compute the contextual representation _c_i$ for web element i. Finally, the visual and contextual representations of the web element are concatenated and passed through the FC layer to obtain the classification output.