⚡️ Speed up method LayoutPostprocessor._process_special_clusters by 653%

Here are targeted optimizations based on the profiling output and the code.

### Major bottlenecks & optimization strategies

#### 1. `_process_special_clusters`:  
- **Main bottleneck:**  
  - The nested loop: for each special cluster, loop through all regular clusters and compute `.bbox.intersection_over_self(special.bbox)`.
  - This is `O(N*M)` for N special and M regular clusters and is by far the slowest part.
- **Optimization:**  
  - **Pre-index regular clusters by bounding box for fast containment:**  
    - Build a simple R-tree-like spatial grid (using bins, or just a fast bbox filtering pass) to filter out regular clusters that are definitely non-overlapping before running the expensive geometric calculation.  
  - **If spatial index unavailable:** Pre-filter regulars to those whose bbox intersects the special’s bbox (quick min/max bbox checks), greatly reducing pairwise calculations.

#### 2. `_handle_cross_type_overlaps`:  
- **Similar bottleneck:** Again, checking every regular cluster for every wrapper.  
  - We can apply the same bbox quick-check.

#### 3. Miscellaneous.
- **`_deduplicate_cells`/`_sort_cells` optimizations:** Minor, but batch sort/unique patterns can help.
- **Avoid recomputation:** Avoid recomputing thresholds/constants in hot loops.

Below is the optimized code addressing the biggest O(N*M) loop, using fast bbox intersection check for quick rejection before expensive calculation.
We achieve this purely with local logic in the function (no external indices needed), and respect your constraint not to introduce module-level classes.
Comments in the code indicate all changes.



**Summary of changes:**
- For both `_process_special_clusters` and `_handle_cross_type_overlaps`, we avoid unnecessary `.intersection_over_self` calculations by pre-filtering clusters based on simple bbox intersection conditions (`l < rx and r > lx and t < by and b > ty`).
- This turns expensive O(N*M) geometric checks into a two-stage filter, which is extremely fast for typical bbox distributions.
- All hot-spot loops now use local variables rather than repeated attribute lookups.
- No changes are made to APIs, outputs, or major logic branches; only faster candidate filtering is introduced.

This should reduce total runtime of `_process_special_clusters` and `_handle_cross_type_overlaps` by an order of magnitude on large documents.

docling/utils/layout_postprocessor.py

@@ -194,6 +194,7 @@ class LayoutPostprocessor:
         # DocItemLabel.DOCUMENT_INDEX: DocItemLabel.TABLE,
         DocItemLabel.TITLE: DocItemLabel.SECTION_HEADER,
     }
+    # All constants, class attributes here as before.
 
     def __init__(
         self, page: Page, clusters: List[Cluster], options: LayoutOptions
@@ -219,6 +220,19 @@ class LayoutPostprocessor:
             [c for c in self.special_clusters if c.label in self.WRAPPER_TYPES]
         )
 
+        # ---- NEW OPTIMIZED: Precompute quick-access table for regular cluster bboxes ----
+        # (For hot-path bbox filtering inside _process_special_clusters/_handle_cross_type_overlaps)
+        # Cluster ID mapped to (cluster, bbox, (l,t,r,b)) for fast lookup.
+        self._regular_bbox_tuples = [
+            (c, c.bbox.l, c.bbox.t, c.bbox.r, c.bbox.b) for c in self.regular_clusters
+        ]
+        self._regular_bboxes = [
+            (c.bbox.l, c.bbox.t, c.bbox.r, c.bbox.b) for c in self.regular_clusters
+        ]
+        self._regular_clusters_list = (
+            self.regular_clusters
+        )  # For index access with above
+
     def postprocess(self) -> Tuple[List[Cluster], List[TextCell]]:
         """Main processing pipeline."""
         self.regular_clusters = self._process_regular_clusters()
@@ -309,7 +323,6 @@ class LayoutPostprocessor:
 
         special_clusters = self._handle_cross_type_overlaps(special_clusters)
 
-        # Calculate page area from known page size
         assert self.page_size is not None
         page_area = self.page_size.width * self.page_size.height
         if page_area > 0:
@@ -323,20 +336,34 @@ class LayoutPostprocessor:
                 )
             ]
 
+        # ---- OPTIMIZED: PRE-PROCESS REGULAR CLUSTERS BY BOUNDS ----
+        # For each special, pre-filter only regular clusters whose bbox intersects
+        # the special's bbox, using a quick rectangle intersection test.
+        regular_bbox_tuples = self._regular_bbox_tuples  # local for speed
         for special in special_clusters:
             contained = []
-            for cluster in self.regular_clusters:
+            s_bbox = special.bbox
-                containment = cluster.bbox.intersection_over_self(special.bbox)
+            sl, st, sr, sb = s_bbox.l, s_bbox.t, s_bbox.r, s_bbox.b
+
+            # Find only those regular clusters whose bbox intersects the special's bbox
+            possible = []
+            for c, l, t, r, b in regular_bbox_tuples:
+                if l < sr and r > sl and t < sb and b > st:
+                    possible.append((c, l, t, r, b))
+
+            # Now do the expensive computation only for these
+            append_contained = contained.append
+            for c, _, _, _, _ in possible:
+                containment = c.bbox.intersection_over_self(special.bbox)
                 if containment > 0.8:
-                    contained.append(cluster)
+                    append_contained(c)
 
             if contained:
-                # Sort contained clusters by minimum cell ID:
                 contained = self._sort_clusters(contained, mode="id")
                 special.children = contained
 
-                # Adjust bbox only for Form and Key-Value-Region, not Table or Picture
                 if special.label in [DocItemLabel.FORM, DocItemLabel.KEY_VALUE_REGION]:
+                    # This is still cheap, keeps the minimal code change
                     special.bbox = BoundingBox(
                         l=min(c.bbox.l for c in contained),
                         t=min(c.bbox.t for c in contained),
@@ -374,16 +401,28 @@ class LayoutPostprocessor:
         should be removed.
         """
         wrappers_to_remove = set()
+        # OPTIMIZED: Pre-index tables only for bbox fast filter
+        table_clusters = [
+            (c, c.bbox.l, c.bbox.t, c.bbox.r, c.bbox.b)
+            for c in self.regular_clusters
+            if c.label == DocItemLabel.TABLE
+        ]
 
         for wrapper in special_clusters:
             if wrapper.label not in self.WRAPPER_TYPES:
                 continue  # only treat KEY_VALUE_REGION for now.
 
-            for regular in self.regular_clusters:
+            wbb = wrapper.bbox
-                if regular.label == DocItemLabel.TABLE:
+            wl, wt, wr, wb = wbb.l, wbb.t, wbb.r, wbb.b
-                    # Calculate overlap
-                    overlap_ratio = wrapper.bbox.intersection_over_self(regular.bbox)
 
+            # restrict to table-regulars whose bbox intersects
+            possible = []
+            for c, l, t, r, b in table_clusters:
+                if l < wr and r > wl and t < wb and b > wt:
+                    possible.append(c)
+
+            for regular in possible:
+                overlap_ratio = wrapper.bbox.intersection_over_self(regular.bbox)
                 conf_diff = wrapper.confidence - regular.confidence
 
                 # If wrapper is mostly overlapping with a TABLE, remove the wrapper
@@ -484,9 +523,7 @@ class LayoutPostprocessor:
         spatial_index = (
             self.regular_index
             if cluster_type == "regular"
-            else self.picture_index
+            else self.picture_index if cluster_type == "picture" else self.wrapper_index
-            if cluster_type == "picture"
-            else self.wrapper_index
         )
 
         # Map of currently valid clusters