I am working on a simple search engine.
Search engine have 2 modules: Search result suggester and ranker.

Suggester: suggest many search results.
Ranker: ranks the output of suggester before sending to user. (Mostly based on RL)

Search result suggester pools search results from many sub-suggesting systems. Sub-suggesters select search results from a common set of documents and shared data with one another(feedback data). Currently many similar sub-suggesters exist and cannibalise one-another. For example if 1 sub-suggester suggest broad documents for exploration and another sub-suggester use the click data generated by 1st sub-suggester to suggest only explored documents on similar queries, 2nd sub-suggester will looks really well but its actually cannibalising the first sub-suggester.

I want to know, how can I test between 2 sub-suggesters without separating their data.

I cannot split the queries or documents between the sub-suggesters also.