What was an experiment in the Marcel Proust's In Search of Lost Time (Volume 1, Swann's Way) and André Gide's The Counterfeiters?
Your question is worded a little ambiguously. Since both Proust's and Gide's works are essentially all about continual and various social experiments, I'm going to conclude that what you are really asking about is the innovative experimental characteristics of the works themselves: "How are Proust's In Search of Lost Time and Gide's The Counterfeiters experimental works of literature?" If this is a mistaken understanding, please do clarify your query and post it in another Question.
Proust's work is described as a novel in seven volumes, seven large volumes. The first and most often studied volume is Swann's Way (1922). Proust himself planned to write an experimental novel. In the Introduction written by Richard Howard (in epistolary form, i.e., like a letter), Proust is quoted as saying, while contemplating Search, that if he writes, what he writes may not actually be called a novel and that it will diverge from what he has previously written:
"Work while you still have the light...." Does this mean I am going to write a novel? How should I know? I don't know if it will be possible still to cal a "novel" the work which I desire to write and which I expect to break with the nature of my previous writings.
In this work, Proust sought to define characters by their innermost feelings and perceptions, perceptions so deeply ingrained and perhaps smothered that it is only within our "inmost" being that these parts of self can be examined and, as it were, "reconstructed":
If we mean to try to understand this self it is only in our inmost depths, by endeavoring to reconstruct it there, that the quest can be achieved. (Proust, quoted in D. J. Enright's introductory note)
Thus the experimental nature of Proust's work is that he seeks to find, identify, and reconstruct the innermost beings of the characters he examines. This reconstruction comes about through having characters, particularly the few principal characters, reveal in-depth psychological detail about their perceptions, feelings, reactions, emotions and thoughts.
Gide's work revolves around individuals who are in one form or another thought of as social or personal counterfeits, a perception metaphorically represented by the money counterfeiting ring that operates through the efforts of children (which itself symbolizes the deep-rooted beginnings of counterfeit lives).
What is experimental in Gide's work is (1) the density of characters, (2) the intensity of the interconnectedness between characters and (3) the quantity of violations of social norms and cultural mores that underlie the seemingly normal lives of the characters, in other words, the number of violations that make all the characters' normal lives counterfeit lives. While novels exposing violations of social norms were not new, for example, Zola's realist Nana (1880), the scope and intensity of the social violations presented by Gide was new; it was indeed experimental as were the pervasive relationships between so many characters.
The very beginning of the novel alludes to the upcoming complexity of counterfeit character relationships by introducing Bernard's family, his friend Oliver and his newly discovered connection to an unknown father (making Bernard a counterfeit from the first page):
"The time has now come for me to hear a step in the passage," said Bernard to himself. He raised his head and listened. Nothing! ... "There no better cure for the fear of taking after one's father than not to know who he is. ... The only thing to do is to welcome deliverance .... When His Honour [the Judge (adoptive father)] comes in he must find a letter from me on his writing table, informing him in eloquent terms of my departure."