It’s been publicized a few times that AI chat communications have shown up on Google search. People think because they are talking with a Large Language Model (or AI) in a window that it’s between them and the model, but it’s not. At least, it isn’t right now. Government is working on that problem:
At present, most digital interactions fall under the Third-Party Doctrine, which holds that information voluntarily disclosed to other parties — or stored on a company’s servers — carries ‘no legitimate expectation of privacy.’ This doctrine allows government access to much online behavior (such as Google search histories) without a warrant…. To leave these conversations legally unprotected is to invite a regime where citizens must fear that their digital introspection could someday be used against them.