You shouldn’t trust a government-run chatbot to give you good advice

Up-to-date York’s “MyCity” AI chatbot is off to a rocky start. City officials implemented the technology five months ago in an effort to assist residents interested in running a business in the Large Apple find useful information.

Although the bot will be content to answer your questions with seemingly reasonable answers, An investigation by The Markup discovered that the bot was lying-Very. For example, when asked whether an employer can collect a portion of its employees’ tips, the bot replies yes, even though the law says so Bosses cannot accept tips from employees. When asked if buildings must accept Section 8 vouchers, the bot replies no Landlords cannot discriminate based on a potential tenant’s source of income. When asked if you can make your store cashless, the bot answers: go ahead, but in reality Cashless establishments have been banned in Up-to-date York since early 2020 – when it says “there is no law in Up-to-date York that requires businesses to accept cash as a form of payment” it is full of bullshit.

To the city’s credit, the site warns users not to rely solely on chatbot responses rather than professional advice and to verify any statements via the links provided. The problem is that some replies contain no links at all, making it even more complex to check whether what the bot is saying is factual. The question arises: who is this technology for?

AI is prone to hallucinations

This story will not be shocking to anyone who follows recent developments in the field of artificial intelligence. It turns out that chatbots just make things up sometimes. This is called hallucination: AI models trained to answer user queries will surely conjure up an answer based on training data. Because these networks exist So complicated, it’s demanding to know exactly when and why a bot will decide to post a specific piece of fiction in response to your question, but it happens often.

This isn’t really Up-to-date York defect that their chatbot is hallucinating that you can extort tips from your employees: their bot keeps working Microsoft’s Azure Artificial Intelligence, a common artificial intelligence platform used by companies like AT&T, Reddit, and Volkswagen to provide a variety of services. The city likely paid for access to Microsoft’s artificial intelligence technology to power the chatbot in a earnest effort to assist Up-to-date Yorkers interested in starting a business, only to find the bot hallucinating wildly wrong answers to vital questions.

When will the hallucinations stop?

It is possible that these unfortunate situations will soon be behind us: Microsoft has a recent security system to catch and protect customers from the darker sides of artificial intelligence. In addition to tools that assist block hackers from using artificial intelligence as a malicious tool and assess potential vulnerabilities in AI platforms, Microsoft is introducing Ground detection, which can monitor potential hallucinations and intervene if necessary. (“Ungrounded” is another word for hallucination.)

When Microsoft’s system detects a possible hallucination, it can allow customers to test the current version of AI against the version that existed before its implementation; point out the hallucinatory statement and either fact-check or engage in “knowledge base editing”, which presumably allows the core training set to be edited to eliminate the problem; rewrite the hallucinatory statement before sending it to the user; or evaluate the quality of synthetic training data before using it to generate recent synthetic data.

Microsoft’s recent system runs on a separate LLM called natural language inference (NLI), which continuously evaluates AI claims based on source data. Of course, since the fact-checking system in the LLM is itself an LLM, couldn’t the NLI be hallucinating about its own analysis? (Probably! Kidding, kidding. Kind of.)

This could mean that organizations like Up-to-date York that run their products on Azure AI will be able to address this LLM in real time to get rid of hallucinations. Perhaps when the MyCity chatbot tries to say that you can conduct business cashless in Up-to-date York, PIP will quickly correct the claim so that what you see as an end user is the true, correct answer.

Microsoft has only just rolled out this recent software, so it’s not yet clear how well it will work. But for now, if you’re a Up-to-date York resident or someone using a government chatbot to find answers to legitimate questions, you should take these answers with a grain of salt. I don’t think “MyCity chatbot said I could!” he will hold up in court.