You’ve probably noticed that generative AI tools like Google Gemini and ChatGPT are making their way into most of the technology we operate every day. These tools rely on giant gigantic language models (LLMs): networks trained on massive amounts of human data so they can spit out realistic text or images or video.
You don’t need a cloud app to access these LLMs – you can also run them on your computer. You can operate everything these models offer when you’re offline and don’t have to hand over your prompts and conversations to devices like Google or OpenAI.
Now Nvidia has launched its own local LLM application – harnessing the power of the RTX 30 and RTX 40 series graphics cards – called Chat with RTX. If you have one of these GPUs, you can install the generative AI chatbot directly on your computer and customize it to suit your needs.
How to start
Before you get started, make sure you’re using the latest Nvidia graphics drivers – the GeForce Experience app on your PC will support – then go to Talk to the RTX download page. To run the tool, you need Windows 11, a GeForce RTX 30/40 series graphics card (or an RTX Ampere or Ada Generation graphics card with at least 8 GB of VRAM) and at least 16 GB of RAM.
Please note that Nvidia refers to this application as a “demo application”, which we assume means that it is not fully completed (some bugs and glitches may appear as a result). It’s also a gigantic download, weighing in at around 35GB, as it contains several LLM files – worth keeping in mind if you’re constrained on disk space or internet speed. The installation tool also takes some time to complete all its tasks.
Chat with RTX takes place from your computer.
Source: Lifehaker
Eventually, you should find the Chat with RTX app added to your Start menu. Run it, and after a few moments of processing, the program’s interface will appear in your default web browser. In the top left corner you can select the open source AI model you want to operate: Mistral Or Lama. Once you’ve done this, you can start sending notifications just like you would with ChatGPT or Google Gemini.
If you’ve used LLM before, you know what these generative AI engines are capable of: get support creating emails, documents, text messages, and more, simplify complicated topics, or ask questions you might otherwise lead to an internet search ( e.g. “What is a good party game for four adults?”).
The application also keeps the command line window open.
Source: Lifehaker
Standard AI bot rules apply – try to be as specific and detailed as possible, and be careful not to place too much trust in the answers you receive (especially since this is a “demo”). Chat with RTX cannot search for current information on the Internet, so it is not suitable for creating replies that need to be up-to-date, but it will always try to provide an answer based on the mass of online text that has been sent trained.
At the bottom of the interface there is a button to generate a fresh reply based on the last prompt (if you are not completely satisfied with the current prompt), an undo button (to return to the previous prompt), and a delete chat button which will clear your chat history so you can get started all over again. There is currently no way to export responses other than copying and pasting the text.
Add your own data and YouTube videos
Even in this early form, Chat with RTX has some useful tricks, one of which is the ability to base your responses on the documents you provide: maybe a week’s worth of research, or a series of reports you need to analyze, or even all the fanfic you write. Under Data set header, select Folder paththen point the program towards the folder containing the documents you want to operate.
The app will scan the folder you specify (which may take a minute or two if there are a lot of files in it) and you can start entering queries. The bot will scan the text for appropriate answers and then name the file(s) it is using at the end of the answer. You can request summaries, check facts, or ask the bot to generate fresh text based on the text you give it.
Chat with RTX can also work with your own files.
Source: Lifehaker
Again, this is an early version of technology that is known to be less than 100 percent exact – management decisions should not be based on it yet. However, it’s fun to play with an LLM who can work from the documents you give him, whether they’re interview transcripts or books of poetry.
Speaking of transcriptions, Chat with RTX can also analyze YouTube videos and offer responses based on them via transcripts linked to the clips. (Based on our testing, it can automatically generate transcripts for videos that don’t already contain them.) This even works with entire YouTube playlists, so you can run the program through a whole series of clips at once.
You can also point the program towards YouTube videos.
Source: Lifehaker
To choose YouTube URL as a dataset, then paste the video or playlist address into the box below. If you’re working with a playlist, you can specify the number of videos you want to include in the box on the right. Finally, click the download button on the right and Chat with RTX will download and process the transcript text, ready for any prompts.
As with the document scanning feature, it can be useful for getting summaries or selecting snippets of essential information, but we found this to be the most buggy part of the program – a program that, frankly, is marked as version 0.2. The app sometimes didn’t know which video it was talking about, but if you need quick answers about long videos and playlists that you don’t have time to watch, Chat with RTX can be a useful tool.
