In recent years, artificial intelligence (AI) has rapidly gained popularity, particularly due to its many applications across various sectors, including virtual assistants, automation, robotics, autonomous vehicles, medical diagnosis, data analysis, and much more. In particular, thanks to tools like ChatGPT, “Generative AI” has become increasingly popular and discussed, thanks to various applications and tools available to users, both free and paid.
What is Generative AI?
But what is meant by “Generative AI”? It refers to AI systems designed to autonomously generate content, such as text, images, music, or other forms, using machine learning approaches, particularly generative neural networks, to produce output that can appear creative and capable of imitating human styles. These technologies have applications in various fields, including generative art, multimedia content creation, scenario simulation, automatic text generation, and more. However, it is important to note that, despite their successes, there are also ethical challenges associated with the use of Generative AI, such as the potential to create false or manipulated content.
Dataiku
But how can companies leverage and use these new tools for their operations? Many machine learning and data science platforms are striving to answer this question, and Dataiku is among them. Dataiku is a platform designed to quickly prototype advanced machine learning and AI models, integrating tools for deploying and managing these models and projects. The platform offers a collaborative environment where the entire team, both technical and non-technical, can contribute to building data pipelines, working together to explore, prepare, and analyze data, and create and implement machine learning models. It is an evolving platform that releases new features through periodic updates, keeping pace with market demands.
Dataiku and Generative AI
Regarding Generative AI, Dataiku has incorporated features in its recent versions that help users leverage the potential of Generative AI and provide various materials and use cases on the subject. Specifically, there is a new feature called LLM Mesh, where LLM stands for Large Language Model. It is a set of tools that allows for the supervision, governance, and centralization of LLM projects, addressing common obstacles in applying these models, such as:
- Connecting to a large number of large language models via APIs and locally, for example, OpenAI or Google Vertex PaLM;
- Full support for locally hosted HuggingFace models;
- Integrated support for Retrieval Augmented Generation (Models?), through connectors and other tools;
- Toxicity testing, which in this context refers to the model’s ability to generate offensive, discriminatory, or inappropriate output;
- Cost monitoring;
- With LLM Mesh, it is possible to build and “orchestrate” a pipeline of text files (including PDFs) and provide responses to prompt questions as output.
To support pipeline construction, there is a prompt studio available where users can test questions using different LLM models and select the most appropriate one.
Additionally, two new nodes have been released that allow for two of the most common LLM-based tasks: text classification and document summarization, not forgetting the already available features for text analysis and data cleaning.
Blue BI and Dataiku
Blue BI, a Dataiku partner for several years, can guide your company in adopting the platform and support you in its use, involving all teams from technical to non-technical. Blue BI can assist and support you in prototyping your first classic machine learning models and, why not, even very advanced Generative AI models.
We realize Business Intelligence & Advanced Analytics solutions to transform simple data into information of freat strategic value.
