A strategic Overview for Decision Makers

Entering the realm of big data solutions marks a transformative step for any organisation, demanding a blend of strategic foresight, meticulous planning, and a deep dive into technological possibilities. This guide is crafted for the astute manager and decision maker poised at the brink of this significant transition, often navigating without extensive prior knowledge of big data technologies. You’re not just choosing a technological path; you’re setting the course for how data can redefine your organizational landscape. From preparation to engagement, this guide demystifies the journey, enabling you to chart a course that not only aligns with your organisational goals but propels them to new heights.

Laying the Groundwork for Transformation

Deciphering Your Big Data Ambitions

Understanding and articulating your needs and objectives is the cornerstone of a successful big data strategy. This clarity of purpose guides you through the myriad options and solutions available, ensuring that your choices resonate with the unique contours of your organizational goals. Dive into this analysis by considering two primary dimensions:

Addressing Pains, Problems, and Bottlenecks 🛠️

Your first beacon is the existing landscape of challenges that hinder your operational efficiency or strategic growth. Big data solutions offer a powerful lens to not only view these issues in high resolution but to navigate through them with precision. Whether it’s through enhancing decision-making capabilities, streamlining operations, or uncovering insights buried within your data, the aim here is to dismantle these barriers, paving the way for a smoother organizational journey.

• Identify Specific Challenges: Begin with a granular analysis of operational pain points, inefficiencies, or data-related hurdles your organization faces.
• Blueprint for Resolution: Map out how big data technologies could potentially offer solutions, from predictive analytics to advanced data management systems.

Optimising Potential Use Cases 💡

Beyond the immediate resolution of problems, big data opens a vista of opportunities to enhance, innovate, and lead. This proactive stance looks at not just where you are, but where you could be. It’s about leveraging data to its fullest potential, identifying areas where analytics, machine learning, and data-driven strategies can introduce efficiencies, innovations, and competitive advantages.

• Innovation and Improvement: Pinpoint areas within your organization where data analytics could bring about transformative change or offer a competitive edge.
• Predictive and Prescriptive Analytics: Explore how utilizing big data for forecasting and strategic advice could streamline operations, enhance customer experience, or optimize resource allocation.

By integrating these two approaches, you create a comprehensive picture of your big data aspirations, capturing both the remedial and the visionary. This dual lens not only addresses where you stand today but positions you to leap forward, turning potential into reality. Remember, the most effective big data solutions often serve a dual purpose: rectifying existing issues while unlocking new opportunities.

Enhance Your Understanding Through Our Complimentary Webinar

We recognise the complexity of navigating big data solutions and are committed to empowering decision-makers. Our complimentary webinars delve into the wide-ranging applications of big data and the essential considerations for decision-makers, without promoting our services. These informative sessions are tailored to your organization’s specific interests and needs, aiming to bolster your big data strategy. Contact us to arrange a webinar designed to empower your team with critical insights.

Exclusive Video Resources

In addition to our webinars, we offer concise, impactful video guides on crucial topics:

• Mastering Machine Learning Projects: Explore the essentials of leading Machine Learning (ML) projects, designed for managers with minimal data backgrounds. This guide simplifies ML project management, covering team roles, goal setting, data quality, and more. Watch the video.
• Understanding Data Quality: Dive into the significance of data quality, a critical factor in the success of any project. This Q&A video addresses common questions, highlighting the risks, benefits, and responsibilities associated with data quality. Explore the video.

Assessing Your Current Infrastructure

Understanding your current technological landscape is crucial for identifying the need for big data solutions. This involves a thorough inventory and assessment of your existing data systems, technologies, and infrastructure. Here’s how to approach this step effectively:

• Conduct an Inventory: Engage with your IT department or technology team to catalogue all current data systems, software, and technologies in use. If your organization lacks a dedicated IT team, consider consulting with external IT professionals or technology auditors.
• Identify Limitations: Work closely with your IT team to highlight any challenges or limitations in your current setup, including data processing capabilities, storage issues, or scalability concerns. This step is vital for understanding the gaps that big data solutions can fill.
• Consult Various Departments: Different departments may utilize distinct systems or have unique insights into the limitations of the current infrastructure. Ensure to gather input from across the organisation for a holistic view.

Establishing a Budget Framework

Setting a realistic budget for big data solutions requires an understanding of both your financial capacity and the market costs of these technologies. Here’s how to navigate this:

• Assess Financial Capabilities: Review your organization’s financial resources, considering both current budgets and potential future investments. Collaboration with your finance department is key to accurately determining your spending capacity.
• Research Market Costs: Given the variability and complexity of big data solution pricing, gathering reliable cost information can be challenging. To navigate this, consider:
  • Consulting with Industry Peers: Reach out to your professional network or industry groups to gain insights into the costs they have encountered.
  • Leverage Expert Guides: To receive our comprehensive budget guide detailing price ranges for various big data projects, simply send us an email with the subject ‘Budget Guide’. This indispensable resource is tailored to assist you in setting realistic expectations aligned with current market standards. By requesting this guide, you’ll gain insight into budgeting for your big data initiatives and will be automatically subscribed to our newsletter, keeping you informed on the latest trends and insights in big data solutions. Please note that subscribing to our newsletter is part of this process, offering you ongoing value beyond the guide.

Understanding Your Data

A deep understanding of your organization’s data is fundamental to selecting the right big data solution. This involves:

• Evaluating Data Characteristics: Work with your data management team or data scientists to assess the volume, variety, velocity, and veracity of the data your organization handles. If such expertise is not available in-house, consider engaging with data consultants.
• Considering Compliance and Privacy: Ensure you are aware of any industry-specific data privacy or regulatory compliance requirements. This might involve consulting with your legal department or external legal advisors specializing in data protection laws.

Engaging Your Stakeholders

Incorporating stakeholder input is crucial for ensuring alignment and support for big data initiatives. This step should be integrated with identifying your needs and objectives to ensure a cohesive strategy:

• Gather Diverse Insights: Engage with stakeholders from various departments to compile a comprehensive list of needs, expectations, and concerns. This can include executives, department heads, and end-users who will interact with the big data solutions.
• Facilitate Collaborative Discussions: Organize workshops or meetings that bring together these stakeholders to discuss and refine the objectives and expectations for implementing big data solutions. This collaborative approach ensures that the project aligns with organizational goals and addresses key pain points effectively.

By following these detailed steps, you’ll be well-equipped to prepare for engaging with big data solution providers, ensuring that your organisation’s needs are accurately identified and that you have a solid foundation for making informed decisions.

Initiating Contact with Providers

Engaging with the right big data solution providers is a critical step towards implementing a system that aligns with your organization’s needs. Here’s how to navigate this process effectively:

Research and Shortlist Providers

To find providers that not only offer big data solutions but also understand your industry’s unique challenges, follow these steps:

• Utilize Industry Forums and Reviews: Platforms like Clutch are invaluable for identifying reputable providers, offering a wealth of user reviews and ratings. This firsthand feedback can give you insights into the providers’ strengths and weaknesses.
• Evaluate Expertise and Scalability: Look for providers that have demonstrated expertise in your industry. Those recognized as experts often contribute to the community by speaking at events, leading workshops, or publishing insightful case studies. This involvement can be a good indicator of their capability to scale solutions to meet your needs.
• Explore Provider Portfolios: Delve into the websites of potential providers to gather detailed information about their services, client success stories, and case studies. This exploration can offer a clearer picture of their capabilities and the types of solutions they’ve successfully implemented in the past.
• Network for Recommendations: Tap into your professional network for personal insights and recommendations. Firsthand accounts of experiences with big data solution providers can be invaluable in shortlisting candidates.

Reach Out

Once you have a shortlist of potential providers, reaching out effectively is key:

• Prepare a Brief: Compile a concise overview of your organization, including its needs, objectives, and any specific challenges you hope to address with big data solutions. This brief should summarize the critical information providers need to understand your project.
• Initiate Contact: Use the contact information on providers’ websites to share your brief. This initial outreach is your opportunity to make a strong first impression and lay the groundwork for productive discussions. It’s efficient to send this brief to multiple providers to broaden your options.

Schedule Initial Consultations

Engaging in consultations with several providers allows you to compare and contrast their approaches, expertise, and the solutions they offer:

• Consult with Multiple Providers: Aim to schedule initial consultations with 3-5 providers. This range gives you a broad perspective while remaining manageable. Each provider may offer unique insights, benefits, or approaches that could be crucial for your project.
• Communicate Your Availability: Be clear about when you’re available for these discussions. Providing a range of dates and times can help expedite scheduling.
• Inquire About Scope and Involvement: During scheduling, ask providers to outline what will be covered during the initial consultation. Also, inquire about the estimated timeline for project completion and the level of involvement required from your team. Understanding these factors upfront can help you prepare for the consultations more effectively.

By thoroughly researching potential providers, preparing a clear and concise brief, and strategically scheduling initial consultations, you’re setting the stage for productive engagements that will lead to identifying the best big data solution for your organization. This process ensures that you have all the information needed to make an informed decision, aligning with your strategic objectives and operational needs.

Making Your Selection

After engaging with potential providers and receiving their proposals, the next step is a critical analysis to determine the best fit for your organization. This phase involves careful consideration of how each proposal aligns with your strategic needs, budgetary constraints, and long-term objectives. Here’s a structured approach to making your selection:

Compare Proposals Against Your Criteria

• Align with Needs and Objectives: Systematically compare each proposal’s offerings against the specific needs and objectives you’ve identified for your big data project. This includes considering how well each solution addresses your current challenges and facilitates your strategic goals.
• Budget Compatibility: Assess each proposal in the context of your budget framework. Consider not only the upfront costs but also any ongoing expenses associated with the solution, such as licensing fees, data storage costs, and additional services.
• Strategic Goals Alignment: Evaluate how each provider’s solution aligns with your broader strategic goals. This could involve considerations of innovation potential, market competitiveness, and the ability to adapt to future challenges.

Evaluate Long-term Benefits and Scalability

• Scalability: Ensure the proposed solution can grow and adapt as your organization’s needs evolve. This includes the ability to handle increasing volumes of data, integrate new data sources, and expand in functionality.
• Support and Maintenance: Consider the level and quality of support and maintenance services offered. Reliable post-implementation support is crucial for resolving any issues that arise and ensuring the longevity of your big data solution.
• Long-term Value: Beyond the immediate solution, assess the long-term benefits each provider offers. This could include ongoing updates, training opportunities for your team, and access to additional resources or community support.

Achieve Stakeholder Consensus

• Engage Stakeholders: Present the analyzed proposals to your key stakeholders, highlighting how each aligns with the organization’s needs and objectives. This presentation should be detailed and include your recommendations based on the evaluation criteria.
• Facilitate Discussion: Encourage open discussion among stakeholders to hear their perspectives, concerns, and preferences. This collaborative approach ensures that the decision-making process is inclusive and considers the insights and needs of different departments or teams.
• Consensus Building: Work towards building a consensus on the provider that best matches your organization’s vision, objectives, and budget. This may involve negotiating compromises or revisiting specific criteria to ensure alignment with organizational priorities.

Finalise Your Selection

Once a consensus is achieved, proceed with finalizing your selection of the big data solution provider. This involves reaching out to the chosen provider to begin the negotiation of terms, finalizing the contract, and planning the implementation phase. Ensure that all legal and financial considerations are thoroughly reviewed and agreed upon before finalizing the agreement.

By following this structured approach, you ensure that your organization selects a big data solution provider that not only meets your current needs but also positions you for future success. This careful selection process paves the way for a fruitful partnership and a successful big data initiative.

The Consultation Process

Engaging with potential big data solution providers through consultations is a pivotal part of the selection process. Here’s what you should expect and how to navigate these meetings effectively:

What to Expect:

• Initial Meeting: This first encounter is an opportunity to discuss your organization’s specific needs and objectives in detail and to receive an initial overview of the provider’s solutions. Approach this meeting as a chance to see how well the provider understands your challenges and whether they have the expertise to address them.
• Follow-Up Meetings: These sessions often delve deeper into the technical aspects of the proposed solutions. Expect to engage in discussions about customizing solutions to fit your needs, presentations from technical specialists, and possibly even demos of the solutions. This phase is crucial for assessing the provider’s capability to tailor their offerings to your requirements.
• Proposal Review: After these discussions, providers will present a formal proposal. This document should outline the solution in detail, including implementation timelines, costs, and the framework for how the solution will be integrated into your existing infrastructure.

Key Questions to Ask:

To ensure you gather all the necessary information to make an informed decision, consider asking the following questions:

• Industry Experience: “Can you share examples of your work with similar companies or within our industry?”
• Solution Alignment: “How do your solutions specifically address our needs and objectives?”
• Case Studies and References: “Could you provide case studies or references from past projects similar to ours?”
• Implementation Timeline: “What is the expected timeline for the full implementation of your solution?”
• Pricing Structure: “Can you provide a detailed explanation of your pricing structure, including any potential additional costs we should anticipate?”
• Data Security and Compliance: “How does your solution ensure data security and compliance with relevant regulations?”
• Support and Maintenance: “What kind of post-implementation support and maintenance services do you offer?”

Preparing Your Data for Consultation

To effectively engage with big data solution providers, having a clear understanding of your data landscape is essential. Here’s how to prepare:

• Data Inventory: Conduct a comprehensive review of the types of data your organization processes. This task is often best managed by your IT department or data management team. If such expertise is not readily available within your organization, consider hiring external data consultants to assist.
  • Actionable Steps: Start by listing all data sources, types, and storage locations. Categorize data based on sensitivity, usage frequency, and format. This inventory will be crucial for providers to understand the scope of your data and how their solutions can be applied.
• Identify Data Challenges: Clearly outline any issues your organization faces regarding data management, analysis, or security. This step is crucial for finding a solution that addresses these challenges effectively.
  • Who to Consult: This information might be spread across various departments, such as IT for technical challenges, legal for compliance issues, and operational teams for workflow-related challenges. Consolidate this information to present a comprehensive view of the data challenges to the provider.
• Growth Projections: Anticipate the future scale of your data needs to ensure the proposed solution can accommodate growth. This includes estimating increases in data volume, variety, and velocity.
  • Collaboration is Key: Engage with strategic planning or business development teams to forecast future data growth based on business projections. This foresight ensures the solution remains viable as your organization grows.

By thoroughly preparing for the consultation process and clearly understanding your data landscape, you set the stage for productive discussions with potential providers. This preparation enables you to ask informed questions, accurately assess proposals, and ultimately choose a solution that best fits your organisation’s needs.

Conclusion

By following this guide and leveraging the provided resources, you’ll be well-equipped to engage with big data solution providers effectively. The path to a successful partnership and implementation is paved with preparation, open communication, and a deep understanding of your organizational goals. Our webinars and video resources are here to further enhance your knowledge and confidence in navigating the big data landscape.

The post Navigating Big Data Solution Adoption: A Managerial Guide appeared first on TantusData.

Optimising Digital Efficiency: A Corporate Imperative

In today’s digital-driven corporate landscape, the environmental footprint of our online activities, which encompass cloud services, computing, and the broader internet infrastructure, often escapes scrutiny. These critical components of modern business operations are not merely sources of operational inefficiency and financial expenditure; they significantly contribute to environmental degradation. Therein lies a pivotal call to action for businesses globally: to reassess and refine their digital infrastructure, spanning from the intricacies of cloud storage systems to the depths  of computing processes. By elevating digital cleanliness to a strategic priority, organisations stand to not only meet emerging standards of digital sustainability reporting—an accolade in corporate environmental stewardship—but also unveil substantial avenues for cost savings. Diving deeper into the ethos of digital sustainability reveals its capacity to forge a future that is not only environmentally harmonious but also streamlined for greater efficiency and financial prudence.

It’s crucial to acknowledge that certain areas prone to digital wastefulness, including redundant data storage, inefficient computing practices, and overlooked aspects of cloud utilisation, may remain hidden beneath the surface of day-to-day operations. There are also those which are even less obvious such as less energy efficient programming languages. These undercurrents of inefficiency represent missed opportunities for optimisation and sustainability. Importantly, while digital sustainability reporting is not currently a mandated practice in triple reporting, it serves as a powerful testament to an organisation’s commitment to environmental stewardship and technological foresight. Embracing this proactive approach not only positions a company as a leader in ecological responsibility but also as a visionary in harnessing the potential for long-term cost-effectiveness through digital sustainability. Let’s explore how embedding these principles within your corporate ethos can catalyse a transition towards a more sustainable, efficient, and economically viable future.

The Environmental Impact of Our Digital World

In today’s interconnected landscape, every digital interaction—from a cloud storage retrieval to sophisticated computations and online streaming—plays a part in energy consumption and elevates CO2 emissions. Merely scaling back online activities is not a sufficient response; a comprehensive and strategic approach to #DigitalSustainability is essential. The surge in AI and ML technologies integration, such as ChatGPT, into the fabric of corporate strategies has underscored this urgency. Recent statistics reveal that as of 2023, approximately 49% of companies have already incorporated ChatGPT into their operations, with an additional 30% planning to embrace it. This trend is reflective of a broader embrace of artificial intelligence across industries, where a significant number of companies have adopted or plan to adopt AI technologies. Moreover, the AI market is poised for substantial growth, with projections indicating an annual growth rate (CAGR 2024-2030) of 15.83%. These figures not only highlight the rapid integration of AI into business processes but also emphasise the growing need for solutions that prioritise energy efficiency and sustainability in the digital domain. As AI and digital technologies become increasingly central to corporate operations, the push for environmentally conscious and sustainable digital infrastructures becomes critically important, urging companies to adopt practices that mitigate their digital environmental footprint.

Clarifying the Internet’s Broad Reach

The term “internet” extends far beyond browsing and email; it encompasses cloud services and computing, pivotal components of our digital ecosystem’s environmental impact. This vast infrastructure in 2020 was notably accountable for 1.4% of global emissions and consumes 4% of the world’s electricity, standing shoulder to shoulder with the aviation industry in terms of environmental impact. As we delve deeper into the digital age, the situation grows more pressing with digitalisation’s energy consumption marked at 10% of worldwide energy consumption in 2023. This figure is on an upward trajectory, especially with the proliferation of language learning models like ChatGPT, highlighting a concerning trend towards increasing energy demand. Further insights from a comprehensive analysis echo these sentiments, projecting a continuous rise in energy consumption fuelled by digitalisation.

In light of these developments, aligning with the Paris Agreement’s ambitious goals becomes even more critical. Specifically, the global ICT industry faces the daunting task of reducing its greenhouse gas (GHG) emissions by 45% from 2020 levels by 2030, a target that underscores the urgent need for sustainable practices across all sectors of digital infrastructure. This challenge, while formidable, presents an opportunity for organisations to lead by example, not just in adhering to but also in surpassing triple reporting standards, thereby demonstrating an unwavering commitment to environmental stewardship and sustainable development in the digital age.

Why Digital Sustainability Matters for Your Business

In the past decade, sustainability has transformed from a niche interest into a core component of corporate strategy, reflecting not only business priorities but also growing consumer concerns. However, the swift pace at which technology evolves, alongside slow-moving regulatory frameworks, has left many businesses scrambling to keep up. The expanding environmental footprint of big data, cloud computing, and AI technologies has become impossible to ignore, compelling companies to proactively adopt sustainable digital practices.

Adding to the urgency is the heightened public and corporate awareness brought about by innovations like ChatGPT and most recently Gemini, which have thrust the topic of big data into mainstream discussions. This visibility is likely to attract increased scrutiny from regulatory bodies and could result in sustainability metrics being incorporated into triple reporting standards in the near future. By taking the initiative to audit and refine your data management practices now, your company can not only stay ahead of potential regulatory changes but also demonstrate leadership in digital sustainability.

With Earth Day approaching, there’s a timely and compelling opportunity to position your company at the forefront of sustainable digital operations. This moment serves as a reminder of the importance of integrating environmental considerations into every aspect of your business, especially in areas that are often out of the public eye, such as digital infrastructure. By committing to sustainable digital practices today, your company can lead by example, showing that a commitment to the planet is not just good ethics—it’s also good business.

Identifying Opportunities for Digital Streamlining

For companies dedicated to minimising their digital environmental impact, a thorough examination of digital operations is paramount. Identifying inefficiencies and areas for optimisation can significantly reduce digital waste and energy consumption. Here’s a deeper look into key areas for digital streamlining:

• Evaluating Data Pipelines: Assess whether all data pipelines in use are essential or if some have become redundant. Redundant pipelines can lead to unnecessary energy consumption and complicate data management. For a practical example, explore our case study on a silently failing pipeline, demonstrating the importance of vigilance in pipeline efficiency.
• Ensuring Infrastructure Efficiency: Verify that all cloud and computing infrastructures are functioning optimally. Inefficient infrastructures can increase energy usage and operational costs. Our case study on improving the orchestration of pipelines highlights the benefits of streamlining infrastructure for enhanced efficiency.
• Addressing Computing Errors: Issues and errors in computing not only delay operations but also waste computational resources, leading to increased energy usage. The case of malfunctions case study showcases strategies for addressing computing errors effectively.
• Optimising Job Execution: Examine if there are issues with running jobs inefficiently. Streamlining job execution can reduce processing time and energy expenditure. Our case study on a non-efficient bespoke solution illustrates the importance of optimising job execution.
• Purifying Data Storage: Identify and eliminate clean data repetitions or redundancies. This is most often the issue resulting from low data quality. Efficient data storage reduces the need for extensive computing power to manage and access data. You can find more on ensuring data quality in our video guide.
• Decluttering Cloud Space: Remove unnecessary items from cloud storage. Very often, even the most organised companies leave old documents even after they have become entirely obsolete. Compared to physical copies cluttering the office, the cloud versions are easier to forget. Make sure that you are not paying for the storage of unnecessary items. This practice frees up space and reduces the energy required for data retrieval and maintenance.
• Avoiding Data Duplication: Ensure data is not duplicated across departments. Quite often, organisations are divided based on their functions. In those cases, various departments collect the same data instead of using one database. This duplicates the storage with each team. Centralised data management enhances efficiency and reduces storage requirements.
• Streamlining User Journeys: Simplify user interfaces and processes to reduce the computational power required for navigating through systems.
• Enhance LLM Efficiency: To maximise digital sustainability and conserve computing resources while using energy-intensive LLMs like ChatGPT, consider a multifaceted approach that streamlines information retrieval and optimises query responses. By refining your system to deliver more relevant answers, you can significantly reduce the volume of queries. Implementing advanced indexing techniques and preprocessing data ensures that searches are both energy-efficient and produce contextually richer responses. This strategy not only accelerates access to information through efficient retrieval processes but also minimises unnecessary data processing. Furthermore, incorporating covered queries by leveraging indexed documents pre-answers common inquiries, thus enhancing performance and saving additional energy and resources. This holistic approach enhances LLM’s operational efficiency and contributes to a more sustainable digital environment.
• Choosing Efficient Programming Languages, Frameworks and Databases: Some programming languages are more energy-efficient than others. Selecting an efficient language can reduce the computational power needed for the same tasks.
• Removing Unused Code: Libraries often contain code that is no longer in use. Identifying and removing such code can decrease the amount of processing required during operations.

By focusing on these areas, companies can take significant steps towards reducing their digital footprint. Each of these strategies not only contributes to a more sustainable digital environment but also enhances operational efficiency and cost-effectiveness. These are some key considerations, and many more aspects can be identified and analysed during a comprehensive digital audit to further streamline operations and minimise environmental impact.

Partnering with TantusData for Enhanced Digital Sustainability

TantusData is at the forefront of assisting companies in navigating the complexities of digital sustainability. Our focus on optimising data management, computing efficiency, and overall digital infrastructure supports not only your environmental objectives but also contributes to operational cost savings. By collaborating with TantusData, your organisation can achieve sustainability milestones and receive certification for your green digital practices, enhancing your corporate value in the process.

Your Business’s Path Forward: A Sustainable Digital Future

Adopting sustainable digital practices is more than an environmental responsibility—it’s a strategic advantage in today’s corporate world. As we continue to navigate the digital revolution, it’s imperative for companies to integrate sustainability into their digital operations. This is an invitation for your business to explore how TantusData can transform your digital footprint into an asset for both the planet and your bottom line. Reach out to us to discover how we can help you achieve a leading edge in digital sustainability.

Let’s get in touch.

The post Amplify Your Corporate Green Initiatives: Triumph in Sustainability and Reporting appeared first on TantusData.

What should a Business Leader know before investing in AI?

ChatGPT is everywhere. Chances are, you’re already using it for everyday tasks – its language skills are unmatched. Using it to fix spelling in your email is really easy, but let’s imagine transforming it into a frontline warrior – a chatbot that not only chats but also solves customer problems. We will use this business application of a chatbot as an example in this article.

Applying chatbots to more complex business processes is very tempting but can be full of pitfalls. There are many aspects to consider when building such an application. From the privacy of your data to the cost of implementation. And from the correctness of the responses provided by chat to the maintenance in the future. In this article, we will take all the challenges one by one, describe them and give a guideline on how to approach them pragmatically. This leads to building a successful project, providing a return on investment.

What is a Large Language Model? What LLM is not?

Remember the last time you chatted with ChatGPT? The conversation was smooth, and it even knew who the US president was. But pose a niche question about, say, ‘how to change the invoice number in my system,’ and it might stumble. Even more, it might confidently tell you something off-base. 

I bring that up because it’s essential to understand that ChatGPT or any other general LLM is not a general source of knowledge. Picture this: You wouldn’t ask a history professor about advanced rocket science, right? Similarly, out of the box, an LLM may not ace a deep dive into your specific business.

But it becomes a potent tool when you plug it with information about your domain and instruct it on what to do with that knowledge. A tool that might appear intelligent.

Getting correct answers

Imagine AI as a car and data as its fuel. Without the right fuel, it won’t take you where you want. Suppose you’ve set up your AI for customer support, and someone asks, ‘How do I get the invoice?’. Instead of the AI drawing a blank or guessing, it should know exactly where to fetch the answer – like a librarian who knows which shelf a book is on.

Data is the fuel for AI. For the chat to answer questions specific to your organisation, you need to provide it with the relevant information. Let’s say you want it to serve customer support purposes. Let’s assume the user asks the system, ‘How do I get the invoice?’ 

The most standard approach to this problem is building an application that searches for relevant information and provides it to the chat. In our case, the application seeks information about the invoicing process. All searching is done in vector databases, which are great tools for finding the appropriate information in text documents. No wonder they’re becoming more popular since the rise of LLMs. Once it finds the clue, it hands it over to our chat. 

Let’s stop here and amplify the message – general purpose LLM does not know much about your business. Here’s the gist: While AI is great with words, it only knows the specifics of your business if you tell it. Feed it the correct details to transform it from a general chat buddy to a helpful assistant. Think of it as training a new employee.

In the simple case described above, most of the ‘magic’ is done by vector databases. But in some cases, just providing the information ‘on the fly’ is not good enough. For the LLM to perform better, you must fine-tune the vector embeddings or the LLM model itself. This is more advanced, so we will describe it in a separate article.

Privacy concerns

Imagine sending a personal letter and having someone else read it. That happens when you use ChatGPT or similar tools – the data goes to their home base. No worries if you’re chatting about the weather. But what if it’s confidential customer details? That’s where things get tricky.

Picture a customer sharing personal info on your support chat. Now, where does that data go? Outside your walls? And are you breaking any rules by letting it?

Confirming the legal aspects is the minimum you should do. But if you are not allowed to send the data to third parties, you need to consider privately hosted LLM so your data never goes outside your data centre. The selection of open-source LLMs available to use as privately hosted is broad. Broad selection means you have many options, but at the same time, you need expertise to make wise selections. Especially since the market is hot and keeping up with all the changes is challenging. The right decision requires careful consideration of aspects like the model’s performance on your hardware and the ability to scale up and down if needed.

A cherry on top? Going private means you aren’t tied down by vendor lock-in. You can move your infrastructure to your own data centre or cloud. It’s a critical point in the context of mitigating the risk of a vendor making drastic changes in the pricing or in the offered service itself.

Cost and defining the scope of the project

When building a house, you wouldn’t start with the roof or fancy decor, right? You’d plan the foundation and budget from day one. It’s the same with an AI. If your supplier changes the rules, you’ve got expenses like engineering time, hardware, cloud, and possible surprise fees.

When building an AI application based on Language models, it’s essential to control the project’s scope and expected costs from day one. 

It’s easy to dream big. After all, you’ve got all the tools at your fingertips. But to deliver the business value, you need to analyse the problem you are solving and set realistic milestones for the AI project. And it’s essential to start with issues which are common and easy to solve. E.g. When building a customer support application, it makes sense to begin by addressing issues which are repeated over and over again. You’d know them if you chat with your support team or skim through past tickets. If you approach the problem correctly, it’s likely that after two or three iterations with your product, you realise that you have solved 80% of the issues. Suddenly, you’ve got a solid ROI and can decide if you want to add those fancy trims or if your house is good enough.

Maintenance

Keep in mind that the more complex the application you are building, not only the implementation cost but also the cost of using it in the future increases – the maintenance and the price you pay for hardware or API calls. The maintenance might include aspects such as deployment complexity, testing, reacting to changes introduced by third parties or feeding your application with newer data.

That’s another reason to think early about the scope and be flexible during the implementation. So, before diving in, sketch out your costs. Ask yourself: How much is each chatbot chat worth to you? How much would you pay to help one customer if it’s a support bot? What conversion rate justifies the effort and cost of maintaining a chatbot if it’s for shopping? Knowing your numbers helps you steer clear of unwanted surprises.

Summary

As outlined above, it’s critical to understand the challenges when selecting the scope of the LLM project. But with a clear understanding of potential roadblocks from the beginning, you’re paving the way to the ROI you expect. The spectrum of challenges in implementing LLM for solving business problems in your organisation is vast, and you should:

• Ensuring correct behaviour makes it essential to provide your LLM with correct, well-structured data.

• Addressing privacy concerns means understanding the legal prerequisites. Remember that you have options of hosting LLM in your Data Center, which simplifies the legal aspects but introduces extra technical challenges.

• Budgeting for a project means considering all costs: development, API charges, and maintenance. It’s important to verify the assumptions after every milestone of the project and make sure the scope and the cost are under control.

• Thinking ahead is essential, so estimate maintenance efforts. Ensure these efforts align with the expected benefits of its integration into your organisation.

The post LLMs can make your business soar or sink, so tread carefully. appeared first on TantusData.

In the previous article, we covered anonymization and pseudonymization – techniques used in the context of ensuring data privacy, and more specifically, in the context of GDPR. While the basic idea and goal of anonymization are intuitive, GDPR introduces more subtle regulations, such as the Right to Be Forgotten. This concept is tricky because it is subject to interpretation. The complexity is further compounded by technical implementation details within a complex ecosystem that handles large datasets.

Open to interpretation

If you possess customer data and the customer revokes their consent, requesting to be forgotten, you must take action. You need to ensure that you remove the data. What’s tricky about this? There are at least two aspects to consider:

1. If you decide to simply delete the data, ensure that the deletion actually occurs on a physical level. Modern data platforms excel at abstracting certain operations, so just because you think you deleted the data doesn’t mean it’s unrecoverable or that it’s no longer present in your systems.

2. Alternatively, you might not need to remove the data if you choose to anonymize it instead. In this case, you must ensure that you correctly identify all the Personally Identifiable Information (PII) fields and that the anonymization process is accurate. More on this topic follows below.

The devil lies in the implementation

Deleting data when a customer revokes their consent may seem appealing from a security perspective. The assumption is that once the data is gone, you are safe, right? While this is true, the deletion process itself is not so straightforward. The complexity arises from the varied ways in which tools within the Data & Analytics ecosystem handle data removal. The deletion behavior in these tools does not always align with the general intuition of ‘deletion.’

Most modern data processing tools are optimized for ingestion and analytics queries, and their ability to perform fine-grained data deletion is limited. Sometimes, this means that you cannot simply remove individual records. At other times, it means that the deletion is not immediate. Finally, it might be that the data is merely marked as unavailable, but not actually deleted.

Do you see how some of these options might not be acceptable from a regulatory perspective? Therefore, it is crucial to consider technical limitations when deciding on a strategy for the Right to Be Forgotten.

Below, we will explore some commonly used data processing tools:

1. Traditional relational databases – These databases are good at operations involving single rows at a time, making it easy and safe to delete specific rows.

2. Hadoop and HDFS – It is impossible to delete a single record. If you want to remove the data of a single user, you need to rewrite the entire dataset. The rewrite comes with its own considerations, but the main one is whether it’s acceptable from a performance and cost perspective to rewrite the entire dataset just to remove a single record.

3. Delta, Hudi, Iceberg – These systems allow for the deletion of individual records, but you need to understand that the ‘deletion’ is merely creating another delete-marker record. If you want to ensure that the data physically disappears eventually, you need to carefully design the compaction strategy specific to the storage you are using.

4. NoSQL databases – Each NoSQL database has its own way of handling deletion. For instance, Cassandra uses tombstones – data-deletion markers – which lead to similar concerns as described in the previous point. A carefully designed compaction strategy is a must.

5. BigQuery – Large-scale deletion is not what BigQuery is optimized for, which is why the tool has quotas for these kinds of operations. You also need to be aware that the data does not disappear immediately. That’s why it’s worth considering a full rewrite or crypto shredding (please refer to the next point) in certain scenarios.

Does this cover all options?

Unfortunately, the variety of tools makes it impossible to mention them all in a single article. You have to carefully investigate your ecosystem in the context of the Right to Be Forgotten. I only mention some of the commonly used tools to illustrate the complexity of the problem.

The right to be forgotten is not fulfilled if the data is not erased on a physical level. Then it’s simply hidden. Left behind till somebody finds it.

Does anonymised = forgotten?

Deleting records is not the only option for forgetting a customer. You can consider anonymizing the records of customers who revoke their consent. Anonymizing the dataset has the obvious benefit of retaining some of the data (without PII) so it can be used for analytics purposes in the future.

However, remember that you need to be very careful when designing the architecture and procedures. Things to keep in mind:

1. Anonymization should be a non-reversible process.

2. You have to correctly identify and anonymize the PII fields.

3. Anonymizing just PII fields might not be enough. You have to ensure that you prevent indirect identification of the user’s data. More about this will be covered in the next article.

4. You must have a formal process in place and follow it when handling your datasets. Automate as much as possible to limit the chance of human errors and privacy breaches.

5. Naive anonymization of individual records comes with problems similar to deletion itself. You have to consider the properties of the analytics tools in your stack before deciding on this approach.

One of the specialized techniques for handling the Right to Be Forgotten is crypto-shredding. The basic idea is to encrypt customer records and remove the encryption key when the customer requests to be forgotten. Using this technique can free you from concerns related to the cost of reprocessing the entire dataset. At the same time, it must be carefully thought through, especially when it comes to storing, securing, and removing the encryption keys.

Look for a description of crypto-shredding in one of the upcoming articles.

What Have You Missed?

Once you have determined the appropriate method for addressing the Right to Be Forgotten (RTBF), you are nearly prepared to begin the implementation process. Why nearly? A crucial aspect of this process is identifying all datasets containing sensitive information. It goes without saying that managing your master data is essential. However, are you fully aware of the life cycles of your datasets? Can you confidently say that you are handling all derived datasets? In other words, are you dealing with a data lake, data lakehouse, or perhaps a data swamp?

Maintaining control over data lineage is a critical component of Data Governance, and we will discuss its significance in the context of RTBF in an upcoming article.

The post GDPR, the forgotten done right appeared first on TantusData.

Sit back and relax as our experts tell you all there is to know about any search project.

Whether you intend to set up a search engine or improve the one you have, there are a few matters, which are crucial to consider before you start.

Benefits of search engines

As you surely know many websites and e-commerce sites benefit greatly from having search engines operating well. But how do they do that? Why do others fail to deliver such ROI?

Are there other needs than simply letting consumers find the products they are looking for faster?

What to prepare before you start to save money and improve ROI faster.

And, does the project really needs to be that expensive and long-term, or is there an alternate option.

Sounds interesting? Make sure to check out our video prepared by our top search experts Maryna and Jakub.

Take me to the video!

The post All about search appeared first on TantusData.

Recently, we have been challenged to build an ML model that will boost company revenue by increasing click-through and conversion rates. The company has a web page with properties that one can book. The current approaches they tried, even though very reasonable, weren’t able to produce the expected result. They needed a fresh look into the problem, and this is how they found us.

Picking the best image

We started with a click-through rate problem. We needed to pick the best image of a property to present as a miniature. We needed to pick from two images. Let’s look at those examples:

It is hard to tell by only looking at images which one would be more attractive to potential clients. This is precisely the challenge the AI should help us with!

Apart from images, we had additional information about the properties, such as price, region, type of property, max pets, Wi-Fi and parking availability, etc. In total, we had 49 columns.

To summarize the problem: We have two images and additional data about properties, and we need to decide which one will have higher click through rate. We are given historical CTRs for images from A/B testing. The graphic below presents the highest level view of the process:

Data preparation

We have started with the analysis of additional data for properties to pick up the best features. We have picked 20 features (based on their correlation with the target metric) and did the following preprocessing:

• standardization for numerical features
• one hot encoding for categorical features
• mean imputation for numerical features
• most frequent imputation for categorical data

Regression model

In the first approach, we built a model that takes an image and additional data to predict the click-through rate (CTR). The data came from SnapTrip, which conducted A/B tests—showing different webpage versions to different user groups to measure which one performs better. Their previous models were likely ineffective, leading to inaccurate results.

For the images, we used a pre-trained EfficientNetB4 network and fine-tuned it. The categorical data was processed through a five-layer fully connected network. The outputs were concatenated at the end. We used mean squared error as the loss function and performed hyper-parameter tuning to optimize the model architecture. All images were processed through an augmentation layer with standard operations. Here is a code snippet:

def augument(image):
    image = tf.image.random_flip_left_right(image)
    image = tf.image.random_hue(image, 0.08)
    image = tf.image.random_saturation(image, 0.6, 1.6)
    image = tf.image.random_brightness(image, 0.05)
    image = tf.image.random_contrast(image, 0.7, 1.3)
    return image

Using this approach, we got the following result: The accuracy is calculated as a proportion of correctly selected images. The correct image is an image with a higher CTR. We run images through the model and pick the one with the highest accuracy.

The visualization of this approach looks like this:

Classification model

In the second approach, we used both images as input (together with additional data) as we expected that seeing both images could help the model make better decisions. The loss function wasn’t mean squared error, but it was standard cross entropy as we treated this as a classification problem where the model needed to decide whether the image first or second should be selected. We had two neurons as an output, and when the first neuron was activated them, the first image should be picked and vice versa.  We performed the same image augmentation as in the previous approach, and additionally, we swapped image positions in every case. So, the same images were presented to the model twice in swapped order.

We created a dedicated Keras layer for the images that looks like this:

class EfficientNetV2B1_TF(layers.Layer):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        image_size = 260
        self.base_model = keras.applications.EfficientNetV2B2(include_top=False,
                                                            weights='imagenet',
                                                            input_shape=(image_size, image_size, 3)
                                                            )
        self.base_model.trainable = False
        self.global_pooling = tf.keras.layers.GlobalAveragePooling2D()
        self.embedding_model = keras.Sequential(
            [
                layers.Input(shape=(2816)),
                tf.keras.layers.Dropout(0.25),
                layers.Dense(2),
            ]
        )
    def set_fine_tuning(self, fine_tune_at=3):
        self.base_model.trainable = True
        for layer in self.base_model.layers[:-fine_tune_at]:
            layer.trainable = False
    def call(self, image_1, image_2):
        image_1_base = self.base_model(image_1)
        image_2_base = self.base_model(image_2)
        embb_1 = self.global_pooling(image_1_base)
        embb_2 = self.global_pooling(image_2_base)
        images = tf.concat([embb_1, embb_2], axis=1)
        output = self.embedding_model(images)
        return output

Using this approach, we got the following result. It is better, than the one from regression.

The visualization of this approach looks like this:

Having developed those models, we decided to conduct an A/B test for the classification model. After deploying it to production and waiting a week for results, we observed that the CTR did not increase as much as we expected. Detailed investigation revealed that many images with higher CTRs in our training data now had lower CTRs. The issue was that we received an A/B test dataset from a specific time period, where certain images had higher CTRs and served as our reference. We trained the model to discover why some images were better than others, and it performed well on that dataset.

After conducting the A/B test again, we found that users often preferred different images than before, rendering our model ineffective. We are not entirely sure why user preferences changed suddenly, but it is likely due to seasonal variations that affected the A/B test data outcomes. Therefore, we recommend clients consider the time sensitivity and changing tastes of their potential customers.

To address this, one of the models we provided did not choose between two images but instead returned a specific CTR for each image. In theory, this model could process all images for a given property and select the one with the highest CTR. We decided to collect data over longer periods and try again in the future. 

Not only A/B tests but also customer data would likely help, similar to how other sites operate by checking what users viewed or booked previously and basing recommendations on that.

Note: Access to comprehensive data is crucial for effective modeling and accurate predictions.

Sorting search results

The second problem we wanted to solve using AI was search result sorting. After the user specified what properties they were looking for, for what dates, and for how many people, we presented our offer. The problem is which properties we should show first, as usually users only open the first few so that there are maximum chances for a successful booking.

Data preparation

Here, the data was similar to the previous example. The images and property information were the same, with the new addition of textual data in the form of property descriptions. We investigated the tabular data with property information and selected 15 features that best correlated with the target value. These features were chosen based on statistical models that identify correlations between features. The preprocessing for this data was the same as in the previous problem.

We’ve pre-process description data using nltk. We carried out the following operations:

• lower casing
• html tag removal
• urls removal
• word tokenization
• punctuation removal
• stop words removal
• lemmatization

The target value was the historical conversion rate for each property.

Creating models

xgboost

The first model we created was based on XGBoost. It received input consisting solely of tabular data from SnapTrip’s historical A/B tests. We performed hyper-parameter tuning to optimize the following parameters (the numbers represent the best parameters found):

• colsample_bytree=0.7
• learning_rate=0.01
• max_depth=3
• n_estimators=1000

Here is the result we got with this approach:

Neural network

We started creating a second model, the neural network model, by combining the tabular and textual data. We decided not to complicate it by including the images from the beginning. As we didn’t have that much data (something like 4000 rows), we started with shallow networks. We used hyperparameter tuning to find the best network architecture for us. Here is the code snipped with all configurable parameters.

def build_keras_hp_model(hp):

model = Sequential()
    kernel_regularizer = None
    kernel_regularizer_value = hp.Choice('kernel_regularizer', values=[0.01, 0.02, 0.04, 0.06, 0.08])
    type = hp.Choice('kernel_regularizer_type', values=["L1", "L2", "L1L2", "None"])
    if type == "L1":
        kernel_regularizer = regularizers.L1(kernel_regularizer_value)
    elif type == "L2":
        kernel_regularizer = regularizers.L2(kernel_regularizer_value)
    elif type == "L1L2":
        kernel_regularizer = regularizers.L1L2(kernel_regularizer_value)
    activity_regularizer = None
    activity_regularizer_value = hp.Choice('activity_regularizer', values=[0.01, 0.02, 0.04, 0.06, 0.08])
    type = hp.Choice('activity_regularizer_type', values=["L1", "L2", "L1L2", "None"])
    if type == "L1":
        activity_regularizer = regularizers.L1(activity_regularizer_value)
    elif type == "L2":
        activity_regularizer = regularizers.L2(activity_regularizer_value)
    elif type == "L1L2":
        activity_regularizer = regularizers.L1L2(activity_regularizer_value)
    for i in range(hp.Int('layers', 4, 12)):
        model.add(Dense(hp.Choice(
            'units', [50, 60, 70, 80 ,90, 100]),
            kernel_initializer=hp.Choice(
            'kernel_initializer', ['glorot_normal', 'glorot_uniform', 'he_normal', 'he_uniform', 'lecun_normal', 'lecun_uniform', 'random_normal']),
            kernel_regularizer = kernel_regularizer,
            activity_regularizer = activity_regularizer
        ))
        if not hp.Boolean("do_batch_norm_after_dropout") and hp.Boolean("do_batch_norm_before_dropout"):
            model.add(BatchNormalization())
        model.add(Activation(hp.Choice('activation', ['elu', "relu", 'selu', 'gelu', 'leaky_relu', 'swish'])))
        model.add(Dropout(hp.Choice('dropout', values=[0.0, 0.1, 0.15, 0.2, 0.25, 0.3])))
        if not hp.Boolean("do_batch_norm_before_dropout") and hp.Boolean("do_batch_norm_after_dropout"):
            model.add(BatchNormalization())
    model.add(Dense(1))
    model.compile(
        optimizer=tf.keras.optimizers.Adam(hp.Choice('learning_rate', values=[1e-2, 1e-4, 1e-5, 1e-6])),
        loss='mean_squared_error',
        metrics=[tf.keras.metrics.RootMeanSquaredError()])
    return model

The best-performing number of layers was 8, and the number of neurons was 70. The model also liked a small dropout (0.1) and L1 regularization with a value of 0.01.

We used mean squared error as a target value.

Here are the results for this model:

Ensemble model

Finally, we decided to combine both models via a simple ensemble. We ran our data through both models and averaged the results. We got these results using this approach, which are slightly better than any of the individual ones.

Results

Our decision to push this model into production was not without its challenges. After a few weeks of A/B testing, we were thrilled to see a conversion rate increase of over 10% on several pages. However, we also encountered a roadblock on two pages, where the target customers had different tastes and needs. To address this, we had to find an alternative approach, which involved developing dedicated models for these pages.

Note: We initially received a single dataset without information on its collection process. For optimal results, it’s important to have separate datasets for different client pages, as preferences can vary significantly. Dedicated models for each page type can help achieve better conversion rates. Therefore, we delivered customized models to address these variations and improve overall performance.

The post How to convince AI to help us boost our profit appeared first on TantusData.