Migrating Data Platform, data warehouse or data lake?

When deciding to move your data to the cloud, many people focus on costs, expected gains, easier maintenance, or simplified development. Sometimes, lower costs or easier maintenance drive the decision. However, it’s crucial to also consider the potential cost of exiting the cloud. What happens if, at some point, you decide you no longer want to be on a particular cloud platform? This could happen due to rising costs, new technology options, or even legal or political reasons.

Did you know that the exit-cost of your data platform from a Cloud might be a more expensive project than an original migration to the Cloud?

If you plan to migrate your data platform to the cloud, thinking about future migration costs now is a sign of responsible migration. Understand what will be the cost in terms of dollars, time, and effort if at some point you decide to exit that specific cloud vendor. This means recognizing that exit costs aren’t always obvious and that you’ll need to consider the various aspects of vendor lock-in.

What exactly are the risks associated with vendor lock-in:

• Long term costs rise. If you don’t have an easy to move alternative you are at risk of becoming a hostage.
• Lack of flexibility – even if your solution is great now you always risk that it will not be developed in the future. 
• High transfer fees. Most clouds are charging you if you transfer data from it (egress fee). That needs to be calculated when planning a migration from a specific cloud vendor.
• Contractual agreements – Vendor lock-in is not only about proprietary technology or data formats. What’s in the contract might be another trap which might become painful in the future.
• Lost optimization opportunities – there is no such thing as free lunch. Cloud solutions usually are easier to start with but your engineers might lose ability to do low level tuning if you even need that.

When planning your cloud migration, don’t overlook the long-term exit costs – ask your provider about compatibility of specific technology with other tools in the market. 

What can you do to mitigate the risk:

• Evaluate the technology and alternatives – consider if there are open-source alternatives to your chosen data warehousing solution or if it’s compatible with other vendors
• If you decide on proprietary technology make sure the cost of in and expected exit cost is justified by what you gain (usually easier, faster development)
• Consider hybrid cloud. It’s more expensive but gives you more flexibility in the long run.
• Consider favouring well-known standards and open source technologies and data formats. A good example is Kubernetes – it’s a technology which you can have on your own servers as well as with any major cloud providers.

Considering moving to the cloud providers, ping me on Linkedin (https://www.linkedin.com/in/marcin-szymaniuk/) for specific calculations.

Summary

The data space is moving fast. 

Always ask yourself what is the risk that within 5 years you have to migrate again.

Always ask yourself what will happen if you can’t use the selected data platform anymore. How long notice would you need to migrate to another solution? 

The post Vendor lock-in when selecting a Cloud Data Platform architecture. appeared first on TantusData.

Whether you’ve joined us in the past or are planning to attend our upcoming events, there’s always something exciting on the horizon. Let’s take a look at where we’ve been and where we’re headed next, so that you can gain valuable insights from attending our speeches, participating in our workshops, or exploring our other content. This season, our focus is on Large Language Models (LLMs) and Apache Spark, offering you a wealth of knowledge and practical skills to enhance your expertise. There’s a lot you can gain from engaging with our content and events, so don’t miss out!

Past Events: Highlights and Memories – Top Learning Places to Keep in Your Calendar for Next Year

Big Data Europe, November 2023

We were honored to host two insightful workshops: “ChatGPT, LLMs, and LangChains” and “Apache Spark Performance Tuning”. The event was a phenomenal success, and it was great meeting everyone there. If you missed it or want to relive the experience, check out our exclusive videos here.

SMG Data Summit, January 23-24, 2024

This two-day event, hosted by Google and organized by SMG Swiss Marketplace Group, was a deep dive into the world of data analytics and innovation. We led two workshops: “ChatGPT, LLMs, and LangChains” and “Deep Dive into AI & ML for CxOs, Managers, and Business Leaders”. It was a fantastic opportunity to enhance our data expertise. More details can be found here.

Warsaw IT Days, April 5-6, 2024

Celebrating its 15th anniversary, this iconic event gathered over 10,000 IT and Data Science enthusiasts. Our speeches, “AI Chats – What Nobody Told You: The Conundrums of Business Integration” and “Optimising Apache Spark and SQL for Improved Performance,” were well-received. Discover more about this event here.

If you missed any of these events, there’s still time to sign up for our upcoming ones or look out for next editions. We also invite you to explore the wealth of content already available on our blog and YouTube channel. Dive into our extensive library of articles, videos, and tutorials to stay updated and inspired.

Conf42 LLM, April 11, 2024

Marcin Szymaniuk will be diving into the complexities of integrating AI like ChatGPT into business frameworks at Conf42 LLMs. Learn how to optimize your resources for successful AI adoption. Don’t miss out – details here.

Big Data Technology Warsaw Summit, April 24-25, 2024

Join us for technical presentations, interactive roundtables, and networking opportunities with over 600 attendees. We’ll be leading a roundtable on “What Product Owners and Managers Should Know About ML and LLMs: The Challenges of Business Integration”. Find out more here.

Data Analytics Meeting, May 17-18, 2024

Our keynote, “A Short History of Data – Where Are We Aiming with AI?” will explore the evolution and future of data. This conference supports student development through discussions and networking. More information is available here.

Infoshare, May 22-23, 2024

The biggest tech and startup event in CEE, bringing together thousands of enthusiasts. We’ll be discussing “Optimising Apache Spark and SQL”. Don’t miss out – sign up here.

Jfokus, May 28, 2024

Join us at the Jfokus Training Camp in Stockholm for a focused one-day workshop on “LLMs and LangChains”. This is a great chance for hands-on learning and networking. Secure your spot here.

Conf42 Machine Learning, May 30, 2024

Marcin will share insights on Apache Spark SQL at this prestigious event. Check out the details here.

Upcoming Events: Join Us and Stay Ahead

Voxxed Days Luxembourg, June 20-21, 2024

We’re excited to deliver an LLM workshop at this developer-focused event. Learn more and register here.

PyCon Estonia, September 5-6, 2024

Join us for a workshop on LLMs at one of the largest Python gatherings in the Nordics. More details can be found here.

SREDAY, September 19-20, 2024

We will be offering an LLM workshop at this in-person conference in London. Stay tuned for more updates here.

Looking Ahead

We have more exciting events coming soon and will be adding them to this article as they are confirmed. Stay tuned for updates and new opportunities to connect, learn, and innovate with us.

Trending Topics: Apache Spark and LLMs

Two key topics that garnered significant interest at our events were Apache Spark and LLMs. Due to this high demand, we have dedicated articles on these subjects that are added to the recommendations below.

Stay tuned for more updates and see you at our next event!

The post Unleashing Innovation: A Glimpse into Our Exciting Event Journey appeared first on TantusData.

Clutch hasannounced its recognition of TantusData as a 2024 Spring Global Award winner for Qlik, Hadoop, Tableau, Big Data Compliance, Fraud, & Risk Management services on Clutch, the leading global marketplace of B2B service providers. 

Honourees are selected based on their industry expertise and ability to deliver scores that are calculated based on the client feedback from thousands of reviews published on Clutch. TantusData is honoured to be recognised as a 2024 Spring Clutch Global Award winner. This award is a testament to the excellent client work we have delivered this year as recognised through the voice of our customers in their reviews on Clutch. We’re proud to be recognised as aleader on a global scale. Clutch Global Awards showcases the very best in the B2B services industry worldwide.

“We are incredibly proud to receive the 2024 Spring Clutch Global Award. This recognition is a direct reflection of our team’s hard work and dedication to delivering outstanding results for our clients. It’s an honour to see our efforts acknowledged on such a prestigious platform, and we are motivated to continue setting high standards in the industry.”

Marcin Szymaniuk, CEO of TantusData

Since joining Clutch, TantusData has been delighted to receive multiple recognitions from the platform. Previously, we were listed as a leading service provider on Clutch, being named one of Poland’s industry game-changers in big data analytics. Read more about this recognition here.

“It is a joy to witness the incredible success of leading companies worldwide on our platform, and an even greater joy to recognise these companies as Clutch Global honourees. Their dedication to delivering next-level services to clients has not only bolstered their own success but empowered numerous clients to thrive as well. In recognising this spring’s Clutch Global honourees, we aim to showcase industry leaders and encourage connections for Clutch users seeking tailored services to achieve their goals.” 

Sonny Ganguly, Clutch CEO

View our recent work and reviews on our Clutch profile.

If you’re interested in hiring us for your next project, get in touch with us here.

The post TantusData Recognised as a Clutch Global Leader for Spring 2024 appeared first on TantusData.

Optimising Digital Efficiency: A Corporate Imperative

In today’s digital-driven corporate landscape, the environmental footprint of our online activities, which encompass cloud services, computing, and the broader internet infrastructure, often escapes scrutiny. These critical components of modern business operations are not merely sources of operational inefficiency and financial expenditure; they significantly contribute to environmental degradation. Therein lies a pivotal call to action for businesses globally: to reassess and refine their digital infrastructure, spanning from the intricacies of cloud storage systems to the depths  of computing processes. By elevating digital cleanliness to a strategic priority, organisations stand to not only meet emerging standards of digital sustainability reporting—an accolade in corporate environmental stewardship—but also unveil substantial avenues for cost savings. Diving deeper into the ethos of digital sustainability reveals its capacity to forge a future that is not only environmentally harmonious but also streamlined for greater efficiency and financial prudence.

It’s crucial to acknowledge that certain areas prone to digital wastefulness, including redundant data storage, inefficient computing practices, and overlooked aspects of cloud utilisation, may remain hidden beneath the surface of day-to-day operations. There are also those which are even less obvious such as less energy efficient programming languages. These undercurrents of inefficiency represent missed opportunities for optimisation and sustainability. Importantly, while digital sustainability reporting is not currently a mandated practice in triple reporting, it serves as a powerful testament to an organisation’s commitment to environmental stewardship and technological foresight. Embracing this proactive approach not only positions a company as a leader in ecological responsibility but also as a visionary in harnessing the potential for long-term cost-effectiveness through digital sustainability. Let’s explore how embedding these principles within your corporate ethos can catalyse a transition towards a more sustainable, efficient, and economically viable future.

The Environmental Impact of Our Digital World

In today’s interconnected landscape, every digital interaction—from a cloud storage retrieval to sophisticated computations and online streaming—plays a part in energy consumption and elevates CO2 emissions. Merely scaling back online activities is not a sufficient response; a comprehensive and strategic approach to #DigitalSustainability is essential. The surge in AI and ML technologies integration, such as ChatGPT, into the fabric of corporate strategies has underscored this urgency. Recent statistics reveal that as of 2023, approximately 49% of companies have already incorporated ChatGPT into their operations, with an additional 30% planning to embrace it. This trend is reflective of a broader embrace of artificial intelligence across industries, where a significant number of companies have adopted or plan to adopt AI technologies. Moreover, the AI market is poised for substantial growth, with projections indicating an annual growth rate (CAGR 2024-2030) of 15.83%. These figures not only highlight the rapid integration of AI into business processes but also emphasise the growing need for solutions that prioritise energy efficiency and sustainability in the digital domain. As AI and digital technologies become increasingly central to corporate operations, the push for environmentally conscious and sustainable digital infrastructures becomes critically important, urging companies to adopt practices that mitigate their digital environmental footprint.

Clarifying the Internet’s Broad Reach

The term “internet” extends far beyond browsing and email; it encompasses cloud services and computing, pivotal components of our digital ecosystem’s environmental impact. This vast infrastructure in 2020 was notably accountable for 1.4% of global emissions and consumes 4% of the world’s electricity, standing shoulder to shoulder with the aviation industry in terms of environmental impact. As we delve deeper into the digital age, the situation grows more pressing with digitalisation’s energy consumption marked at 10% of worldwide energy consumption in 2023. This figure is on an upward trajectory, especially with the proliferation of language learning models like ChatGPT, highlighting a concerning trend towards increasing energy demand. Further insights from a comprehensive analysis echo these sentiments, projecting a continuous rise in energy consumption fuelled by digitalisation.

In light of these developments, aligning with the Paris Agreement’s ambitious goals becomes even more critical. Specifically, the global ICT industry faces the daunting task of reducing its greenhouse gas (GHG) emissions by 45% from 2020 levels by 2030, a target that underscores the urgent need for sustainable practices across all sectors of digital infrastructure. This challenge, while formidable, presents an opportunity for organisations to lead by example, not just in adhering to but also in surpassing triple reporting standards, thereby demonstrating an unwavering commitment to environmental stewardship and sustainable development in the digital age.

Why Digital Sustainability Matters for Your Business

In the past decade, sustainability has transformed from a niche interest into a core component of corporate strategy, reflecting not only business priorities but also growing consumer concerns. However, the swift pace at which technology evolves, alongside slow-moving regulatory frameworks, has left many businesses scrambling to keep up. The expanding environmental footprint of big data, cloud computing, and AI technologies has become impossible to ignore, compelling companies to proactively adopt sustainable digital practices.

Adding to the urgency is the heightened public and corporate awareness brought about by innovations like ChatGPT and most recently Gemini, which have thrust the topic of big data into mainstream discussions. This visibility is likely to attract increased scrutiny from regulatory bodies and could result in sustainability metrics being incorporated into triple reporting standards in the near future. By taking the initiative to audit and refine your data management practices now, your company can not only stay ahead of potential regulatory changes but also demonstrate leadership in digital sustainability.

With Earth Day approaching, there’s a timely and compelling opportunity to position your company at the forefront of sustainable digital operations. This moment serves as a reminder of the importance of integrating environmental considerations into every aspect of your business, especially in areas that are often out of the public eye, such as digital infrastructure. By committing to sustainable digital practices today, your company can lead by example, showing that a commitment to the planet is not just good ethics—it’s also good business.

Identifying Opportunities for Digital Streamlining

For companies dedicated to minimising their digital environmental impact, a thorough examination of digital operations is paramount. Identifying inefficiencies and areas for optimisation can significantly reduce digital waste and energy consumption. Here’s a deeper look into key areas for digital streamlining:

• Evaluating Data Pipelines: Assess whether all data pipelines in use are essential or if some have become redundant. Redundant pipelines can lead to unnecessary energy consumption and complicate data management. For a practical example, explore our case study on a silently failing pipeline, demonstrating the importance of vigilance in pipeline efficiency.
• Ensuring Infrastructure Efficiency: Verify that all cloud and computing infrastructures are functioning optimally. Inefficient infrastructures can increase energy usage and operational costs. Our case study on improving the orchestration of pipelines highlights the benefits of streamlining infrastructure for enhanced efficiency.
• Addressing Computing Errors: Issues and errors in computing not only delay operations but also waste computational resources, leading to increased energy usage. The case of malfunctions case study showcases strategies for addressing computing errors effectively.
• Optimising Job Execution: Examine if there are issues with running jobs inefficiently. Streamlining job execution can reduce processing time and energy expenditure. Our case study on a non-efficient bespoke solution illustrates the importance of optimising job execution.
• Purifying Data Storage: Identify and eliminate clean data repetitions or redundancies. This is most often the issue resulting from low data quality. Efficient data storage reduces the need for extensive computing power to manage and access data. You can find more on ensuring data quality in our video guide.
• Decluttering Cloud Space: Remove unnecessary items from cloud storage. Very often, even the most organised companies leave old documents even after they have become entirely obsolete. Compared to physical copies cluttering the office, the cloud versions are easier to forget. Make sure that you are not paying for the storage of unnecessary items. This practice frees up space and reduces the energy required for data retrieval and maintenance.
• Avoiding Data Duplication: Ensure data is not duplicated across departments. Quite often, organisations are divided based on their functions. In those cases, various departments collect the same data instead of using one database. This duplicates the storage with each team. Centralised data management enhances efficiency and reduces storage requirements.
• Streamlining User Journeys: Simplify user interfaces and processes to reduce the computational power required for navigating through systems.
• Enhance LLM Efficiency: To maximise digital sustainability and conserve computing resources while using energy-intensive LLMs like ChatGPT, consider a multifaceted approach that streamlines information retrieval and optimises query responses. By refining your system to deliver more relevant answers, you can significantly reduce the volume of queries. Implementing advanced indexing techniques and preprocessing data ensures that searches are both energy-efficient and produce contextually richer responses. This strategy not only accelerates access to information through efficient retrieval processes but also minimises unnecessary data processing. Furthermore, incorporating covered queries by leveraging indexed documents pre-answers common inquiries, thus enhancing performance and saving additional energy and resources. This holistic approach enhances LLM’s operational efficiency and contributes to a more sustainable digital environment.
• Choosing Efficient Programming Languages, Frameworks and Databases: Some programming languages are more energy-efficient than others. Selecting an efficient language can reduce the computational power needed for the same tasks.
• Removing Unused Code: Libraries often contain code that is no longer in use. Identifying and removing such code can decrease the amount of processing required during operations.

By focusing on these areas, companies can take significant steps towards reducing their digital footprint. Each of these strategies not only contributes to a more sustainable digital environment but also enhances operational efficiency and cost-effectiveness. These are some key considerations, and many more aspects can be identified and analysed during a comprehensive digital audit to further streamline operations and minimise environmental impact.

Partnering with TantusData for Enhanced Digital Sustainability

TantusData is at the forefront of assisting companies in navigating the complexities of digital sustainability. Our focus on optimising data management, computing efficiency, and overall digital infrastructure supports not only your environmental objectives but also contributes to operational cost savings. By collaborating with TantusData, your organisation can achieve sustainability milestones and receive certification for your green digital practices, enhancing your corporate value in the process.

Your Business’s Path Forward: A Sustainable Digital Future

Adopting sustainable digital practices is more than an environmental responsibility—it’s a strategic advantage in today’s corporate world. As we continue to navigate the digital revolution, it’s imperative for companies to integrate sustainability into their digital operations. This is an invitation for your business to explore how TantusData can transform your digital footprint into an asset for both the planet and your bottom line. Reach out to us to discover how we can help you achieve a leading edge in digital sustainability.

Let’s get in touch.

The post Amplify Your Corporate Green Initiatives: Triumph in Sustainability and Reporting appeared first on TantusData.

Gotcha! Our announcement of Underwater Data Centers was a playful April Fools’ Day jest. While we won’t be storing data in the deep blue sea, we’re serious about the importance of digital sustainability. As we all navigate the digital landscape, the environmental impact of our online activities requires thoughtful consideration and action.

We encourage you to dive into our genuine insights on digital sustainability and learn how we can collectively reduce our environmental footprint in the digital age. It’s time to make every click, upload, and stream count towards a more sustainable future.

Explore our strategies and join the conversation at Amplify Corporate Green Initiatives with Digital Sustainability.

The post April Fools’ Reveal: A Journey to Digital Sustainability! appeared first on TantusData.

§ 1 [Definitions]

The phrases and terms used in the content of these Rules shall have the meanings indicated below:
1. „Organizer” – TantusData Sp. z o.o. based in POLAND, ul.  Alexa Niepodległości 132/136 unit 3, 02-544 Warszawa, under KRS number: 0000930059, NIP: 5213944638, REGON: 520344085, being the owner of the TantusData brand;
2. “Rules” – these Rules of the Contest entitled “TantusData Summer Challenge”, binding for the Organizer and Contestants, regulating the terms and conditions of the Contest, in particular terms of participation in the Contest, rights and obligations of the Organizer, Contest Committee and Contestants in relation to their participation in the Contest;
3. “Contest” – the Contest entitled “TantusData Summer Challenge”, binding for the Organizer and Contestants, regulating the terms and conditions of the Contest, in particular terms of participation in the Contest, rights and obligations of the Organizer, Contest Committee and Contestants in relation to their participation in the Contest;
4. “LinkedIn” – the website under the domain www.linkedin.com , where the Contest is partially held, owned by the Microsoft corporation.
5. “Fanpages” – the profile named “@TantusData” on the LinkedIn website available at: https://www.linkedin owned and administered by the Organizer;
6. “Contestant” – a LinkedIn user who has made a correct and effective entry to the Contest, meeting the eligibility requirements for participation in the Contest as described in § 3 of the Rules;
8. “Contest Committee” – a team consisting of persons selected by the Organizer, which evaluates the entries;

9. „Winner” – a User who has submitted a correct and effective entry to the Contest and whose entry was awarded by the Selection Committee in accordance with the provisions of §4 and §6 of these Terms and Conditions.

§ 2 [General provisions]

1. The Organiser is the founder of the prizes.
2. The Organizer is the administrator of personal data provided by Contestants.
3. Providing personal data is optional, but necessary for the Participant to enter the Competition. Persons who make their data available have the right to access, modify or delete them.
4. These Rules define the conditions of the Contest.
5. The Contest is not created, administered, endorsed or sponsored by the social networking site LinkedIn or by the Microsoft Corporation or by Kaggle with Google LLC. “LinkedIn” is a registered trademark by the LinkedIn Corporation.
6. The competition is carried out on User’s profile and Organizers Fanpage.
7. The Organizer’s Contest Committee supervises the correctness and course of the Contest, i.e. providing information on the Contest and dealing with complaints.
8. The Competition is conducted from month 21st July 2023 till September 16th 2023.

§ 3 [Contestants]

1. Participants in the Contest may only be natural persons who are consumers, have full legal capacity, are LinkedIn users, have an active account on LinkedIn, abide by the rules of LinkedIn service and who have accepted these Rules.
2. Contestant declares, that he/she/they:
a. is a natural person with full legal capacity;
b. is domiciled in the territory of European Union or United Kingdom of Great Britain and Northern Ireland;
c. is familiar with the content of the present Rules and voluntarily enters the Contest;

d. agrees to and accepts the terms of these Rules, including the procedure for claiming prizes, and fully accepts them;

e. undertakes to comply with the provisions of these Rules and the rules of the LinkedIn service;

f. has agreed to the processing of his/her/their personal data for the purposes of participating in the Contest;

g. is a registered user of the LinkedIn social network.

3. The employees and collaborators of the Organizer cannot participate in the Contest.

§ 4 [Prize]

1. the Prizes will be awarded to the first three (3) persons who, in the opinion of the Contest Committee, best complete the Contest Task and are selected by the Contest Committee as Winners.

2. The Organizer will award 3 prizes to the Participants who will be declared Winners (“Prizes”), one award per winner. The Prizes in the Contest are respectively:

a) For first place – an office survival kit containing:  50 EURO Amazon gift card, a notebook, 2 post its, 5 pens, a mug and a tea, a bag, a t-shirt, cookies, an adult colouring book, crayons,

b) For second place – an office survival kit containing:  50 EURO Amazon gift card, a notebook, 2 post its, 5 pens, a mug and a tea, a bag, a t-shirt, cookies, an adult colouring book, crayons,

c) For third place – an office survival kit containing:  50 EURO Amazon gift card, a notebook, 2 post its, 5 pens, a mug and a tea, a bag, a t-shirt, cookies, an adult colouring book, crayons,

3. Prizes will be issued only in the form specified herein, without the possibility of exchanging them for another material prize or for a cash equivalent. 

4. The prizes will be issued to the Winners in a manner communicated to them in further communication.

5. The Winners will be informed about the prize and the conditions of Prize collection in the content of the announcement on Fanpage and through a private message sent to the Winners on LinkedIn within 10 working days after the end of the Contest.

6. By entering the Contest, the Contestant agrees that the Organizer may communicate with him/her in connection with the Contest via the LinkedIn account.

7. in order for the Winner to receive the Prize, the necessary data must be sent to the Organizer within 48 hours of the prize communication being sent to the Winner (to the email provided by the Organizer in the message). The necessary data for the Prize release to take place are:

a) name and surname;

b) residence address (for tax purposes)

c) correspondence address (for the purposes of delivering the Prize, it may be different from the residential address);

d) telephone number;

e) e-mail address.

9. Dispatch of the Prizes will take place without unnecessary delay from the day of the end of the Contest. The prizes shall be sent via post to the address indicated by the Winner. The Organizer shall not be liable for the actions or omissions of the delivery service providers.

10. The Winner may surrender the Prize but shall not be entitled to the cash equivalent or any other prize. In the event of the Prize being forfeited, the Organiser reserves the right to award the Prize to another Entrant or decide not to award a Prize. The Winner may not transfer the right to the Prize to third parties.

11. The Winner’s failure to provide the Organiser with the data referred to in point 7 above or exceeding the permissible time of response, or sending incorrect data, the Winner loses the right to the Prize.

12. The Organizer reserves the right to verify whether the Winners fulfil the conditions stipulated in the Regulations as well as in legal regulations. For this purpose, the Organizer may require the Winner to make specific statements, provide specific data or submit specific documents. Failure to comply with the Regulations or relevant provisions of law, as well as refusal to comply with the above demands, may result in the exclusion of a given person from the Contest, and shall entitle the Organiser to refuse to award the Prize without any claims against the Organiser.

13. All participants who complete the Contest Task appropriately as determined by the Contest Committee, will be awarded with online certificates of successful completion. The certificates will be issued within 14 days of the Contest closing date and will be sent to the participants via the LinkedIn’s message service. The certificates confirm only the successful completion of the Contest Task and the Organiser is not responsible for the verification of the User’s completion process, only of the correct solution submitted.

§ 5 [Rules of the Contest]

1. The Contestant’s task is tocreate a model to detect whether the same author has written two texts using the datasets provided by TantusData within the challenge on the Kaggle platform. The data provided was collected from articles from our blog and two publicly available datasets of news and blog posts. Participants can also use external datasets. The participants are asked to create a smart solution which can take advantage of even a small dataset. The participants must present this in the format specified on Kaggle under the evaluation tab. The participants are also asked to write a description in a Kaggle notebook and make it public after the submission deadline. The participants are required to submit their entries in Kaggle under the submissions tab for the competition, as specified in the overview. This is the complete Summer Challenge task (hereinafter: “Contest Task”). Full instructions and guidelines are available on Kaggle under the challenge section.

2. One Contestant may make multiple submissions, but will be asked to select their best attempt at the Contest Task. The Contestant must work alone.

3. Contestant is forbidden to take any action in connection with participation in the Contest contrary to the law and good manners, and to use data obtained in connection with participation in the Contest for illegal purposes. In particular in the content of an answer to a Competition Task a Contestant cannot include:

a) materials and symbols against the law;

b) materials and symbols violating the rights of third parties or the Organizer, especially

those violating intellectual property rights or the personal rights of third parties or the Organizer;

c) expressions commonly regarded as morally reprehensible or socially inappropriate as well as content violating good morals;

d) obscene or pornographic materials and content;

e) materials and symbols propagating violence or discrimination, inciting racial, religious or ethnic hatred, socially recognised as offensive, vulgar, etc.;

f) content violating the rules of so-called “netiquette”;

g) personal data of other people in an unlawful scope, in particular it is forbidden to send an answer to a contest task using someone else’s name and surname in order to impersonate a particular person;

h) materials, the use of which by the Organiser may hinder or prevent the operation of other programs used by the Organiser, especially materials containing computer viruses.

4. The answers to the Contest task, which do not meet the requirements specified in the Rules, will be excluded from the Contest.

5. In order to ensure the proper organisation and conduct of the Contest, and in particular to assess the accuracy of entries and select the Winners, the Organiser appoints a Contest Committee, which will supervise and arbitrate the Contest.

6. The Contest Committee will award the accurate solutions to the Contest Task, most swiftly and appropriately realising the Contest Task. Submissions are evaluated based on the accuracy score between the predicted label and the true target and a description of a scaled and optimal solution that best fulfils the task with sensible use of resources.

7. From among the submitted Contest Tasks, the Contest Committee will select the Winners, indicating the decisive features of the selection and deciding also on the allocation of places and the distribution of Prizes.

8. The Organizer will also inform about the Winners in a public announcement on the Fanpage.

9. Detailed information about the contest will be available on the Organizer’s website.

§ 6 [Conditions of participation in the Contest]

1. Access to the Contest is free of charge.

2. Necessary conditions for participation in the Contest are:

a. acceptance of the Rules and the correct completion of all tasks described in § 5 Paragraph 1 of the Rules;

b. granting by the Contestant consent to the processing of personal data described in these Regulations by the Organizer (in particular consent described in § 8 of the Rules);

c. transfer of copyrights as referred to in §10 of the Rules.

§ 7 [Organiser’s responsibilities]

1. The Organiser shall not be responsible for the accuracy and truthfulness of the data of the Contestants, including the inability to pass the Prizes, due to reasons attributable to the Participant, in particular if the Participant did not provide a real mailing address, or if the data provided is incomplete or outdated.

2. The Organiser declares that it does not control or monitor the content posted by Participants in terms of reliability and truthfulness, subject to actions related to the removal of violations of the Regulations or generally applicable laws.

3. The Organizer reserves the right to exclude from the Contestants whose actions contravene the law or the Rules, LinkedIn rules, in particular Contestants who:

a) post content that contravenes the applicable law, Rules, LinkedIn rules (in particular, containing offensive content, both in the text and graphic layer);

b) Complete the Contest Task in with help (not on their own);

c) Interfere with the Contest mechanism;

4. Organizer is not responsible for any malfunctions in data communications links, servers, interfaces, browsers, LinkedIn platform, Kaggle platform.

5. The Organizer is not responsible for temporary or permanent blocking of the LinkedIn profiles and pages or any of its mobile applications.

§ 8 [Processing of personal data]

1. Personal data of Contestants, including image, shall be processed by Organizer only for the purpose of performing activities necessary for proper conduct of the Contest.

2. Personal data of the Competition Participants will be stored by the Organiser only for the time necessary to conduct the Competition and award the prizes to the awarded Participants.

3. Personal data will be processed for the following purposes: publication of competition posts and their promotion in:

a. any social media of TantusData brand (LinkedIn; YouTube);

b. on the Organizer’s website in the domain www.tantusdata.com ;

4. Participants have the right to access, correct and delete processed data. Data is provided on a voluntary basis, with registration on the ?? submission form (e.g. Google form/name of the server) network required for participation in the Competition. The Organiser is not responsible for the way LinkedIn (the service used) processes personal data.

5. The Contestant has at any time the right to request the deletion of his personal data and the data of minors from his post. Upon deletion of the data, the Contestant loses the possibility to participate in the Contest.

6. The use of personal data takes place on global communication channels, without territorial restrictions and without time limits (until the removal of data by the Contestants or the cessation of the Organizer’s activities).

§ 9 [Copyright]

1. It is forbidden to infringe in any way the intellectual property rights in the Contest, especially the unauthorized use by the Contestants of works authored by third parties.

2. Contestants in respect of whom the Organiser has received information that they are not the authors of Contest Tasks, or do not have the rights to the answers to the Contest Task, are subject to exclusion from the Contest.
3. In the event that an answer to a Contest Task is a work as defined by the Polish Act on Copyright and Similar Rights, the entirety of the Participant’s economic copyright to the answer to the Contest Task and the right to permit the exercise of subsidiary rights, without any time and territorial restrictions, passes to the Organiser. The Organiser shall be entitled to use and dispose of the work – the contest posts – in the following forms of exploitation:
a) within the scope of recording and multiplying the work in whole or in part – the production of copies of the work using a specific technique, including digital reproduction, printing, reprography, magnetic recording and digital technique;
b) within the scope of dissemination of the work, in its entirety or in part, in a manner other than specified in item a) above – placing on social media, websites, publishing in the Organiser’s promotional and advertising materials, exhibiting, displaying, reproducing, as well as broadcasting and rebroadcasting, as well as making the work available to the public in such a way that everyone can have access to it in a place and time selected by themselves, including via the Internet.

§ 10 [Complaints and notifications of violations]

1. Any complaints regarding the way the Contest is carried out should be submitted by Contestant in writing during the Contest, but not later than within 14 (fourteen) days from the date of issuing the Prizes.
2. A complaint submitted after the deadline shall have no legal effect.
3. A written complaint should contain the name, surname, exact address of the Contestant and
a detailed description and justification of the complaint.
4. The complaint should be sent by registered mail or courier service to the address of the Organiser.
5. Claims will be considered in writing within 30 days from their submittance.

§ 11 [Final provisions]

1. The Regulations shall enter into force on the first day of the competition, i.e. July 21st 2023.
2. Second In matters not covered by these Regulations shall apply the provisions of the Polish Civil Code and other applicable laws.
3. Disputes relating to and arising from the Contest will be resolved by a court of law with jurisdiction over the Organiser’s registered office.
4. Organizer reserves the right to change the rules of the Contest during its duration for important reasons, with the reservation that all possible changes in the Rules will not affect the rights of Contest participants acquired on the basis of the Rules before the date of entry into force of these changes. Information about changes will be posted on the Fanpages.
5. A brief description of the Contest rules can be found in advertising and information materials accompanying the Contest, in particular on the Fanpages. All content included in these materials is for information purposes only. Only the provisions of these Regulations are binding.
6. Contest Regulations are available on Fanpages.

The post The TantusData Summer Challenge: terms and conditions appeared first on TantusData.

Introducing the TantusData Summer Challenge

This summer, we’re not just beating the heat, we’re challenging it. At TantusData, we believe in nurturing the approachable spirit of knowledge-sharing, and to celebrate this, we’re excited to present the TantusData Summer Challenge.

Our challenge is an ode to experts in the making and a refresher for the seasoned professionals. It’s a platform to test your skills and learn, as we will post the optimal solution with instructions later. As a cherry on top, there’s a reward – an ‘Office Survival Kit’, filled with a delightful mix of practical and amusing items. This includes a handy notebook, colourful post-its, essential pens, a sarcastic mug paired with a soothing tea, a stylish t-shirt, a tote, appetising cookies for that quick brain boost, and for moments of relaxation, a funny adult colouring book with crayons. To further enhance your office or workspace, we’re including a 50 EURO Amazon gift card as part of the kit. We have three of these exciting kits ready for our top participants. So buckle up and let your creative juices flow!

At the heart of this challenge lies a model on production topic. We teased this during recent ML conferences and now, it’s time for action. The challenge requires the innovative use of a Siamese Network solution to work with small data sets. Entries will be judged based on the quality and appropriateness of the submission. The finer details are provided below.

And that’s not all – all participants who solve the challenge correctly will be awarded certificates.

But wait! We’ve got a bonus round. A side challenge that promises more thrill and more creativity. Stay tuned to our emails and social media channels, specifically LinkedIn, for the announcement.

A little birdie told us that those who participate in this challenge will have a head start for our upcoming autumn challenge. So, we highly recommend you to keep your work saved.

The how’s and when’s

The challenge instructions: Your task, ‘Authorship Comparison’, is simple but thought-provoking: create a smart solution to determine if two texts share the same author. Rather than developing a massive model, we want to see how well you can optimise a limited dataset. The submissions will be assessed based on the scores and the approaches taken.

All submissions must be through the Kaggle submission system, and if you’re not a member yet, creating a login is easy and free. Check out the detailed instructions and resources on the competition page here.

Deadlines: The Authorship Detection challenge opens on the 21st of July 2023. We strongly recommend participants to commence no later than the 11th of September 2023, to provide sufficient time to develop and fine-tune your model.

Important Update: In response to the feedback from our enthusiastic participants, the submission deadline for the Summer Challenge has been extended. All entries will now be accepted until 16th of September, 2023, 23:59 CEST. This provides additional time for both new participants to join and for existing participants to refine their submissions. We wish everyone the best of luck! No submissions will be accepted after this time, so ensure your entry is submitted promptly.

For beginners and those looking for extra support, we’ve provided tips and resources linked within the challenge on Kaggle. These could be very helpful, so make sure to take a look. But remember, going through these resources might take some time in addition to the time needed for the challenge task itself.

Get started now, and best of luck to you all!

The Fine Print: Here’s the link to all the terms and conditions of the challenge. Yet, the basic rules of participation are also listed on the challenge tabs in Kaggle, which will go live tomorrow. So you can view them there.

Get ready to compete, learn, and win. And remember, the clock is ticking. Keep an eye out for our announcements!

Good luck, challengers!

#TantusDataSummerChallenge

The post The Summer Challenge appeared first on TantusData.

J On The Beach: Where Developers and Data Scientists Meet to learn and unlock the potential of Big Data Technologies

EDUCATE, SHARE, SPREAD, AND LEARN

The technical conference brings together developers, data scientists, and DevOps professionals from around the world to explore the latest trends in big data technologies. This immersive event features workshops, hackathons, and technical talks led by top speakers in the field. Attendees will learn about a range of topics, from data collection and stream processing to machine learning, microservices, artificial intelligence, container systems, and more.

You can read all about this year’s edition here.

On the 10 to 12th of May, fantastic international speakers will join the stage to share their top tips and tricks.

Our own Marcin Szymaniuk will share his best practice takeaways on the 12th of May. You can find more about the topic our expert will present here.

Did we mention that it just happens to be Friday and the conference will conclude with a great party in the evening?

So make sure to book your place and join us in the gorgeous Málaga for invaluable insights and strategies, you won’t learn elsewhere.

The post J On the Beach x TantusData appeared first on TantusData.

Clutch has recently brought to our attention that we are listed as a leading service provider on Clutch; naming us as one of Poland-headquartered industry game-changer big data analytics firms.

For those few who haven’t heard of Clutch yet, it is a B2B ratings and reviews platform based in Washington, DC. The company evaluates technology service and solutions companies based on the quality of work, thought leadership, and other key highlights from clients’ reviews.

We are a data company specialising in big data solutions. Subsequently, we excel at data platforms, data-driven systems, analytics, and advanced machine learning. As such, our services range from data infrastructure and cloud optimisation to ELT and re-engineering mission-critical systems. You can see a selection of our case studies at Clutch in the portfolio section.

Let’s hear it from the important ones

Our clients’ honest and thorough reviews on Clutch have a tremendous impact on us. Because, their feedback helps us improve our services and establish our name in the industry. They also show that we do deliver in a manner that fits our values. Today, we will share some snippets of the reviews we have received so far.

“We chose TantusData because of how flexible they were in the way of delivery. It was important for us to have their expert working alongside our internal team full time.” – Principal Engineer, UK’s fastest growing online accommodation marketplaces.

This doesn’t come as a surprise. After all, flexibility is at our core. Indeed, we start by focusing on you, to meet where and when you need it, and bring a solution that best fits your company and situation. Therefore, we guarantee the results: what we promise, we deliver. Finally, from start to finish, your project is only taken care of by experts only. Those who have long standing experience in the industry.

“The onboarding and collaboration was smooth since both technical and communication skills of the team were very good”. – Director of Data & Advanced Analytics, large scale retail company

We speak your language and ensure that the problem is not only solved but also in a way and at a pace convenient to you. Whether you have a long-term project, need extra hands on board mid-delivery, or need some crucial last-minute help — we’re quick to respond. 

“Effective communication and very quick turnaround when unexpected problems arise.” – CTO, 360 Restaurant Management System provider

“One of the key resources of the company is their unique skill set. Such a mix of data science and engineering coupled with clear understanding of rules of GDPR and ability to tune performance is hard to find on the market.” – Head of Data Science, one of the largest telecom companies in Europe

We are a flexible, agile, and unequaled group of experts. This means we can address any situation and provide the best solutions, 100% tailored and effective long-term. So, it’s not about the new and popular solutions but the ones that will serve you best.

“I appreciated that they investigated several solutions and delivered them to us with an explanation. They genuinely cared about the outcomes of the models they built and stayed actively involved throughout split testing on live environments.” – Principal Engineer, UK’s fastest growing online accommodation marketplaces.

Simultaneously, TantusData ensures you have all the knowledge you need straight from our specialized team so that you can fully leverage the solution to create value but also confidently make decisions regarding data.
“We could see that they really cared about meeting our needs, but most importantly about the project long term, that it goes well.” – CTA, Depict.ai

Our badges

The whole team at TantusData is proud to be the recipients of the special Clutch Badges for the top providers. To be sure, we will continue to go the extra mile, no matter the project or business size. We are delighted to see our clients appreciate our hard work. If you’re curious to see more about the reviews, check out our Clutch profile. There you can also view selected projects from our portfolio.

We help our clients get up to speed with taking advantage of their data. Contact us now!

The post TantusData, the game changer by Clutch appeared first on TantusData.

The SmartData online conference organized by the JUG.ru committee from Russia took place on Oct 11-14. It was full of interesting presentations and discussions going in 3 threads in parallel. I’d like to briefly state here some of the topics presented.

Apache Airflow 2.3 and beyond: What comes next? By Ash Berlin-Taylor from Astronomer.io

Ash has been a contributor to Airflow for almost four years. He was the Release Manager for majority of the 1.10 release series and he also re-wrote a lot of the Scheduler internals to be highly-available and increase performance by an order of magnitude. Outside of Airflow he is the Director of Airflow Engineering at Astronomer.io where he runs the team of developers contributing to the open source Airflow project.

Ash started by mentioning Airflow recent achievements, like performance 10-100 time speed up of Airflow 2.0 released in December last year. Airflow 2.2, which is about to be released, contains following updates:

i. AIP-39: Run DAGs on customisable schedules . Confusing ?execution date? deprecated, introduced ?logical_date?, ?data_interval_start?, ?data_interval_end? instead.

ii. AIP-40: Any operator could ?defer? itself. Deferrable(async) operators are generalizations of  smart sensors introduced earlier. Helps to avoid  waste of resources while waiting for external dependency to complete.

Roadmap for version 2.3 is not yet finalised and Ash presented his own vision of possible Airflow’s future. 

1. DAG should be a joy to write and read which makes Airflow an orchestrator choice for any workflow. Make it easier to operate confidentially.

2. Dynamic DAGs.  Mapped task concept could make it possible to launch as many copies of operators, like map tasks, to process all files in parallel. Parametrized DAG defined once could be used with different parameters.

3. Get rid of the UNIQUE constraint on execution_date value to allow scheduling multiple DAGs at same time.

4. Introduce airflowctl: CLI over REST API.

5. Solve ?untrusted worker? problem – set access control per connection.

6. Make it easier to assign lifecycle hooks and DAG notifications, like send slack notification on failure.

7. Better cross-DAG story: event-triggered DAGs and introduction of a concept of Data object. It should be possible to bind a storage folder with Data object reference and assign a hook which triggers task execution upon content change. When DAG execution is completed all temporary files could be automatically cleaned up.

Among even more distant future ideas these were presented the following:

1. To append DAG versioning and reflect the historical state of DAG on UI. Having introduced versioning will help to simplify DAG deployments. 

2. Streaming support and better support for microbatch and long running batch jobs.

3. Support and integration with Machine Learning tools, model hosting, comparing models.

Big Data Tool presentation by Oleg Chiruhin from JetBrains

Big Data Tool is a powerful plugin for IntellijIdea which allows to work with Zeppelin notebooks, monitor Spark and Hadoop applications and explore cloud storage systems.

I. Browser file storage. Possible options are local FS, HDFS, Google Storage, S3, Azure, Minio and some more. When local and cloud storages are configured it is possible to upload files in any direction. Also it?s possible to preview remote file sample with different view options – plain text(for csv) or table view:

II. Zeppelin notebook editor with ssh tunnelling for accessing remote notebooks within a restricted network.

BDT appends rich scala autocomplete functionality missing in the web interface for Zeppelin.There is also ability to add  external modules and jars for autocompletion and in-place documentation, not only standard library.

Navigation to code declaration and other opportunities of IDE are supported.

III. Markdown support

IV. Easy imports

V. Intermediate type visibility

VI. Plot execution results

VII. Ability to monitor job execution

Hadoop 3: Erasure coding catastrophe by Denis Efarov from OK (Odnoklassniki)

Denis Efarov, a lead developer at Odnoklassniki, has been working with Big data since 2013. He has designed and developed the platform for storing and processing data for the Odnoklassniki project since 2018.

Denis told an extremely exciting and dramatic story about migration from Hadoop 2.7.3 to 3.1.4 and back, which cost the company more than 1 year of useless work and around 115 Tb of lost production data. 

Hadoop 3 introduced erasure coding mechanism instead of replication to protect against data loss. This mechanism appends 3 additional bytes of Reed-Solomon codes after each 6 bytes of original data (parity block). It leads to only 50% disc redundancy compared to 200% in case of replication with factor 3x with similar reliability guarantee. Of course the encoding/decoding process slows down read/write performance, but space preservation was a priority and in scope of OK it was tens of petabytes scale. 

The whole migration process was expected to take 8-10 month, but after half a year they noticed that some parquet files were broken. At first glance damage seemed to appear randomly, but after deep analysis they detected an issue with parity blocks cleanup (see these links: https://issues.apache.org/jira/browse/HDFS-14768 and https://issues.apache.org/jira/browse/HDFS-15186). By this time 2 out of 3 redundant clusters were already migrated and 4 000 000 of files were broken. 90% was restored from the remaining backup cluster. In order to restore the remaining part they developed a tricky decoding tool which was trying to guess original data from all possible byte combinations and based on parquet file structure. This way about 9.9% of files were restored. And approximately 40 000 production files (115 Tb) were lost forever.

The post SmartData conference 2021 appeared first on TantusData.