Generative AI: The missing tool in a data generalist’s belt
The missing tool in the toolbelt or, to be more exact, the Swiss army knife of the information age. While some fear it might pave the way for the erosion of our cognitive abilities, I choose to remain optimistic and embrace this versatile tool.
In my perspective, generative AI acts as a crucial bridge for data generalists, offering them the means to swiftly acquire a broad, albeit shallow, understanding of any specific topic. Instead of shying away from its potential, I firmly believe that by harnessing the power of generative AI, we can accelerate value to businesses, which is ultimately what truly matters.
When I started using chatGPT at the end of 2022, it didn’t take long for me to silently echo the famous Jerry Maguire line “You complete me.” However, I couldn’t utter this out loud because then it would have confirmed everyone’s suspicion of my nerd status, which I tried to suppress for years.
When I previously wrote about the benefits of being a data generalist¹, chatGPT was not yet available, and generative AI was not the buzzword that it is now. Here, my second argument was that with information so readily available, the need for hyper-specialization reduces.
Generative AI strengthens this argument considerably. If we thought information was freely available pre-chatGPT, it has just leapfrogged to a whole other level.
The data generalist was already skilled in being able to quickly assess a myriad of internet search results. They were also skilled in asking good questions and “prompting” their human peers to give them the answers they were searching. chatGPT is like the perfect combination of Google and a knowledgeable peer. For me, prompting chatGPT to correct mistakes or make adjustments comes naturally as I have been doing it to my human peers for years, but now I don’t need to feel bad for wasting its time. It is, in fact, just an algorithm without feelings… right?
Here are some areas where generalists are thriving because they use chatGPT:
Rapid Prototyping
I find it challenging to comprehend theory without practical application. I struggle to muster the motivation to engage in online courses; I need the prototype or project to truly master the theory. However, I used to face a similar level of frustration when I attempted to prototype something. This was due to various mundane hurdles that demanded my attention, including insignificant syntax errors and non-obvious IT glitches. I would spend 60% of my time trying to set up the prototype, 30% bashing my head against the wall, and only 10% actually prototyping the hypothesis I was set out to prove.
That ratio has now significantly changed. Nowadays, one can rapidly prototype anything in a very short period of time. When you are still busy analyzing data, sifting through sample datasets and files, you often deal with various types of mundane and random tasks. Here are some typical examples that I would need to do, which bring me little satisfaction:
- PowerShell/Bash scripts that handle files and filenames.
- Dynamic SQL statements on bulk tables, such as DDL scripts and truncates.
- Handling and deconstruction of files like JSON.
Rapid prototyping was a skill reserved for the savants among us — the cool kids who could build apps overnight. Now I can do it; sure, my code is by no means production-grade, but I am simply prototyping at this stage.
The generalist does not want to get bogged down in nitty-gritty syntax and coding issues; they know there is someone better than them who can solve this. However, they do want to be able to test their theories rapidly. Having this helper plays to the generalist’s advantage. They can think big picture, have an idea, and now within a short space of time, prototype the idea to test the feasibility and highlight any shortcomings.
Is ‘no-code’ still that important with chatGPT
I have never really liked pure no-code solutions. I often find no-code platforms very limiting if you cannot configure the way they generate the code. Eventually, there will be something that the no-code interface cannot do. And if you cannot change the configuration, you will end up doing something outside of the tool.
Some vendors started their journey as no-code applications but later drifted towards the low-code slogan. Still catchy, just subtly different.
One of the reasons that no-code platforms are attractive is because of the ease of use and appeal to non-technical or non-specialist users to maintain the solutions.
With generative AI lowering the bar considerably, now anyone can read and interpret code and can prompt changes to the code. It is a game-changer, and I argue it reduces the appeal for no-code solutions.
No more hours and hours of code debugging — now you can analyze, adapt, and test before boredom strikes. Some specific areas where I have used this are:
- Analyzing a dbt macro or other templating tool’s templates and then adjusting them accordingly.
- Analyzing and adjusting an existing dynamic piece of SQL code.
- Analyzing and adjusting code of languages I am not familiar with.
Removing the feel-like-an-imposter feeling
We generalists sometimes feel silly when talking to specialists because they use words and acronyms like NLP, neural networks, distributed, IoT, GPU, fault-tolerant, idempotent, scalable, and other things that sound complicated. At the beginning of one’s career, you resist the need to stop someone and say, “please explain that to me.” However, as you develop and mature in your career, and specifically in my career, I have made peace with one thing:
Even if I don’t know everything, I trust that I can figure out ANYTHING
So even if I don’t know what they are talking about now, I can figure out what I need to know in a short space of time. Having this in the back of my mind makes me more confident to ask ̶s̶t̶u̶p̶i̶d̶ questions.
Pre-chatGPT, I had to Google, and it might take me a day or two. But now, this has been reduced to a couple of hours.
I remember in the beginning of my career, I used to write down acronyms in a meeting and then Google them after the meeting. These days, with remote working and chatGPT, I can do it quite seamlessly during a meeting. chatGPT also helps you to ask good follow-up questions. And boom — just like that, gone is the feeling like an idiot or imposter.
My prediction points to a future where we witness an upward trend in generalized roles, with AI playing a vital role as our specialized helper. While some may harbor concerns and pessimistic views about AI, I find myself leaning more towards optimism. Only time will validate this, but I firmly believe that accepting AI as an ally in the workplace is essential to maintain relevance in this rapidly evolving landscape.
To remain effective contributors, we must embrace AI’s potential as a helper and focus on our core mission — assisting companies in solving their fundamental challenges. The data generalist, once a pivotal figure even before the advent of ChatGPT, now possesses an invaluable new tool in their toolkit — the Swiss army knife of the information age.
[1]: Range over depth — the value of a generalist in your data team, Medium. Available at: https://medium.com/towards-data-science/range-over-depth-the-value-of-a-generalist-in-your-data-team-174d4650869d
