Introduction
It has been very intriguing to see how AI improves and becomes very accessible. ChatGPT being released to the public on 2022 was the start of a new era in tech.
I was a bit skeptical at first. I had a uni assessment using Haskell, and I was very sure that ChatGPT could not help me with it. I was wrong.
Even when ChatGPT was not as good as now, it still solved a Haskell problem that I struggled for a few hours quite well. I was amazed but still a bit skeptical.
During those times till 2024, I was not using AI much. I was just utilizing ChatGPT as my "Google", to just ask questions and get answers - not as much as a coding tool.
Beginning of 2025, I started to help an AI Ed-tech startup, Quanta, to develop newer features and I decided to give AI a try, still ChatGPT. I was amazed by how much it improved over the years.
It was not just a "Google" anymore. It felt like a pair programmer that I can ask to write code for me, share ideas, and even review my code. It was like having a very knowledgable coworker that I can rely on.
Quanta - by Arsenii Nikolaiev, is a platform where students can submit solutions (text, images or pdfs) and get their feedback from AI. The AI goes through a few phases of checkings (ideas from Arsenii) where it checks for correctness, efficiency, and style.
It is a very useful tool for students to learn and improve their math skills. Kudos to Arsenii for building such a great product.
During the process of developing features for Quanta, I was tasked to onboard two new devs, Faisal and Harikesh. I was amazed by how fast they develop new features and understand the codebase very fast. Apparently they were using Claude Code/Cursor.
Features that would take me days to finish with pure querying ChatGPT, they were able to finish in a few hours. It was like they had a superpower that I did not have. I was very impressed and also a bit jealous.
I hadn't decided to go with Claude/Cursor yet because Harikesh and Faisal's work were very inconsistent that time.. I think that's how AI works, without a good context and memory, it can be very inconsistent. For example: buttons created would not have the same style - most of the times missing cursor: pointer;
After a few months, I left the project, but I still occasionally kept an eye on the product and the features being developed. One thing that made me hesitant about AI-assisted coding tools was seeing how easy it was for teams to move fast while missing important security details.
For example, in some projects, secrets, database access, or environment variables can accidentally end up exposed through public endpoints or frontend code. That kind of issue is not always obvious when the feature appears to work, but it can become a serious production risk.
After I left the project, I started to use Claude to help me with my personal projects - particularly a badminton matchmaking app that I am building. I was able to finish the MVP of the app in a few hours, which would have taken me weeks without AI. AI clearly gives devs a superpower to develop features much faster than before, but after using it more seriously, I also started noticing some problems that are worth talking about.
Problems I see with AI tools in tech
1. People can showcase code they do not really understand
At the time of writing this, it feels like a difficult market for fresh graduates trying to enter tech in Australia. Some major companies have announced layoffs or restructuring while also investing heavily in AI. Atlassian, for example, announced a reduction of around 1,600 employees globally in March 2026, and Telstra’s AI joint venture with Accenture reportedly proposed cutting more than 200 jobs earlier in the year.
I do not think AI is the only reason the job market is difficult, but it is clearly changing how companies think about productivity, hiring, and what they expect from developers.
I am considered lucky to have a job in this market, it's all thanks to my ex-manager, Lloyd. I applied for a technical support role but he hired me as a graduate dev instead. More on Lloyd in another post.
On LinkedIn, I have seen a lot of fresh grads showcasing code that they do not really understand. Even the post that I read at that time was written by AI. The code I read was undoubtedly clean, but I could really tell that it was written by Claude (of course the Claude.md trademark). Some jobs require exposure to AI-assisted development, but I think doing stuff like this is too much.
1 commit, 1 PR, 1 massive feature, all generated by AI. .env files with secrets being shared in public GitHub repos. I have seen all of these things happening in the past few months.
The issue is not that they used AI. I use AI too. The issue is when someone presents AI-generated work as proof of skill without understanding the code, the architecture, or the risks behind it.
There is a big difference between using AI as a tool and being carried by it.
2. AI features can be manipulated if they are not designed carefully
Another issue I have noticed is that some AI-powered apps are surprisingly easy to manipulate.
I once tested an app that had a roleplay mystery game where users could chat with different crime-related characters. The idea was interesting: each character was powered by AI, and the user could investigate the mystery by talking to them.
But when I tested it, I was able to convince the AI to break character, pretend to be someone else, and even reveal information about the model behind the app.
This is where AI development becomes different from normal software development. With a normal backend endpoint, you define the input, output, validation, permissions, and business rules. With an AI chat feature, the user input is much more open-ended. If the system prompt, context, and guardrails are weak, users can manipulate the AI into doing things the developer did not intend.
This is not just a funny bug. In some products, it can become a serious issue. An AI assistant might reveal hidden instructions, expose internal logic, leak sensitive context, or give users access to information they should not have.
That experience made me realise that building AI features is not just about connecting to an API and adding a chat box. Developers need to understand prompt injection, role boundaries, data exposure, and how to design AI systems that fail safely.
3. AI can be a crutch that prevents learning and growth
Another problem I see is that AI can make it easier for developers to skip the learning process.
When we are stuck, it is very tempting to ask AI for the answer immediately. Most of the time, it will give us something that looks reasonable. Sometimes it even works. But if we keep doing that without taking the time to understand the solution, we might get the output without gaining the knowledge.
This is dangerous because struggle is part of learning. Debugging an error, reading documentation, tracing through code, and trying a few wrong approaches are all painful, but they help us build intuition. If AI removes too much of that friction, we may become faster in the short term but weaker in the long term.
I have noticed this in myself too. Sometimes I ask AI to solve something too quickly instead of thinking through the problem first. It feels productive because the task gets done faster, but I do not always retain the lesson as well as I would have if I had struggled with it properly.
This does not mean we should avoid AI. It means we need to use it intentionally. Instead of only asking AI to “fix this”, we should also ask it to explain the issue, compare different approaches, and challenge us to understand why the solution works.
The goal should not be to let AI replace our thinking. The goal should be to use AI to improve our thinking.
4. Security risks become easier to overlook
One of the biggest concerns I have with AI-assisted development is that security risks can become easier to overlook.
AI is very good at producing code that looks complete. It can create login flows, backend routes, database queries, file upload handlers, API integrations, and deployment configs very quickly. The problem is that “working” does not always mean “safe”.
When we move too fast, it is easy to accept AI-generated code just because the feature appears to work. The app runs, the button works, the API returns data, and the demo looks good. But underneath that, there might be serious issues: exposed API keys, weak authentication, missing authorization checks, overly permissive CORS settings, unsafe database queries, public endpoints that should be private, or environment variables accidentally pushed into the frontend.
This is especially dangerous for developers who are still learning. AI can generate patterns that look professional, but if the developer does not understand the security implications, they might not know what to review. They may not notice when a secret key is being used in frontend code, when a database rule is too permissive, or when an endpoint allows users to access data they should not be able to see.
I think this is where AI can create a false sense of confidence. Because the code looks clean and the feature works, it feels production-ready. But production-readiness is not just about functionality. It is also about authentication, authorization, validation, logging, error handling, rate limiting, data privacy, and safe deployment practices.
For example, if AI helps build a feature that fetches user data, the important question is not only, “Does it fetch the data?” The important questions are: “Should this user be allowed to fetch this data?”, “Can they access someone else’s data by changing an ID?”, “Are we exposing anything sensitive in the response?”, “Is this endpoint protected?”, and “Are secrets only used on the server side?”
This does not mean AI should not be used for security-related code. In fact, AI can be very useful for reviewing code, finding possible vulnerabilities, and suggesting safer patterns. But we should not blindly trust it. Security still needs human understanding and deliberate review.
The faster AI allows us to build, the more disciplined we need to be about checking what we are building.
Why I still think AI is extremely useful
Even with all the problems I mentioned, I do not think AI is bad for software engineering. In fact, I think AI is one of the most useful tools developers have today.
The issue is not AI itself. The issue is how we use it.
For me, AI has been extremely useful for learning new concepts, debugging unfamiliar errors, generating boilerplate, reviewing code, and exploring different approaches. It lowers the barrier to starting something. Instead of spending hours stuck on setup or syntax, I can ask AI to help me get moving and then focus more on understanding the bigger picture.
This has been especially helpful for personal projects. When I built my badminton matchmaking app, AI helped me move from idea to MVP much faster than I normally could. It did not mean I could stop thinking. I still had to decide what the app should do, how the data should be structured, how the UI should behave, and whether the generated code actually made sense. But AI removed a lot of the friction between having an idea and seeing something working.
I also think AI is very useful as a learning companion. Sometimes a documentation page or Stack Overflow answer gives you the answer, but not the full explanation. With AI, I can ask follow-up questions, ask for examples, compare approaches, or ask it to explain something in a simpler way. That kind of interactive learning is genuinely powerful.
At work, I think AI can help developers become faster, but only if they already have enough understanding to review the output properly. It can help with repetitive code, test cases, refactoring ideas, documentation, and debugging. But it should not replace reading the codebase, understanding the domain, or thinking through the design.
So my view is not that developers should avoid AI. I think the opposite. Developers should learn how to use AI well.
But using AI well means being intentional. It means giving it proper context, checking its output, understanding the trade-offs, and not letting it do all the thinking for us.
AI gives developers leverage. But leverage only helps if we know where to apply it.