There are countless opinion articles out there on how to be a good manager or what makes an inspirational leader. This is not one of those. I was a manager for no more than 3.5 years, not nearly enough time to know how to do it right – if there is such a way to do it “right”. However, I thought that my experiences, successes, and failures would be useful to those out there who might be considering moving into a managerial role in the data science field, so here you go.
In my view, I was a pretty subpar data science manager, and that’s not the imposter syndrome talking. It’s nobody’s fault. I’m not out to blame anybody, or dump on myself, it’s just an observation. But before I get into why I was subpar, let me tell you the how and why I became a manager.
Going From Individual Contributor Data Scientist to Data Science Manager
I was an individual contributor (IC) data scientist for around five years before I moved into management. Moving into management was a strategic decision on my part. I was a bored IC. Bored with the day-to-day monotony of data science work (collect, clean, transform, and model data, fix bug, rinse and repeat). I did not have any interest in breaking into a new niche of data science, such as computer vision, which (a) wouldn’t be easy to find the time to do and (b) might eventually bore me just the same. I was also doubtful that I could remain employable into my 40s and 50s as an older, slower, more expensive IC with no formal leadership or management experience. The next logical step for me and my career was to move into management and see if it sticks.
It was certainly the right choice for me. I really wanted to give it my best. I’ve always enjoyed mentoring others. I knew that the skillset for being a manager and leader would be different, but I was excited to learn and apply those skills – skills like:
Conflict resolution
Interpersonal and communication skills
Project management
Delegation
Decision-making
Roadblock-removing
If any of those sound terrifying to you, don’t be a manager.
I went through all of the company-provided leadership trainings in subjects like unconscious bias and building high-performing teams. I read blog/LinkedIn posts from well-followed “influencers”. I thought deeply about the traits in my own managers that I liked and didn’t like. I used all of this, and put together a persona of who I thought a good manager is, and what I should do.
My First Attempt at Managing
Things started off…bad. With my first direct reports I was trying so hard to not be a micro-manager that I failed to do much managing at all. You see, I fall into the camp of data science leaders who believe that you should trust your team to be able to get the job done. I believed that data scientists are highly educated, driven, and intelligent individuals that shouldn’t need their hands held, and that the best thing that I can do as their manager is to get out of their way and let them shine. I was wrong. Partially.
Yes, data scientists are educated, academically.
Yes, most data scientists are intelligent, relatively.
No, data scientists are not inherently driven and good at their job.
This was my first lesson learned. Some data scientists require more hand-holding than you expect. Some will not know all of the ins and outs of ML and analysis even though they have advanced degrees and prior experience. Most will not be good at the soft skills such as communication, creating and delivering good presentations, and managing their time appropriately. And many won’t have a high emotional intelligence. As their manager, I should’ve spent the time to assess my directs on each piece, before letting them fly, and watching them crash.
Building My First Data Science Team
After about six months of managing a team I was given my first opportunity to hire new data scientists. Just for me. I was actually excited to start reading resumes, doing interviews, and making hire/reject and salary decisions. As I began this process, I again found myself in a certain camp, the one that believes that the data scientist interview process is ridiculously broken, with time-sucking homework assignments and round after round of difficult unrepresentative technical and coding interviews. In my efforts to fix these problems, I went too far in the opposite direction:
No homework. No coding. Three rounds and done.
My success rate in hiring good candidates with this approach? 50%.
This was my second lesson learned, and I fixed it, with much success, in my subsequent hiring rounds by adding a reasonable coding interview to screen for foundational knowledge, and a more rigorous case study interview. We kept it as succinct as possible, and moved candidates quickly through.
Managing as an Introvert
I am most certainly an introvert. Each meeting I had, no matter how short, drained me. And since most of my meetings were back-to-back, I never had a moment to recharge for the next one. Add to this the other common stresses of managing, such as dealing with people issues and being accountable for all projects my team is working on, and I quickly became exhausted by my job.
I don’t want this to be a post about how introverts find it difficult to thrive in corporate leadership roles. I don’t know if that is even true. But, there are challenges, and I personally couldn’t get passed them.
Going From Hands-on to Hands-off
Earlier I said the skillset for a manager is different, and that was doubly true for me. My role was not one of those 50/50 hands-on/managing roles. I did not write code or look at raw data. I am not even sure it would have been possible for me to be a good manager if I were still trying to focus on meeting project deadlines.
At the end of the work day, I was too burned out from my meetings to even care about data science and AI. I didn’t want to talk about it, hear about it, or read about it. I lost the excitement I once had for the field, and my job had become just “a job” – the thing I do for a paycheck.
I wouldn’t say that I particularly missed being an IC data scientist. I didn’t want to build ML models on crappy data again. I had done enough of that. What I missed was creating something, anything, tangible and of value. I missed seeing results of my work in a plot or a table or a dashboard. It wasn’t rewarding enough for me to see my team be successful and to know I had a part in that.
Should I Quit or Talk to My Manager?
A couple of years into managing I had a large team of nearly 20 data scientists. I was managing managers. And I had started to have feelings of dread about going to work. I began asking myself “should I just quit, or should I talk to my managers about my role?”. I scoured the interwebs, reading blogs and subreddits, to seek advice. The advice I got was near universal – talk to my managers. So I did. I told them I didn’t want to lead a large team anymore, and they were very supportive. They agreed to work with me to narrow the focus of my role and manage a much smaller team.
I realize how fortunate I was to have people in leadership that were understanding enough to try to make things work. They didn’t complain to me, they didn’t try to guilt me into staying in my role, and they didn’t push me out of the company. They were great and they really tried to make me happy. Not everybody out there will be that lucky.
This was my third lesson learned: if something can be better or different at work, talk to my managers instead of running away.
I should note that it was also around this time that I was considering a career change. I actually did end up leaving the company about one year later to pursue a new path in Academia. I will write about this in a future post.
My Failures
I do not have a dynamic charismatic personality (gasp!). When I walk into a room, eyes are not drawn to me. I don’t have “it”. Can you be an inspirational leader without these things? Maybe. But I found it difficult, in a remote role, at a huge company, to be one.
As an introvert, meetings were draining, and I avoided adding more to my calendar. This meant that I did not meet with my team as much as I should have. I didn’t mentor them, I didn’t look at their code, I didn’t ask to see their results, and I’d go so far as to say that I barely led them. This worked out ok for my high-performing team members, and I’m sure most of the team was happy to be left alone. But some projects suffered. And some team members were not happy with my lack of guidance.
My Successes
Overall, despite my failures above, the team was happy and was doing really great work with tangible value for the company.
I had high expectations of my team. I wasn’t satisfied to be a manager that lets their team coast in order to avoid conflict. I wanted strong performers. But, having high expectations meant that I had to deal with issues that quickly wore me down. I didn’t have the stamina to keep it going.
I advocated for my team. I tried to get them raises and promotions when they were deserved.
I cared about my team members. I tried hard to get to know them, to keep them happy, and to have empathy.
I’m very proud of the brilliant and diverse team that I helped build.
I never took credit for anybody else’s work and I took responsibility when a project wasn’t quite successful.
Final Thoughts
This post may seem overwhelmingly negative, and admittedly some of it is a bit of an exaggeration, but I’m glad I chose to be a manager. I did have some good times. I learned a lot and I did my best. But it was difficult, and I wouldn’t say I was great at it. I wrote this post to remind my future self what it was like, in case I ever start having silly thoughts that I might want to do it again.