Lawgeex’s Julia Shub – The Importance of Processes

Liran Haimovitch: Welcome to The Production-first Mindset, a podcast where we discuss the world of building code, from the lab all the way to production.

We explore the tactics, methodologies, and metrics used to drive real customer value by the engineering leaders actually doing it. I’m your host, Liran Haimovitch, CTO and co-founder of Rookout.

Today we’re hosting Julia Shub, DevOps lead at Lawgeex. She’s a rising star in all things DevOps and a pillar of the Israeli community. Thank you for joining us and welcome to the show.

Julia Shub: Thank you for having me, and thank you for calling me a rising star. It’s very flattering.

Liran Haimovitch: Julia, what can you share about yourself besides obviously being a rising star?

Julia Shub: First off, thank you for having me here. What I can say about myself is that I’ve been a DevOps engineer for three years now. These whole three years I have been working at Lawgeex as Lawgeex’s DevOps lead. Before that, I spent another 11 years in the Israeli high-tech community as a QA tester, an automation developer, team leader, project manager, and QA department manager. I spent a lot of my time in management and I think that that’s what helped me switch over to DevOps after ramping up on the whole technical side of things. The wider view I have of systems that I’ve been working on has helped me a lot in understanding the processes and the importance of processes surrounding development and the development life cycle.

Liran Haimovitch: I think that’s super important in everything we do. Today, I think you’re the first guest in this podcast to come from a legal tech company, and I’m wondering as things change, as the world around us is changing, and every different field has its own peculiarities. What’s unique about legal tech, what’s unique about the way Lawgeex is doing technology as compared to what would’ve been done in a different field?

Julia Shub: I think that legal tech is not very different from other fields. Other fields have their sorts of QA and what Lawgeex basically does is it uses an ML process to tag legal concepts in documents, such as NDA contracts or SAAS contracts or all that. What we try to do at Lawgeex is to shorten the lawyer time, and as you know, lawyer time is expensive time, and everybody needs legal advice at some point or another, whether it’s reading a contract that you have received before you sign it or legal advice for different matters.

What Lawgeex does is automatically tag legal concepts in contracts in order for the lawyers themselves to be able to read the contracts quicker, go through them with lower latency, and basically allow them to step out of the scope of just a person reviewing a contract but to think it over. So it makes the lawyer’s job a lot easier and allows them to concentrate on what interests them more, and not just looking for who the entity in the contract is, or what they are trying to do or who the counter-entity is.

Liran Haimovitch: Now, since you’re deep into legal contracts and I have to wonder, did you get any unique regulatory profiles or anything like that that you ended up doing?

Julia Shub: Frankly, yes. Lawgeex is the first tech company that has received approval to practice law in the United States, specifically in the state of Utah in 2021. One of the reasons that we were allowed to practice law is that as a tech company, it’s funny, most of our employees are lawyers. We have a large QA department, a legal QA department that basically trains our models, checks our results, and allows everything that leaves Lawgeex to be at a better state than what it came in.

Liran Haimovitch: What other interesting technical challenges surprised you at Lawgeex?

Julia Shub: I think that what surprised me the most is that before I came into Lawgeex, there was no in-house DevOps. First off, we have a very strong engineering team that took the building and deployment processes into, as a part of their teams, which is not a bad thing. But someone had to handle the infrastructure, and before I came into Lawgeex, that used to be a contractor’s job. There was a specific contractor that used to work two days a week at Lawgeex to handle their infrastructure and infrastructure needs. And I think that it did quite a good job, but someone had to take over and ramp it up a little. Make it better, make it more comfortable. Updated it enough. Make it more secure.

At the time that I came to Lawgeex, there was also a switch in the R&D management. Our previous VP R&D left. Another person, basically the team leader of the engineers, became the VP of R&D, and it was a time of a lot of changes in Lawgeex in terms of how the engineering works. So we had a switch in management. We had a person coming in to do an in-house job, which was never done there before. And it came with challenges of communication, mostly, because the engineering teams were used to working the way they worked and then comes a new person, who is also new to this industry and comes and says, “Well, we have to ramp up our standards. What we’re doing is nice, but we can do better.” Which poses a different kind of challenge.

On the one hand, no matter what professional comes in at a new position and tries to explain to people that “Look, guys, we can’t keep on working like this. This is not scalable.” And especially if that person is new to the role of DevOps, it comes with challenges. Because your role is not – I think I had to prove myself to the developers that what I’m saying actually checks out. So it took a little while, but I did get them to trust me and trust my opinions, and all this could be done only by realizing what their pain points were and trying to address them as best as I could. If we’re talking about CI processes, they had one CI process, which was – well, I wouldn’t say it was very fast.

Liran Haimovitch: [chuckles]

Julia Shub: Basically what I had to do is make their CI processes faster, and since they were already working in microservices, I had to take that process and replicate it to all other services. And since that process is already replicated, why not join it all together and create one general CI process to handle the built-in deployment of all the services? If they’re all the same, why shouldn’t my process be the same? Why shouldn’t it be shared by all the other repos? Those were some of the challenges. It took a while for them to really trust me and to trust that what I’m bringing to the table is good for them, but we got there, and the more we communicated, the more we talked about their pain points, the more we addressed them, the more technical discussions we had, they trusted me more and trusted my opinion more.

But what I actually liked about this process is that they were challenging me to explain to them why we are doing something the way we’re doing it, and I think that is something that is great in engineering. Engineering departments become better the more we communicate and ask ourselves why are we doing things the way we’re doing them? Can we do them better? Should we do them in a different way, maybe? And maybe get new ideas.

Liran Haimovitch: I feel this is a story we keep hearing. It’s a very interesting story, it’s a very important story about a new leader, whether it’s in the infrastructure or platform group, DevOps group, or just a new tech leader as a whole, that comes in and wants to make things better, change the way things are being done. Especially some companies tend to fall behind, and then new leadership wants to invigorate things, get things moving again. And I have to wonder, how was it facing those challenges? How was it? How do you go about aligning the team, setting a higher goal, a higher standard, and getting everyone aligned on that?

Julia Shub: It’s an ongoing challenge. I have to say that I think this challenge is never ending, but it starts with coming into the team and trying to understand how they work and why they built the processes the way they built them. The answer can be: we built this because this is what we know and this is what we’re comfortable with. Which is great, but then you have to come to them and say, “Okay, I think we can do things in a different way. Let me make a POC and let’s try my process without eliminating your current process. Once you see my process works for you, or doesn’t work for you and then we can tweak it, once we all agree that this, a new process that we’re gonna work on together is better, then we can switch.” So I think that the key here is making them involved in this process and giving them the stakes of it. It’s not that I own the processes solely. We think on them together. We work as a team. Even though I’m not a part of any engineering team, because I work in parallel to them, every time I work with developers, my goal is for them to feel and for me to feel that we are all working for the same goal. And our goal is to make life easier and better for all of us.

Liran Haimovitch: As you mentioned, you’ve been in DevOps for just three years. For those three years, you’ve been at Lawgeex. So essentially, it’s your first DevOps role. what was it like hitting the ground running with a DevOps lead position and a lot of responsibility and big shoes to fill?

Julia Shub: It wasn’t easy. [laughs] Frankly, in most of my roles in high-tech companies, I’ve been thrown into deep waters, so to say, or at least that’s how we say it in Hebrew. I’ve had to manage. I usually didn’t have a leader to guide me. Specifically at Lawgeex, what the smart thing that they did, which I very much appreciated, is that they kept the same contractor that worked on their infrastructure for a while until we both felt comfortable that the knowledge transfer has more or less been completed. Or at least that we completed about 70 to 80% of knowledge transfer.

So the contractor kept working for two days a week along with me in my first five or six months. He taught me everything he could teach me about how he does things and how he sees things. He also was the one that hired me for the position, because he interviewed me. Let’s leave the fact that I think he’s a great guy and I appreciate him immensely. I think we did a great knowledge transfer. He did a good job to begin with, which made my job easier to take over whatever he wrote, and to improve on it, which I have.

I mean, when I came into Lawgeex, we used to manage our own Kubernetes clusters for instance. We used to manage them with kOps and frankly I despise kOps. ]laughs] It was good at the time before you had managed Kubernetes services like EKS or AKS or whatever, and since Lawgeex works solely on AWS, and EKS at the time was very expensive, we had to use kOps. We had to manage our own clusters. Once the prices started dropping, and there was a certain point in time where the price for EKS really, really dropped, I had made the decision together with my architect and VP R&D that it’s worthless to waste my time on managing a cluster where I can get a managed service, which is more mature. And it even cost less than keeping the Kubernetes masters we had at the time. Three C2 machines might cost more than an EKS cluster, which is highly available without having to actually do something about it. Then all you have to worry about is updating it to the proper versions that you want it to run on and you don’t have to worry about its availability. So some of the choices we had were either we want to work with an open-source project or a managed service.

And it’s not just Kubernetes. It’s other tools, as well. As a single person doing DevOps in a startup of about 100 people, out of which about 25 are engineers, you have to know where your time is going, and one of the most important decisions that you can make is decide what do you want to spend your time on? Do you want to spend your time on making sure that your Kubernetes cluster is up, that you don’t have any problems, that everything is highly available, that you’re doing a Kubernetes master rollouts as planned? Or do you want to spend your time improving the life of the engineers by working on their CI/CD processes and making them better?

Liran Haimovitch: How do you keep track of all those changes, of all those key answers those dilemmas, I mean I’m sure you’re not keeping track of every new feature release in EKS. You’re definitely not keeping up with every AWS service, at least, and yet somehow you knew that EKS was becoming more mature, that EKS was becoming cheaper. How do you keep track of that?

Julia Shub: First off, I try to sign up for the news of all the tools that I am managing. I sign up for every email that AWS can send me regarding updates on EKS and services that are important to me, that are key services. I can’t say that I read everything, but if I see something that looks important, I will try to read it, such as EKS News. Such as Jenkins Updates, or everything that CloudBees provide. I select a few key tools and I try to follow the news about them as much as possible. Also, I try to follow articles on sites such as Medium, and follow the Terraform registry and key persons who are developing for open-source projects.

For instance, if we were talking about Terraform, there’s a specific Ukrainian developer called Anton Babenko who wrote most of the community modules for Terraform. So following him on LinkedIn was a very good decision on my part. First off, he gives out a lot of information. He does a lot of workshops, I learn a lot from him. Second, it’s important to know when he stops maintaining certain projects, such as the EKS module. For those of you who don’t know, EKS module in the Terraform registry has breaking changes from version 17 to version 18. It is bad. Also a AWS provider has recently switched to versions 4.0.0 and also had a lot of breaking changes. Knowing when breaking changes are coming is a very important part of what we do, because otherwise, we find ourselves in what I like to call dependency hell.

For instance, I was supposed to give a talk tonight about pitfalls of Terraform and one of the worst pitfalls that I had was not locking versions for providers and modules. and I had to spend about two weeks of my time to figure out where my dependencies are, what versions am I running, where am I running them, what happened to my Terraform? Why is it not running, why is everything broken? So yeah, I try to keep up with the key parts of my infrastructure such as AWS updates, Terraform updates, Jenkins updates. I don’t get to everything and yeah, sometimes I get stuck with dependency hells and all that, but we try to do our best, so we try to read and we try to keep up. That’s all we can do.

Liran Haimovitch: That’s all we can do.

Julia Shub: Yeah, we can try.

Liran Haimovitch: You briefly touched upon Jenkins, and I have to wonder, earlier on in our conversation, when you mentioned building out new CI pipelines, was Jenkins the tool you used for those new pipelines?

Julia Shub: It was. I had a decision of whether we’re gonna stay with Jenkins as an open-source project or maybe we would switch to, I don’t know, other providers. Currently, we use both Jenkins and GitHub actions. Each tool has its own specific use case. And frankly, I think that the CI pipelines, it doesn’t matter what tool you’re using. What matters is how you build the process. What you want to integrate into the process. So whether it’s Jenkins Circle CI or whatever other CI provider you want to use, the most important part is how you build the process around it and not in what language or what tool you use specifically.

Liran Haimovitch: Couldn’t agree more. And yet, I know you mentioned dependency hell. Jenkins and its plugins, that’s literally the definition of dependency hell. And that is definitely a lot more to maintain. You’ve mentioned kOps. We all want to keep light on maintenance. Jenkins is not light maintenance. There are definitely easier approaches, whether it’s [inaudible 16:13] is it two instances of Kubernetes spots. It’s getting easier, but still managing Jenkins is a time-consuming task. I kind of have to wonder, what are the key benefits you see to keeping with Jenkins compared to a managed CI service?

Julia Shub: Frankly, I’ve had to realize where I’m gonna spend my money. As a small startup – and Lawgeex is not a unicorn company with an investment of $200 million. We’re a smaller company. We have to decide where our money goes. And since I’m familiar with Jenkins, its pipelines, I’m familiar with Groovy, it’s easier for me to update it rather than go and buy a managed service. I decided to spend my money on other managed services, such as monitoring services for production, which currently are more important to me.

Liran Haimovitch: Yeah, that’s definitely a good reason to figure out which tools to buy in order to rely on open-source. If it’s good enough for your needs.

Julia Shub: Yeah. Sometimes, we grow out of open-source tools. I think I haven’t grown out of Jenkins yet. And as to your question before about the plugin dependency hell, I try to use as little plugins as possible. So I don’t use a monstrous amount of plugins. I use the basic ones that I require. I sometimes try out new ones. Most of them don’t keep and then I keep to the Jenkins basics.

Liran Haimovitch: So the less dependencies you have, the easier it’s going to be in the long run. We’ve touched a bit about Kubernetes. We’ve touched a bit about Jenkins, and earlier on, you mentioned you’re about to give a talk about Terraform. And you’re actually a pretty big fan of Terraform.

Julia Shub: I am. I think that infrastructure as code is the future and as far as Terraform goes, currently, I think it’s one of the best tools around to do infrastructure as code. But that is because I dislike writing JSON files. Yes, CloudFormation, I’m looking at you. I feel very comfortable with HCL and the way it’s written. Yes, it has its own pitfalls and it has its own personality, let’s call it that. And there are things that I wish I had known three years ago when I started writing Terraform. But yeah, I think currently, it’s the best tool around for the job.

Liran Haimovitch: What would you like to see improved in Terraform?

Julia Shub: First off, I think I’d like to see myself improving the way I implement Terraform, whether it’s better usage of workspaces or maybe switching to Terraform Cloud, which looks very nice, but I haven’t implemented it into our production environment yet.

Liran Haimovitch: You’ve mentioned managed services that cost a lot of money.

Julia Shub: Terraform Cloud on the one hand can cost you a lot of money, but there are ways to keeping its costs down. I’m not talking about Terraform Enterprise. I’m talking about Terraform Cloud specifically, which does have a free tier. Which if you think about all the details of how many people are actually running Terraform, if you have – if I’m not mistaken, it’s five free users in Terraform Cloud. If your team is small and you do have a limited amount of people running Terraform configurations, it can work for you. And the price of adding an extra person specifically in Terraform Cloud and not Terraform Enterprise is not that high.

Liran Haimovitch: Good to know. So you’ve mentioned workspaces, you’ve mentioned Terraform Cloud. What else is there to improve?

Julia Shub: I think secret management can improve, but the thing is that since HashiCorp has another product that actually does that, I don’t think that secret management and Terraform will ever improve. So I think that secret management is not going to improve a lot in Terraform because there is the HashiCorp vault*, which is an excellent product as it is. It costs a lot of money, but I’m not sure that that part is going to improve. You have a lot of new tools that you can use for secret management and you really don’t have to use it in Terraform. You can use AWS’s managed secret services, you can use open-source tools, such as Ansible Vault or something like that. I used to use Ansible Vault. I don’t use it anymore because it was less comfortable than I would have liked. There are numerous startups that handle secret management at the moment, even Israeli startups. I haven’t checked their products out yet, so I don’t want to do product pitches. But I think that there’s a lot of improvement that can happen in that field and it’s happening as we speak.

Liran Haimovitch: What would you recommend any listeners who are interested in checking out Terraform, learning a bit more about it?

Julia Shub: I’d say check out the Terraform documentation. One of the reasons that I really love it is that the documentation is great. It’s really understandable, it’s really simple. It’s well-developed, even if we’re talking about the actual Terraform documentation or the documentation of community modules in the Terraform registry. The Terraform registry is huge. Most things you’re not gonna have to write on your own, because the community has already done that. It doesn’t excuse you from reading the documentation of the modules, which are mostly hosted in GitHub. And HCL, as a language, as a configuration language, is easy enough to read for you to be able to follow on what a developer did to create the Terraform module that you’re going to use. So they have great standards when it comes to documentation, which makes it easier to start with.

Liran Haimovitch: There’s one question I ask all of my guests and I would love to hear your take on it. What’s the single bug that you remember the most from your career?

Julia Shub: I actually have two. Both of them have to do with Kubernetes. One of them is mix of Kubernetes and Terraform. The first one was that when we used to manage our clusters in kOps, this was when I was about 80% done with moving our infrastructure to EKS. I was about 80% done with the project itself, but I had to deploy it, and what had happened is that when you run ACD PODS in Kubernetes for over a year, they go haywire and they disconnect your master from your cluster, from your actual nodes. And then you have master nodes that aren’t able to speak with nodes, which is downtime.

Liran Haimovitch: Yeah, sounds bad.

Julia Shub: Yeah, it was bad. It was six hours of downtime in which, thankfully, I had practically completed the whole migration process beforehand and I was lucky that it happened when it did. Otherwise, it would’ve been so bad.

Liran Haimovitch: So you migrated away from the old cluster to a new cluster during unplanned downtime?

Julia Shub: Yeah. It was scary and thankfully, the engineering team was there to – I’m not sure if we can call it hold my hand, massage my back or just support me and just look at what I’m doing and make sure that we’re doing everything right. Even though it’s funny, because most of the developers I worked with at the time were not very familiar with Terraform. They were not very familiar with the way our network was built. And it was good for me to have them looking over my shoulder, even though I can’t say that they could give me technical advice at that moment. But it’s good that we took it as a team and we were all together in it. That’s what caused the downtime to be about six hours. And also it was on a Sunday, which was really, really lucky.

Liran Haimovitch: Yeah, really lucky.

Julia Shub: The second bug is a mix of Terraform and Kubernetes. I work with VS Code, which has a great Kubernetes plugin. Great Kubernetes extensions that I use. I was trying to update the [laughs] the AWS-auth ConfigMap in one of my clusters. And what had happened is that in the Terraform context, I was using a specific environment with my specific exported access keys and in the Kubernetes extensions, I was in the context of one of my production clusters. So I hadn’t noticed that the IM roles for managing my cluster, for my nodes, basically, were switched from one role to another role that does not exist in my production cluster. Therefore, downtime. Thankfully, I figured it out in about five minutes and I brought everything back together.

But then I realized that – at least at the time, I’m not sure if it’s fixed or not – there’s a special role of the cluster creator in EKS, and I’m not sure if the system masters role can even fix broken-off configs the way that the cluster creator can. Or at least they couldn’t at the time. So yeah. Short downtime, but very important. What I learned from this is never keep your context in Kubernetes extensions in VS Code on your production clusters. Leave it on some dev cluster and don’t touch it.

Liran Haimovitch: Yeah, I would say the same goes for AWS CLI and everything else.

Julia Shub: Yeah, your default should always be some dev cluster, development credentials. I learned my lesson, but I learned my lesson later than I had expected, because this happened twice. But it hasn’t happened again.

Liran Haimovitch: I think we first met in the last Devopsdays Tel Aviv, where you actually moderated on the stage in front of hundreds of people. How did that happen?

Julia Shub: Wow, that was partially a fluke, what had happened with Devopsdays is when I first started getting interested in DevOps I was recommended to a boot camp called OpsSchool. And a friend of mine used to teach there, and he had recommended this program to me and it took a while for him to get me to actually sign up. OpsSchool is a nonprofit organization that wants to change lives of people within the tech industry who are in roles such as Sysadmins, QAs, the jobs that are considered less fun or not very appreciated in our community sometimes, and help them make a change to something that is better for them. And n addition to that, I’m trying to remember how exactly how the founder of the program put it. We want to stop seeing bearded Tel Avivian men as DevOps and we want to see more women and more people that you usually wouldn’t see in that position.

Because at the time, we all know that there are less women than men in the whole high-tech industry. Specifically in DevOps, those numbers are way, way smaller than in development. DevOps has been a male-dominated field for a long time. It started with the whole – well, before there was DevOps, there were Sysadmins. Sysadmins and IT people usually were guys because some of their job was actually to carry around equipment and this was a job that was let’s say considered in general less likely for a woman to do. I’m not saying that there were no women who did that job. There were plenty, they are awesome, they are great, and they were the pioneers. But still, there are a lot less women, so in OpsSchool, what they try to do is get more women to sign up and become DevOps engineers and demystify this whole thing that DevOps engineers have to be, I don’t know – well, Tel Avivian bearded males, sorry.

Liran Haimovitch: I’m right here.

[laughter]

Julia Shub: But you’re not a DevOps engineer at the moment.

Liran Haimovitch: Never been. Maybe will be someday.

Julia Shub: Maybe. But DevOps is a movement and not a position, so we could continue talking about that later. Anyway, after I went into the DevOps bootcamp, which was half a year in which, well, I didn’t have a life. It was worth it because two and a half months after I had finished my final project in OpsSchool, I started working as a DevOps engineer, as a DevOps lead in a startup company. So I think they did quite a good job.

One of the better things that happened during this program, and this is something that we try to do with every new person that comes into the program, is mentoring. Which means that at any time, students have mentors who help them out, who advise them. The whole staff of OpsSchool is always very open to students talking to them about everything, which is how I met a few very lovely key people in this industry, such as Avishai Ish-Shalom, who basically convinced me to try to help him organize a community conference called StatsCraft, which is also happening this May. So I started volunteering with StatsCraft. I started volunteering in OpcSchool. Today I am OpsSchool’s general manager and one of the instructors. so after I started volunteering in StatsCraft, I was also introduced to people who manage Devopsdays. And during Covid, because there were no in-person conferences, everything got mishmashed and StatsCraft and a few other conferences such as DevRel IL and DevSecCon, we created what was in 2020 the TLV community summit. So I got to know a few new people and to talk to them.

After giving a talk in Devopsdays 2020, which is also funny because I was nine months pregnant talking about how pregnancy is nature’s CI/CD. I got to know the people that manage it better, and after getting to know Sharone Zitzman a little better, she decided that she wants to pass on being emcee of Devopsdays to me. Why? Because she’s had enough, she wants to take a different role. And my English is good enough and I feel comfortable enough talking to people, which is evidently why we’re here. So that’s how that happened. Quite the long story, but it does get to a point.

Liran Haimovitch: Another CFP is opening up.

Julia Shub: Soon. Yes, it was gonna open up soon. Can’t wait for it. I think it’s gonna be somewhere after Passover.

Liran Haimovitch: So are we gonna see you this year as well in Devopsdays?

Julia Shub: You are going to see me at Devopsdays. I’m not sure if I’m gonna be emceeing or maybe I’ll try to give another talk. I’ll probably stick to emceeing this year. Maybe someday I’ll write something that is more CFP-worthy for Devopsdays.

Liran Haimovitch: We are rooting for you either way.

Julia Shub: Thank you/1

Liran Haimovitch: So if anybody wants to check out Devopsdays Tel Aviv?

Julia Shub: And OpsSchool at opsschool.org.il, because it’s awesome, and if you want to volunteer or if you want to send us new students, we’re always happy to know those.

Liran Haimovitch: It was a pleasure having you on the show.

Julia Shub: It was a pleasure being here. Thank you having me.

Liran Haimovitch: So that’s a wrap on another episode of The Production-first Mindset. Please remember to like, subscribe and share this podcast. Let us know what you think of the show and reach out to me on LinkedIn or Twitter @Productionfirst. Thanks again for joining us.