Garrett Dimon: Today we’re here with Pat Allan, who is the maintainer of Thinking Sphinx, and the purveyor of Flying Sphinx, which is an add-on at Heroku for Search. He’s currently serving as the CTO and co-founder of a small…I wouldn’t call it a start-up, I guess. What’d you call it? A social…?
Pat Allan: A social enterprise.
Garrett: A social enterprise called Limbr. Pat, can you go ahead and talk a little bit about yourself? Give a real quick history—your background, your location, history, that kind of stuff and let us know how you got here.
Pat: Sure. Primarily, I’m a Ruby developer. Thinking Sphinx, as you mentioned, that’s a Ruby gem. I’ve been a professional Ruby developer for…it’ll be 10 years at the end of this year. We’ve got another month.
Prior to that I was a .NET developer, but there wasn't—and maybe there is now, I’m not sure—but there wasn’t a strong open-source ethos to the .NET development community. I picked up on Rails. A good friend introduced me to Rails. I just went, “Wow! I really like this.” This was Rails, zero point whatever, so early, early days.
I tried to do some of the same stuff in .NET. It wasn’t possible. The meta programming stuff wasn’t there. I think it is now. Again, I haven’t touched .NET since those days. I’m sure it’s changed a hell of a lot.
I got into Ruby and, about six months into that, I was working on a contract and I needed Search. Sphinx, at the time, was the new cool search tech. I played with a couple of the existing libraries, and they just didn’t really gel for me.
They were either extremely simple, or they didn’t integrate well into Rails, or they did, but not in a way that I liked. I am opinionated, I guess as many Ruby developers are. I set aside a weekend that was quite clearly my time, as opposed to contracting time, and built the first version of Thinking Sphinx.
Garrett: At the time, did you have any idea that it would maybe turn into something more than just a side project that would help out your consulting work, or is it just something you wanted to do?
Garrett: I can say you did a great job, because Sifter used Thinking Sphinx forever and you, obviously, really, really did an amazing job from a maintenance standpoint, support, all that stuff. It was certainly one of the more positive experiences I’ve ever had working with such a fairly complex technology.
Pat: Thank you. I’m certainly far more proud of the recent versions. That first version, from that weekend, I think I rewrote within a few months because I quite clearly didn’t understand Sphinx. I went, suddenly…three months in, “Oh, that’s different to this and I need to do…Ah, right. OK.” [laughs] it’s come a long way. It had its ninth birthday back in August.
The fact that so many people, particularly in the first five years of its life, jumped on board and went, “Yes. This is the thing we want to use.” The number of contributors I’ve had is astounding. I’m really, really happy with how it’s gone.
Garrett: How long was it just a library before you decided to, not necessarily launch Flying Sphinx, but before you decided, “Hey, this is something that could work”?
Pat: It was probably six years ago, maybe a bit under that, where I knew that I really liked using Heroku for hosting. But Heroku is a very carefully managed system, and so Sphinx wasn’t an option. Maybe it’s my stubborn side, but instead of using another tech that was available, I went, “No, maybe I can make it work somehow.”
Initially, it was very much for my purpose, for my own apps. Some of them personal apps, some of them customer apps. Then went a bit down that path. I’m like, “Oh, well, I have had other people ask if they can use Sphinx on Heroku, and my answer has been, ‘No, sorry.’ It’d be nice to change that and maybe they can pay for it.”
Then I need to figure out how much I’m charging and how I do the infrastructure. That took several months, from just figuring out the tech, rewriting the tech because trying to work with Heroku’s databases at the time was quite tricky. A year in, it became far more easier and I wish that I didn’t have to deal with the old stuff.
Garrett: [laughs] That’s the case with everything, though. Right? Like, “If only I was launching Sifter today…” [laughs]
Pat: Yeah, and then just got some people onto it and opened up the beta and away it went.
Garrett: You were spending about three years on Thinking Sphinx before the idea of Flying Sphinx came along. Even when you got the idea for Flying Sphinx, it wasn’t, “I want to create a business.” It was, “I want to use Sphinx on Heroku and I can’t.” Then people were asking, “Hey, we would like to do that.” it’s pulling you along rather than you forcing an idea into a business. Is that fairly…?
Garrett: I love those stories because, more often than not, those are the businesses that end up getting going quick. Whereas, so frequently, one of the questions people ask is, “How do I find customers?”
It feels like the best way to do it is create something, help people, whatever it is, whether it’s open-source a Gem, or it could be Google Sheets that just happens to really handle all your knowledge, and do it manually, let people ask you and then, once you stop being able to serve those people, find a way to serve them and then make enough money to support it, sustain it, and keep going.
It feels like it’s the best way all around to do that. As a Heroku add-on, it seems like you probably have some different challenges in traditional SaaS. You don’t have some challenges that other people may have, like billing may be easier, but probably it still carries a lot of the uptime requirements and all that kind of stuff.
How would you compare it to just a traditional app? How would you compare the two? What are the pros and cons? Would you suggest somebody else create Heroku add-on or add-on, or an add-on on a platform instead of a creating fully dedicated app? Or, is it the best and worst of both worlds?
Pat: One of the main advantages, as you mentioned, is the billing. I don’t have to deal with billing. I occasionally have to deal with Salesforce, who pay me. Heroku take their 30 percent, or Salesforce, however you want to draw it. Certainly, Salesforce are the ones who manage the payments.
Generally, that goes all smoothly, but every couple years it feels like they go and change their systems and I have to deal with not getting paid for a while longer than I would like, but generally it’s pretty smooth. I don’t have to deal with refunds.
I don’t have to deal with people’s credit cards not working, invoices, none of that, so it’s like 30 percent. That is a bit, but there’s a whole lot of pain there that I don’t have to deal with. Now…
Garrett: What about support? Does that come into, too?
Pat: Yeah. Heroku are very good at integrating their add-on customers into their support system. They will assign tickets to you, to me. They’re, “Oh, this sounds like a Sphinx thing. Hey, Pat, your problem.” I’m sure they do the same with other add-on providers as well.
I have my own support system as well, but customers come through one or two of those channels generally. Support is normal. Uptime is normal. That’s been a bit of shift from just building apps to going, “Oh no, I have paying customers around the world in all time zones. Right. OK.”
As a one person business there’s only so much you can do, but it’s a constant learning experience. It’s like, “OK.” For the last two years, I’ve had a cheap Nokia with a dedicated SIM that sits by my bed. It comes with me in my travels and it’s always on. That way, my main phone with all the different alerts it’ll have—personal and work related—can be off.
But, if a server goes down, my Nokia makes a hell of a racket and wakes me up.
Garrett: That’s a really good idea. I think I thought about that in the past because I hated it having to leave my phone on at night, when the only alerts I cared about were really, really rare. That’s a really good idea, just to keep a cheap phone that just makes noise.
Anything works with SMS and phone these days, with Pingdom or any of that, so that’s a really…I probably should have thought about that a lot more.
Pat: It varies. I guess you’ve been building more of an app rather than infrastructure.
Garrett: Yeah, yeah, and that's…
Pat: Your customers’ apps don’t go down when your service goes down.
Garrett: Yeah, so infrastructure is a whole other level of stress for sure now that I’m at Wildbit and helping out with Postmark and that sort of thing. When something goes wrong, it really, really ripples quick. Whereas, if Sifter was down, it was an inconvenience and some people cared, but it wasn’t like if their whole product was offline because of…
Garrett: Yeah, for sure. Marketing. Do you feel like being an add-on in Heroku really helps generate customers, or do you feel like customers already have…? They want to use Sphinx and so they’re going to use it. Does that help or is it kind of a wash?
Pat: I have never done any sustained marketing. I’ve done indirect in the way that I…There’s a talk I gave at a conference or two around how I built it and the infrastructure challenges of mine in there and all that. That probably helped make some people aware, but just generally going, when people go, “Hey, I’m using Thinking Sphinx and Sphinx, and I want to use Heroku. Can I do that?” I can now go, “Yes, you can.”
Garrett: A lot of it, then, is existing awareness around it and not necessarily Heroku bringing a whole slew of new potential customers. There’s probably some.
Pat: No. Yeah, probably handful, but of the customers I have, and I don’t think that number is growing any more. From what I can see it’s probably slowly shrinking because Sphinx isn’t new, and cool, and hot anymore, and that’s just tech. I would guess that all but maybe one or two, if any, are using Ruby and Thinking Sphinx.
Garrett: I think my point there, too, is can imagine a lot of people being like, “Oh, if I create an add-on then I don’t have to worry about marketing because it’ll just be there and people will use it.” I imagine you wouldn’t have counted on that as being your primary channel for finding customers and growing it.
Pat: Not entirely. I knew that there were a lot of people out there who wanted to use Sphinx. Heroku, particularly when I first built it, it was the platform to get behind, especially if you don’t want to have to deal with your own infrastructure cost yourself, or not directly.
Yeah, I knew that people were going to come and use it. I had no idea how many. It’s just nice to have that growth, but I wouldn’t just build an add-on and go, “Hey. Cool. Here we go. Done.” to do something new.
Garrett: Looking over my questions. What’s your favorite part of building software, especially hosted software? What is it that is exciting about it to you? It doesn’t have to be the technique. It could be the technology, the people, the skill. For me, with Sifter, the coolest thing was that you can build something, and it can reach so many people.
it’s, effectively, not infinite scale but relative to any other business, where you set up a shop in a small town. It can only reach the people in that town. But, with software online, you can reach and help thousands of people with marginally more effort.
For instance, that was, for me, what just always made it exciting, because there’s so many people out there to reach. What is it for you that’s been enjoyable or really relative to other types of work you’ve done?
Pat: From an enjoyable perspective I think the ability to build—and I’m not saying I’ve hit these targets—but build elegant tools. That’s my joy in writing a whole lot of Ruby gems and then, if Flying Sphinx as a service, trying to make the barrier of entry as low as possible and making it a smooth experience for people. There’s very definitely room for improvement there, but…
Garrett: Nothing’s ever done. Right?
Pat: No, for sure. Every now and then I’m like, “OK, maybe I should rewrite Thinking Sphinx or refactor this and do that.” Certainly, with Flying Sphinx, as I mentioned, it’s not really growing anymore. Putting more time into the business probably isn’t wise, but every now and then it’s like, “No, I need to fix that bit, I need to change that bit,” so I still jump in and do things, for sure.
Garrett: I feel like it was always a blessing and a curse. It was a blessing that there’s always something to work on. As a curse there was something to be working on. [laughs] You could never be like, “I’m done. I can relax,” because as soon as you ship something, even that thing’s going to have maintenance and support.
Then there’s the next thing on your list. I feel like software’s a very difficult business to say, “I’ve done all the things. What’s next?” it’s just constantly just trying to stay afloat. Having done a lot of support for open-source and very excellent support, I might add, how would you say that compares to supporting a paid product? Is it better, easier, worse, about the same?
For a little more context, a lot of people I know with SaaS are worried about, “If I launch an app on the side, how am I going to find time to answer all the emails, and do all the support, and manage all of that?”
I know I was worried about it when I originally got started, and I hear a lot of people kind of…that’s their fears. Like, “Well, I’ve got a full time job. How am I going to manage that?”
Pat: Yeah. There are strong overlaps, I guess, between the open-source support and the SaaS support. I will sometimes drop things for SaaS support. It depends what the customer needs. If they’re just like, “Oh, things are a bit slow,” then it might not be an urgent issue I might have a quick look and go, “OK, I’m going to have to come back to you.”
Sometimes there’s time zones involved and, generally, customers are extremely understanding about that, which is great. But, for the most part, it’s like people are irritated that something’s not working, and you want to help them. I think it’s more something about open-source knows about SaaS is that sometimes people get really annoyed, really frustrated, really angry.
I feel that’s maybe a little more warranted with SaaS because they are paying for the service. Open-source, they’re not, and it’s kind of, “Well, is there some entitlement issues there which…?” That’s a whole another talk.
Garrett: It is. [laughs]
Pat: I’ve actually presented it at a couple of conferences covering that and a bunch of other stuff around open-source lately, so I’ve been thinking about that a bit.
For people who are going into it and thinking, “Oh, this might take up a lot of my time,” if you have a decent amount of confidence around how you build things and you feel they are generally pretty stable, then, if you’ve got a lot of people contacting you, that either means that, one, you misjudged how stable it is, which is OK because you can fix that hopefully pretty easily.
Or, two, you got a lot of customers coming in because you wouldn’t expect to have contact with most of your customers. Mostly, you’d expect the documentation and the service itself to help them step through what they need to do. If you’ve got a lot of people jumping in and hassling you, that probably means you’re quite successful, which is a nice problem to have.
Now, whether you want that success is up to you. You might be like, “Well, I actually like my main job, and this was just something fun.” Every person’s going to have to deal with that their own way, I think.
Garrett: Did you anticipate Flying Sphinx being a significant thing, or was your hope always, “Eh, it’ll be cool if this helps pay rent”? Did you have high hopes for it, or just, “I’m doing this anyway, so cool if I can make money off of it and it could pay some bills as well, and help me reinvest time into the whole project”?
Pat: It has varied over time. Certainly, when I started, I think my…I have one friend in particular, Steve Hopkins, who currently works for a company called Culture Amp in San Francisco. He is from Melbourne, where I am. He lived in Sydney for a few years.
I was up there visiting for a conference and staying with him, and had told him about, “Oh, I wouldn’t mind maybe building this thing, Heroku thing,” blah-blah-blah. He’s not a developer, but he’s not only an excellent friend, he’s a smart business person and culture and people person. He’s like, “OK, so what are you doing tomorrow?”
I’m like, “Oh, the conference is over. I might just chill, wander around Sydney, or whatever.” He said, “No, no, no. You just told me what you’re doing tomorrow. You’re sitting down and you’re working on this thing, and I will buy you all the hot chocolates you need,"—because I don’t drink coffee—"and you will start on it.” I’m like, “OK, right.”
It was so great to have…I think Steve spotted the potential of it more than I did at that point. Over the years, even just within that first year, I guess, it was like, “OK, is this something that’s useful to me?” Then you look at the cost and you’re like, “Well, I hope it’s useful for at least a handful of other people.”
Then it starts building and you’re like, “Oh, no. OK, I can make a little bit of money off it.” I still make a bit of money off it. Not enough to live off, but it pays for itself and then some, which is nice.
Garrett: it’s funny because it feels like so many businesses…there two things that that made me think of. It’s there’s so many businesses are accidental business. It was just, “Oh. Hey, people like this, and there’s something there.” It just snowballs until it becomes something.
Then there’s the things where people have an idea and it could very well be a viable business, but it just stays in that idea phase because there’s no tipping point. There’s no spark that makes them go, “You know what? I’m just going to sit down and do it.”
I think a lot of that is fear that, “I have to commit to this huge, big undertaking and set aside six months of my life, or I’ll never be able to pull it off.” The reality is there’s always something you can build and start with and then, just step-by-step, make it a little more advanced or whatever. It’s just awesome to see that story play out over and over again for so many people. It’s funny, to me, how frequently it unfolds.
Pat: Yeah. I have another infrastructure as a service kind of thing called Drumknott, named after a Terry Pratchett character. It is hosted Search, of course, for Jekyll sites because they’re static pages. It’s like, “Well, I want to put my blog onto Jekyll, but I feel like, given so much of what I’ve done over the past 10 years is Search, I should probably have Search on my site, so how do I do this?”
I ended up building this service. Initially, I just built it for me. Then, early this year, I actually had the time to actually put in all the billing code and the invoicing code. None of that was big…I’d done that for contracting projects before as well, so I had some familiarity there. I have one customer besides myself.
Again, I haven’t put any time into marketing and it is probably even more niche than Sphinx itself, but for myself and I have one person paying three dollars a month that slightly offsets my own costs. Fine.
Garrett: Yeah, and you’re not going to get a whole bunch of support in the middle of the night on that.
Pat: No. Right. It also means that I’ve done the hard yards now for my own project, the whole billing and invoicing side of things. I actually have open-sourced the code for that. Not even just I can take it and use it somewhere else, but somebody else can do so as well.
Garrett: Right on.
Pat: Whether the code’s any good, I don’t know.
Garrett: I’m sure it is.
Pat: it’s tested.
Garrett: I think this, probably, is going to vary based on the type of project, but if somebody else had an open-source project they were passionate about, but maybe they felt like they wanted to dedicate more time to it, obviously, it’s a catch-22 because everybody still has to make a living one way or another.
A lot of people, I think, they might be passionate enough about their project to say, “How can I go ahead and evolve this into some sort of business?” Search, that dovetails nicely with Heroku.
Are there other things, or other just tips you’ve thought of, or encountered, or learned that you would have for other people who are in a similar situation that want to take their open-source project and turn it into something more core that they can spend time on and make a living off of?
Pat: it’s a tricky one because, as you say, it’s going to be different with every type of project. You look at Mike Perham, who’s built Sidekiq up and made a nice bit of money through it, which is fantastic because it’s a great project and I use it all the time. Although I, personally, haven’t had something where I’m paying for it. (Editor’s note: I also interviewed Mike Perham about Sidekiq and his business recently as well.)
it’s all very well for me to spruik these things, but I’m not doing everything in my own power to support them, either. It feels like, generally, I wouldn’t set your sights too high because some projects…I’ve written tons and tons of open-source code, and I’ve had one big hit. I am extremely lucky to have had that.
It just happened to be right place, right time really. Not everyone’s going to have even one big hit. Looking at the open-source projects and your open-source code and going, “That. I want to make that my…” Even some income on the side, that’s a big ask. If you’ve got something that is related to infrastructure, then I feel that that’s an easier step to make.
If you’ve got something that’s just popular in some way, then, again, that’s easy to make the step. But, I think some of it’s going to come down to more having the support of your employer or your clients.
I was working with a development team for a few years here in Melbourne, called Inspire9 Development. Generally, when we built things for our clients, we’d see patterns that we’d been doing in different projects and stuff like that, but it’s not core to the product, so we can open-source it.
We just open-sourced a hell of a lot of stuff. Most of it I don’t think anybody else is using because we didn’t really promote it. The team that I worked with were very supportive of going, “Is that generic? Sure, make it open-source. If people have questions about it, sure, spend a bit of time helping them.”
Garrett: Default to open, and then, if there’s a reason to keep it private, keep it private. Otherwise, put it out there.
Pat: Yeah, for sure. Just having an employer who’s like, “Look, you want to spend a bit of time on open-source. We know that feeds your passion. We know that helps our code. Go for it.” That’s a big ask for employers, don’t get me wrong.
Garrett: Not the right ones, but yeah.
Pat: Well, yeah.
Garrett: It is tough, though. There’s a lot of legal implications, and…
Pat: For smaller businesses, right. Not everyone’s a Heroku or a GitHub with tons of cash to go, “Sure. we’ll hire people just to do open-source.” I think I’ve lost my train of thought a little bit there, but I think that…
Garrett: No, there’s a lot to it. I think probably one of the bigger things is—it’s the same for any business—make sure that you have business customers. If it’s an infrastructure project that you’re working on, there’s a chance that business could leverage that. Then there’s value in it.
There’s surely a way you can, like you said, with Mike—who I’m supposed to talk to, I think, later in the week, actually—about how can you tier it such that there’s still value in the open-source, but then find a way for the enterprises that are really getting a lot of value out the project to pay for it and contribute back into it.
In a way, it turns into this virtuous cycle and it allows you to improve the project, and everybody wins. Just like anything else, there still has to be some business viability there so it can sustain itself.
Garrett: For sure. We’re coming up close to the end. I think the last question is, if you could do it all over again…
Garrett: This is kind of two-fold, and you can answer it either way. What was the most painful thing that you’ve dealt with, and what would you do differently to fix that pain or avoid it entirely?
Pat: I mentioned the database issues earlier. That may have been the biggest pain point in a way that it’s far more visible, I guess. There are probably things with my choices of how I built Flying Sphinx over the years that just little bits of pain here and there, but that one was a bit more sustained. That was, the original Postgres databases on Heroku, you couldn’t access externally.
Sphinx, at least in the feature-set that it had available then, had to talk to your database, had to make SQL queries direct to either MySQL or Postgres database. What I ended up having to do…Initially, I looked into using RDS and it was only MySQL at that point in time.
I switched a couple of projects over while I was building out the infrastructure and I went, “Aye, yep, yep. Oh, wait. I just got a bill for Amazon and I can’t ask my customers,” my potential customers at that point, “to spend $80 a month just to have a database when they get it for free with Heroku.”
What I was doing instead was doing a reverse SSH proxy back to my servers. Then my servers could connect from your app. Then my servers could connect through your app to your database, and it worked. There were tweaks I made one way to make it a little bit faster, but it wasn’t elegant by any means, obviously, nor ideal, but it worked.
Then, of course, about a year in, Heroku like, “Oh, no. We’re doing a new version of our databases and it allows external access.” it’s just like that. [snaps fingers]
Pat: That would’ve been really nice a year ago. It would’ve removed so many head…I brainstormed in this coworking space with my peers around like, “Right, this is a problem.” One of them’s a network engineer and he’s like, “Well, you could try this you could try that.” All of the smart people are putting in thoughts. All of that was amazing…this and this and put it all together and, “Aaah.” Don’t worry, I was just…the frustration of having to talk about mistakes that…
Garrett: In a way I think that’s a good anecdote, because there are so many things that, when you’re building something, it’s so easy to hit it and just be like, “This is ridiculous. How am I ever going to get around this?” it’s so easy to let it just really, really drag you down when there’s just too many thing you have to fight through like that to make anything work.
If it’s not that it’s something else, whether it’s credit card fraud or whatever it is. There’s always something that’s going to really, really suck the life out of you and really take the wind out of the sails on a project. I think too often, no matter how much I know that, when that happens to me in a project, it’s just like, “All right, I’ve got to take a day off. I’m defeated. I give up.”
We all deal with that. Right? That’s just the nature of software. It’s nice for everybody to know that it’s not always smooth sailing, I guess, is where I’m going with that for sure. That’s really the gist of it.
Is there anything else you want to share about your experiences or any random anecdotes that people have heard you tell before and they’re like, “Oh my Gosh, that’s so fascinating”? Anything like that you want to…?
Pat: Nothing comes to mind at this point, no.
Garrett: All right, that’s fair. Well thanks so much for doing this. I really appreciate it. I think there’s a lot of good anecdotes in here and insight for people.
Garrett: I really appreciate it.
Pat: No problem. Happy to have the chat.