Planning for the unplanned

There’s an expression in Hebrew: “Baltam”. It’s a shorthand form for something unplanned, or more precisely, it strongly implies: [something that is] impossible to plan. I think it has its roots in the military. In the battle field, you always have to account for some surprises. You cannot possibly have everything planned. Israelis are also (in)famous for improvising. Not so famous for planning ahead.

As an (ex?) Israeli, I recently felt awkward, essentially being accused of being overly bureaucratic. And by a German colleague, of all people. Can you imagine it?? :)

Some things take you by surprise

Ok, and just to clarify one thing, this post isn’t about cultural stereotypes, but rather trying to figure out a practical approach to a real problem that my team is facing with new ideas and features:

How do you deal with new tasks or ideas, especially small ones?

rails Security Technology

simple and secure cron using AWS Lambda

Many apps require some tasks to execute on schedule: cleaning up inactive user accounts, generating daily, weekly or monthly reports, sending out reminders via email, etc.

cron is a simple and trusted scheduler for unix, and used on pretty much any unix-based system I come across.

So cron seems like a natural candidate for triggering those job executions. But it’s not always the best solution.

In our case, we’ve used the whenever gem for rails successfully for a long while. The gem acts as a cron DSL and lets you inject and manage cron entries from your rails app.

The problem starts however when you start growing, and your app spans more than one server. Or even if you only use one server, but want to be able to fail-over, or switch from one server to another.

Why? suddenly you have more than one cron launcher, and jobs that should execute once end up executing once on each server. This can cause some weird and unexpected lockouts, duplication and other issues.

So what’s the alternative?

marketing optimization

why I stopped using Intercom

This post has been on the back of my head for a couple of years now. I think we actually switched-off Intercom in 2016 or so… But the reasons should still stand now, or might even be stronger. Of course, things might have shifted, so please forgive me if some features are totally different by now.

For those who don’t know (now, well, I think you probably do know it, but maybe not by name. It’s the technology (or company) that adds those little “bubbles” on websites, with friendly faces offering to help.

How intercom works (taken from

Of course, isn’t the only one now, and there are a few competitors in this space. The principle is pretty similar though. I think intercom was the most successful company doing this, or the first, or both. But it’s not really important. It’s mostly about intercom as a concept, rather than a specific implementation.


The short, simple, and most crucial reason: it didn’t work. How do I know? We A/B tested it. Over a fairly long time and a large number of people.

food marketing Technology

marketing lessons from the street market

When you walk inside the Ben Tanh market in Ho Chi Minh City, Vietnam, you’ll eventually end up inside the food area. There are probably hundreds of stalls selling local food. Lots of delicious Banh Mi sandwiches, noodle soups, fruit juices and summer rolls.

One thing that you can’t ignore however, is that as soon as you walk around, you’ll get approached by one of the stall owners. They’ll simply hand you the menu to choose from.


take your pick

The book piqued my curiosity, so I picked it up and took a peek at the first page. It was written by a artist at the peak of her career.

As a non-native speaker, I guess when I pronounce any of those words: pick, peek, peak, or pique, they all sound the same. So it’s even harder for me to clearly memorise. I mostly get it right, but can occasionally confuse some forms.

Especially peek and peak.

It doesn’t happen with meet and meat, feet and feat, leek and leak though. I wonder why.

Technology work

Innovation, Promises, Lies and Toupées

I recently finished reading “Bad Blood – Secrets and Lies in a Silicon Valley Startup”, by John Carreyrou. It’s a remarkable piece of investigative journalism and an amazingly grabbing read. I just couldn’t let it off my hands.

I think it particularly stood out, because the amazingly stark contrast with another book I just recently wrote about: “It doesn’t have to be crazy at work”, by the co-founders of Basecamp.

food health

A Guava a day

This isn’t exactly a standard type of post for this blog, but then perhaps I shouldn’t be too strict with myself as far as things I write about. After all, it’s my personal blog. I make (and break) the rules. And anyway, nobody reads it. If you are reading this, consider yourself one of a very select few.

I’m no health specialist, and this is just a sample of one, and much less scientific than my A/B testing for coffee (which wasn’t scientific at all), but I’m totally crazy about Guava, and what I perceive to be its health benefits for me.

Growing up in Israel, guavas were kinda pungent, slightly mushy, yellowish fruit. It was also one of those things the local Israeli folklore qualified as “either you love it or you hate it” (we have no Marmite in Israel, not to my knowledge anyway. Or maybe there’s a strong consensus and everyone hates it? anyway, I digress).

I guess I was in the “love it” camp, but I don’t recall being particularly crazy about it either. I think the local wisdom was also that it causes constipation, so I guess I tried not to have too much of it.

I no longer live in Israel. But I also don’t come across Guavas. At all.

I lived in London for a number of years, and I don’t recall eating any there, or even seeing them. Maybe pink, artificial guava juice. And I now live in Berlin for several years, and I can’t think of seeing any here either. How come??

I do see them everywhere in Thailand and Vietnam though. It’s literally around every street corner. Every fruit stand would typically have them besides Papayas, Pineapples and Watermelon. You can also have a proper, fresh, guava juice or shake in lots of places.

White Guava
rails ruby Security Uncategorized

invisible reCAPTCHA v3 with Rails and Devise

We’re recently being hit with more and more bots.

Some of them are crawling our site and hitting valid or invalid endpoints. We’ve seen plenty of credential stuffing attacks as well. Most of them distributed across different IPs, with each IP hitting us at low frequency.

And most recently, someone abused our registration form to spam their recipients via our system.

It was quite clever actually. When you register, you enter your name, email and password. We then send a confirmation email saying something like

“Hey Roberta, thanks for joining. Please click here to confirm your account”.

Now those guys used their victim’s email address, and used the name field to link to a URL. So those users would get an email

“Hey lottery tickets, thanks for joining. Please click here to confirm your account”.

Slimey. Naturally our own email system took the hit of sending spam. Double ouch.

Luckily, we had some anomaly detection in place, and we blocked those guys quickly. They used some browser automation from a fixed set of IPs, so it was easy to block. At least until the next wave…

I’ve been dealing with those types of scenarios with fail2ban, and it’s really quite effective. We define regular expressions to inspect our log files matching certain patterns, and then ban if we see repeated offensive behaviour. fail2ban is limited though in some aspects.

First of all, those rules are a bit of a pain to create and maintain, and you need to make sure the offending IP appears on the application log record you want to capture. In some cases it’s easy, but not always. The bigger problem however is that fail2ban doesn’t scale. The more servers you have — let’s say in a load-balanced setup — the less accurate fail2ban becomes. Or you need to aggregate all your logs on a single fail2ban host, creating a single point of failure or a bottleneck…

So I was searching for a better solution. Sadly there aren’t many. Cloudflare, which we also use, offers some degree of protection. But it’s not as flexible. And of course there’s reCAPTCHA. You know, those annoying things asking you to pick traffic signs, or even just click “I’m not a robot”?

Now I was initially hesitating to use it. I’m not sure why, but the fact that it doesn’t really have any real competition bothers me. Plus, as a user, I’m frequently annoyed by those challenges, and I hate this experience.

Luckily, the latest version of reCAPTCHA (v3) doesn’t present any user-facing challenges. It’s completely invisible. The no-competition problem is not something I can solve. I discovered that even Cloudflare itself uses reCAPTCHA in some cases! And these guys have their own Javascript challenge and what not… So I decided to bite the bullet, and give it a shot.

Setting it up is surprisingly simple, and from my limited experience, quite effective. That is, the scores it produced were surprisingly accurate. Albeit my ability to test different scenarios was limited.

I’ll try to give some pointers for implementing reCAPTCHA v3 with Rails 5.1 and Devise 4. The implementation can work on any form or controller however, and not just with Devise.


Is it zen at work?

I really enjoyed reading It Doesn’t Have to Be Crazy at Work recently. It’s another bestseller from Basecamp. After reading Rework before, a lot of things felt a bit familiar. Too familiar, perhaps. But their new book still has a few new ideas and covers things from a different angle. Well worth a read.

Working remotely, and at a company with very similar culture and values to Basecamp, a lot of what they write about resonated. Much of the way we structure things at work was inspired or wholesale copied from Basecamp to be completely honest. Why reinvent the wheel when someone hands you an instruction manual for building a perfect one?

But some things caught me by surprise. It felt a little too zen, or even contradictory in some cases? But it definitely gave me pause. Maybe we’re doing some things wrong, and can improve even further? I’m still unsure, but hope we can experiment with some ideas. Let me jump into a few examples…


SmugMug video data loss

I’ve written only recently about SmugMug, and expressed my frustration as a developer who built an open-source tool for their platform. This has led me to try to get my data out of SmugMug as I was considering moving away from it as well… Only to discover that some of my video data is lost and/or not being made available. This applies only for videos. Both the quality is potentially degraded, and the metadata that is available on SmugMug cannot be downloaded or exported out of their platform.

If you upload a video to SmugMug, they don’t actually store the original video for you. Here’s a quote from their official page:


We don’t keep a copy of the original video you upload. We make high-quality display copies, which are probably altered from what you send us.

I’m not sure what this high-quality display copy means in actual terms, but I won’t be surprised if some quality is lost in the process. For a company that prides itself caring for photographers, where quality and reliability is key, I find it rather vague and disconcerting.

Furthermore, what isn’t mentioned on this page is that if you want to download your videos again, those videos would be stripped-out of the original metadata as well. This metadata includes information about the Camera you used, the date/time of the video, location information etc. All of this data is still stored on SmugMug, but you can’t get it back when you download it. It’s locked-in. For me, personally, this is even worse than losing video quality. My video memories are very tightly linked to the time and location of those original scenes. Without this info, the videos are next to useless. I just can’t find them (without going manually through hundreds or thousands of dateless and location-less videos, that is)