It's been a big week for us: we're now homeowners. The last few days have revolved around cardboard boxes, being thoroughly terrified of the wobbly ladder to our attic, and trying to work out what to do first in an endless list of furniture to be bought and things to be painted. Making such a big step in life has the tendency to make one feel reflective, and as I sit here in our kitchen looking out the window at the trains trundling by, I'm thinking about a number of things.
The first, naturally, is about technology and the Internet. It wouldn't be my blog if it that wasn't the first item on the list. I've not had a connection for the previous few days, partly because we were waiting for an engineer to climb the pole to the junction box at our new place, and partly because Three has the worst mobile reception I've ever encountered in Brighton; in fact, I'm convinced that Three is an exercise in meditation, letting go and controlling my own urges, as when I get the twitch to check my emails or Twitter account I usually can't. But anyway, that's an aside. Each time I go without an Internet connection I really notice how much of my brain I've delegated elsewhere to be replaced via quick searches, emails or my bookmarks. Yesterday I considered where to buy paint from, yet I had no idea where the nearest supplier was, and when considering further and thinking about asking our neighbours, I realised I don't even know half of the road names around here, since I lazily delegate that to my GPS. And in my very own Big Yellow Taxi moment, it became apparent just how much the technology that we're building has had a profound effect on our lives, and how I really wouldn't want to be without it. I was considering cycling ever-bigger concentric circles on my bike, but it was raining, so that was put on the back-burner.
The second is perhaps more profound, and I wonder whether I can muster the eloquence to portray my thoughts here, but I'll give it a go. The house we're in now has wonderful natural light; there are windows in every room, and especially notable is a beautiful long window as you walk up the staircase, which lets in the light in the morning. The flat we lived in previously was quite the opposite; there was very little light from the outside world, and much like hibernating animals we lived in stasis for a number of years, putting everything aside for a moment in the future which has now passed. Given the light and mirrors I've seen myself from a number of different angles over the past few days, and I've changed; a bit older, a bit wiser, a bit busier. Technology is addictive, and it's a pleasure to spend my professional life helping build it. But away from the screen the seasons are changing, and whether we like it or not, so are we. Look around.
Wednesday 19 November 2014
Sunday 9 November 2014
You can build a lot in a day
On Saturday at Brandwatch HQ, we hosted our inaugural student event, which we called BestIn.class. When I was an undergraduate student, I was often intimidated by industry, especially when reading and hearing about interview horror stories; they gave me the same feeling I used to get worrying about exams. We thought it would be a good idea to open up the office to university students in the area (Sussex undergraduates were the biggest attendees) for a day of coding with a show and tell session at the end. The premise was that the next generation of programmers could come and see that we aren't so scary after all, and that much of the code that they are writing from day to day isn't all that different from what we're writing from day to day.
I was hesitant going into the event, as I was primarily concerned that people wouldn't turn up. I was wrong there. I was also concerned that 6 hours wasn't enough time for people to build anything cool. I was wrong there also. We gave the attendees a number of code samples ahead of time so that they could prepare. These ranged from code to connect to and parse the Twitter stream with Hosebird to a Spring Boot application that could login and access your friends on Facebook. However, we kept the datasets secret until the day to test everyone's creativity. We chose a dataset of 2500+ news articles on the day of the World Cup 2014 final, another with 5000 posts from the last week about ebola, and another with 5000 posts around the week of Robin Williams' death.
What our attendees came up with was extremely impressive:
I was hesitant going into the event, as I was primarily concerned that people wouldn't turn up. I was wrong there. I was also concerned that 6 hours wasn't enough time for people to build anything cool. I was wrong there also. We gave the attendees a number of code samples ahead of time so that they could prepare. These ranged from code to connect to and parse the Twitter stream with Hosebird to a Spring Boot application that could login and access your friends on Facebook. However, we kept the datasets secret until the day to test everyone's creativity. We chose a dataset of 2500+ news articles on the day of the World Cup 2014 final, another with 5000 posts from the last week about ebola, and another with 5000 posts around the week of Robin Williams' death.
What our attendees came up with was extremely impressive:
- An attempt to automatically classify tweet locations when no geo-coordinates are given (a problem we struggle with ourselves)
- An implementation of a Caesar cypher for tweets
- A map plot of the most talked about African countries in the ebola dataset
- A map plot of where the ebola articles were posted from
- A live Twitter stream with interactive cat GIFs driven by the emotion of the posts (a rules-based classifier)
- A search engine for sentiment in different countries for World Cup players using Apache Lucene
- A sentiment graph over time for the articles on the World Cup finals day
- A 3D word mesh of the common words and their links in the posts around Robin Williams' death
Not only were the projects impressive; the enthusiasm for the event was extremely high, with attendees fighting to get everything finished before the presentations, and a real urge to show off what had been worked on. In short, I was extremely impressed, and I'm really looking forward to hosting another event like this in the future. Thanks folks!
Thursday 30 October 2014
Some thoughts on microservices
One of the themes that reoccurred through many of the talks at JAX London this year was microservices. Now, I'll be the first to admit that I really don't like the term microservices because it's very ambiguous: what's a service? And why is it "micro"? In comparison to what exactly? What does it do? However, regardless of arguments over the nomenclature, it's clear that service-oriented architectures are widely discussed. One may even say they're a bit of a buzzword right now.
You may find yourself faced with a monolithic codebase already, and you want to reuse a part of it from another application in your architecture. This would be a good opportunity to pull that code out into a separate repository and run it as a standalone application, as you can decrease the complexity of the monolithic code at the same time. Just decide on an interface for the other parts of your architecture that are calling it, and away you go. It may even be a good opportunity to write that code from scratch if it's sufficiently small enough, knowing what you do now. Why not open source it while you're at it?
Dealing with scale can naturally steer you towards a service-oriented architecture. Perhaps there is a part of your data collection process that is becoming a bottleneck. It could be split out into a service. This brings additional benefits such as allowing multiple instances to be run for load balancing purposes, allowing you to horizontally scale that part of the code without horizontally scaling the rest of it at the same time.
It's worth bearing in mind that while splitting your architecture into smaller services can reduce the complexity of the code and make it easier to understand various parts of the system in isolation, it pushes out the complexity into managing how the services communicate with each other. If you change the REST responses of a service, or alter the Java class that you're serialising to send down the wire, which other parts of the system will it break? It's hard to know at compile time. A team might not know that they are subtly breaking the interface with another team's service. Communication, monitoring and regression testing are extremely important here.
So, services. I like them. Maybe you will too, but be careful and apply them gently.
If your organisation is structured similarly to Spotify's matrix (and we find it works well), then there will be few cross-team dependencies. Each team will able to get on and build stuff without anything or anybody getting in their way. Some teams will build new features, some will improve various parts of the platform, and others may focus on scaling the architecture. The general gist is that your engineering teams are all going to be building very different parts of the system with little to no overlap. Naturally, your team, when faced with the task of building a new large architectural feature, will jump at the opportunity to stop contributing to the big ball of mud that they usually commit to. "Let's have our own repo!" they cry.
Conway's Law conjectures that the architecture of the software that is produced in an organisation reflects the communication paths and team structures, and to an extent I think that's true. In an ideal world, each team would like to have their own codebase and process. This pushes the inter-team complexity down and keeps the need to look at "that" code at a minimum. The team may want to control their release cycle, to do their own continuous deployment, and generally feel like they have complete ownership over what they are producing and when they are producing it. This is great for team morale. A new feature can be built as a standalone application. Some months down the line, another team start building another new feature that would get out of the door really quickly if they were able to reuse the work that the previous team did, but neither of them want to work in each others' repositories; it just feels wrong. How can an interface be provided for other internal teams to work with? Thinking about features as services can act as guidance here.
You may find yourself faced with a monolithic codebase already, and you want to reuse a part of it from another application in your architecture. This would be a good opportunity to pull that code out into a separate repository and run it as a standalone application, as you can decrease the complexity of the monolithic code at the same time. Just decide on an interface for the other parts of your architecture that are calling it, and away you go. It may even be a good opportunity to write that code from scratch if it's sufficiently small enough, knowing what you do now. Why not open source it while you're at it?
Dealing with scale can naturally steer you towards a service-oriented architecture. Perhaps there is a part of your data collection process that is becoming a bottleneck. It could be split out into a service. This brings additional benefits such as allowing multiple instances to be run for load balancing purposes, allowing you to horizontally scale that part of the code without horizontally scaling the rest of it at the same time.
There's no right and wrong answer here, and there's no silver bullet for all situations. It may help to categorise what sort of contract your service will have with the rest of the system. Here are some examples.
RESTful services
If your service provides timely responses to requests, then a REST API may be a good approach. For example, part of your architecture may compute a set of recommended products based on a given product. Using a REST API also gives the added benefit of considering allowing external access in the future, either for free or for a price. Spring Boot allows you to get webapps up and running extremely quickly, and I'd recommend looking at it for a new project. They also have examples of how to write REST consumer applications so that your services can talk to each other with minimal effort.Pipeline services
You may be splitting up a data collection pipeline because a certain part is a bottleneck. Since this area of the system will never have an external facing API, using a message broker to pass intermediate data is a good idea. We've been using Apache Kafka for this at Brandwatch with great success. You can decide how to distribute the load between your various instances with a lot of flexibility. I gave a talk on how we're using leader election to do that in our event-detection pipeline.Slow services
Some services can take a long time to provide a final response. For example, you may be firing off a batch job like updating a large portion of a search index, or performing a large MapReduce task. A REST API could work well here, with the request returning the location that the output is expected to be stored ahead of time. Polling can wait for it to appear, and this kind of task can be delegated to a background thread in your application.It's worth bearing in mind that while splitting your architecture into smaller services can reduce the complexity of the code and make it easier to understand various parts of the system in isolation, it pushes out the complexity into managing how the services communicate with each other. If you change the REST responses of a service, or alter the Java class that you're serialising to send down the wire, which other parts of the system will it break? It's hard to know at compile time. A team might not know that they are subtly breaking the interface with another team's service. Communication, monitoring and regression testing are extremely important here.
So, services. I like them. Maybe you will too, but be careful and apply them gently.
Sunday 26 October 2014
Three years later
I would ordinarily scoff when I looked at a blog and saw that it hadn't been updated for a long period of time. I am now scoffing at myself. The last post that I made was just before I started working at Brandwatch, which was a few months after I finished my Ph.D., which was, more precisely, 3 years and 1 month ago. I've now been working here for longer than I was undertaking my doctoral studies and time has passed very quickly indeed. Work and life are very different now.
There's a lot to be said about teams of engineers tackling big problems. My doctoral years were very solitary in terms of my work. You toil on a deep problem in a niche so narrow that at the end of it all you are the world expert on it. This is an empowering proposition, however, it does mean that nobody truly shares your burden and successes along the way apart from yourself. I thrived in this environment because I can command a substantial amount of self-determination, and to an extent, stubbornness. This is what got me out of bed in the morning and kept me pushing through until late in the night, day in and day out. It paid off. But it's not sustainable. Research in academia is being overtaken by the pure creative power of industry. I'd make the conjecture that industry is solving the most important and interesting problems in computer science right now, and that's where I want to be.
The true joy of engineering is not sitting in an ivory tower and becoming a specialist on a particularly arcane area that few people know about. The true joy is building things with people, for people. Fundamentally, software engineers are not that different from traditional engineers. When Brunel built bridges or railways, he did so to solve people's problems; to connect people. Software engineering is the same, and connecting people resonates both inside and outside of the workshop.
Inside the workshop, engineers connect with each other to dream, design and build things that help enrich the lives of others. Teams of engineers, when they work together well and have proper guidance, can do incredible things. Working closely with exceptionally smart people has been extremely rewarding. My programming skills have improved vastly. My ability to transfer this knowledge on to others has also improved, and I feel the audience has been infinitely more attentive than those that I taught at the university (intriguingly those that were most receptive in my seminars now work for the same company). Through pull requests, pair programming and technical talks the interactions where I learn something new can be counted daily.
Outside of the workshop, engineers are connecting people together. When I read and hear feedback from our users that compliments our work, I feel genuinely happy to have made a difference. Even if the contribution was something small, it doesn't matter - we've made someone's day better: perhaps we fixed a common frustration, maybe we saved them a few minutes each morning, or we've delivered some great new functionality. When hundreds of clients turn up to one of our events to learn more about the platform and products we are building, I feel part of something much greater than myself. When I spoke at JAX London earlier this month, a whole room of people were really listening to what I had to say and I felt that I'd given something useful back to the community. That was the biggest struggle I had with academic work; collaboration seemed fueled by the need to publish more papers for one's own citation count, rather than tackling a problem the world really needed solved.
If you're an engineer and you want to make a difference, then know full well that you can. There's never been a more exciting time to be alive with the skills that you have. Join a start-up and help it grow into a world-class organisation. Join a world-class organisation and make it better. Work for yourself while traveling from place to place with a laptop and a minimal set of clothing - it's all possible. There's so much to build for everyone out there - there's just not enough time to do it all.
There's a lot to be said about teams of engineers tackling big problems. My doctoral years were very solitary in terms of my work. You toil on a deep problem in a niche so narrow that at the end of it all you are the world expert on it. This is an empowering proposition, however, it does mean that nobody truly shares your burden and successes along the way apart from yourself. I thrived in this environment because I can command a substantial amount of self-determination, and to an extent, stubbornness. This is what got me out of bed in the morning and kept me pushing through until late in the night, day in and day out. It paid off. But it's not sustainable. Research in academia is being overtaken by the pure creative power of industry. I'd make the conjecture that industry is solving the most important and interesting problems in computer science right now, and that's where I want to be.
The true joy of engineering is not sitting in an ivory tower and becoming a specialist on a particularly arcane area that few people know about. The true joy is building things with people, for people. Fundamentally, software engineers are not that different from traditional engineers. When Brunel built bridges or railways, he did so to solve people's problems; to connect people. Software engineering is the same, and connecting people resonates both inside and outside of the workshop.
Inside the workshop, engineers connect with each other to dream, design and build things that help enrich the lives of others. Teams of engineers, when they work together well and have proper guidance, can do incredible things. Working closely with exceptionally smart people has been extremely rewarding. My programming skills have improved vastly. My ability to transfer this knowledge on to others has also improved, and I feel the audience has been infinitely more attentive than those that I taught at the university (intriguingly those that were most receptive in my seminars now work for the same company). Through pull requests, pair programming and technical talks the interactions where I learn something new can be counted daily.
Outside of the workshop, engineers are connecting people together. When I read and hear feedback from our users that compliments our work, I feel genuinely happy to have made a difference. Even if the contribution was something small, it doesn't matter - we've made someone's day better: perhaps we fixed a common frustration, maybe we saved them a few minutes each morning, or we've delivered some great new functionality. When hundreds of clients turn up to one of our events to learn more about the platform and products we are building, I feel part of something much greater than myself. When I spoke at JAX London earlier this month, a whole room of people were really listening to what I had to say and I felt that I'd given something useful back to the community. That was the biggest struggle I had with academic work; collaboration seemed fueled by the need to publish more papers for one's own citation count, rather than tackling a problem the world really needed solved.
If you're an engineer and you want to make a difference, then know full well that you can. There's never been a more exciting time to be alive with the skills that you have. Join a start-up and help it grow into a world-class organisation. Join a world-class organisation and make it better. Work for yourself while traveling from place to place with a laptop and a minimal set of clothing - it's all possible. There's so much to build for everyone out there - there's just not enough time to do it all.
Friday 23 September 2011
New things
It's been quite a busy few months. After handing in my PhD thesis at the end of June, I did manage to have some time off, but as is always the case, I ended up involved with some new projects and the inevitable job search that followed afterwards.
Some smaller, nice things first. I had one of my pictures exhibited at the Duke of York's Picturehouse during August as I've previously mentioned. Back then, at the time of writing, I hadn't actually seen the finished framed result. I had a custom frame made for me by the nice folks down at Spectrum Photo in Hove, and they did a great job. For a pretty reasonable price I got an archival dibond mount and frame. Here's a particularly rubbish picture of it because it's hard to not have anything reflecting in it when you take the shot.
Then, at the beginning of September, thanks to the invitation from Ollie Glass, I installed a modified version of my PixelWorms simulation at the after-party of Update 2011, which was hosted in Brighton Museum at night (which was very cool). Rather than the simulation being based on the weather as before, I made it into a hands-off game where visitors could pop their name in via a website, and then the worms would roam, breed and fight on the display. Judging by the number of people hanging out near it on the night, it seemed to go down pretty well. The log file that PixelWorms generated that night is available here if you fancy having a look through it. There was one brilliant benefit to having made something; the Royal Banquet at Brighton Pavilion on Sunday night, of which I got two complimentary tickets. It was just incredible, and this 360 degree panorama doesn't need too many words.
Shortly after, I got another year older. Rebecca made me an awesome cake.
I'm also happy to say that after a fairly gruelling few months of job interviews, considering commutes (and some were really epic commutes), companies, academia and industry, I chose to take up a position at Brandwatch. Out of all of the places I went, and there were a few, I couldn't find any that gave me the same "this is right" feeling. Plus, I walk to work and it takes me about 15 minutes. And there's pastry Mondays. And Fridays. And 20% personal project time. And private healthcare. And free food and drinks. And there's a bunch of smart folk there. All in all, bon oeufs.
My PhD viva is lined up for the 14th October, so that's the next thing in the pipeline. The Winter issue of ACM XRDS has wrapped up and the Fall issue has just been published. Here's to a good rest of the year, because it All Worked Out Fine In The End.
Friday 15 July 2011
Fun: PixelWorms
I've had the luxury of having a bit of spare time to do some fun programming on the side since I submitted my thesis the other week. The last couple of days I was having a play with the Yahoo! weather feeds, which let you get weather information based on a geographic location.
I made a little project called PixelWorms, which is a model of a patch of grass in Brighton, England. It uses the aforementioned weather feed to work in real time with the current weather conditions. The pixel earthworms are more likely to appear and reproduce if it is cloudy, night time, or cool outside. Worms are less likely to appear and reproduce if it is overly cold, dry, or hot. If it's raining outside, then the worms show a lot of activity. If it's night time outside, the visuals are meant to (sort of) look like you're viewing the ground through a pair of night vision goggles.
Thanks to Daniel Shiffman for providing an example of how to use the Yahoo! weather feeds.
I made a little project called PixelWorms, which is a model of a patch of grass in Brighton, England. It uses the aforementioned weather feed to work in real time with the current weather conditions. The pixel earthworms are more likely to appear and reproduce if it is cloudy, night time, or cool outside. Worms are less likely to appear and reproduce if it is overly cold, dry, or hot. If it's raining outside, then the worms show a lot of activity. If it's night time outside, the visuals are meant to (sort of) look like you're viewing the ground through a pair of night vision goggles.
Thanks to Daniel Shiffman for providing an example of how to use the Yahoo! weather feeds.
PixelWorms from James Stanier on Vimeo.
Monday 11 July 2011
Art: sliders
I had an idea for a painting today, however it dawned on me that I cannot paint. So, instead, I programmed it. Here's five of an infinite selection of random number distribution visualisations of which I am being pretentious in calling the "Slider" series. Blogger appears to have some size limitations, so I'll post more of these on my actual website soon.
 |
| Slider (1) |
 |
| Slider (2) |
 |
| Slider (3) |
 |
| Slider (4) |
 |
| Slider (5) |
Subscribe to:
Posts (Atom)