tag:blogger.com,1999:blog-37097610460408181192024-02-18T21:10:45.646-08:00Here be dragonsJames Stanierhttp://www.blogger.com/profile/05102706474455020954noreply@blogger.comBlogger11125tag:blogger.com,1999:blog-3709761046040818119.post-4710624702313614732014-11-19T01:55:00.000-08:002014-11-19T01:55:46.963-08:00DowntimeIt's been a big week for us: we're now homeowners. The last few days have revolved around cardboard boxes, being thoroughly terrified of the wobbly ladder to our attic, and trying to work out what to do first in an endless list of furniture to be bought and things to be painted. Making such a big step in life has the tendency to make one feel reflective, and as I sit here in our kitchen looking out the window at the trains trundling by, I'm thinking about a number of things.<br />
<br />
The first, naturally, is about technology and the Internet. It wouldn't be my blog if it that wasn't the first item on the list. I've not had a connection for the previous few days, partly because we were waiting for an engineer to climb the pole to the junction box at our new place, and partly because Three has the worst mobile reception I've ever encountered in Brighton; in fact, I'm convinced that Three is an exercise in meditation, letting go and controlling my own urges, as when I get the twitch to check my emails or Twitter account I usually can't. But anyway, that's an aside. Each time I go without an Internet connection I really notice how much of my brain I've delegated elsewhere to be replaced via quick searches, emails or my bookmarks. Yesterday I considered where to buy paint from, yet I had no idea where the nearest supplier was, and when considering further and thinking about asking our neighbours, I realised I don't even know half of the road names around here, since I lazily delegate that to my GPS. And in my very own Big Yellow Taxi moment, it became apparent just how much the technology that we're building has had a profound effect on our lives, and how I really wouldn't want to be without it. I was considering cycling ever-bigger concentric circles on my bike, but it was raining, so that was put on the back-burner.<br />
<br />
The second is perhaps more profound, and I wonder whether I can muster the eloquence to portray my thoughts here, but I'll give it a go. The house we're in now has wonderful natural light; there are windows in every room, and especially notable is a beautiful long window as you walk up the staircase, which lets in the light in the morning. The flat we lived in previously was quite the opposite; there was very little light from the outside world, and much like hibernating animals we lived in stasis for a number of years, putting everything aside for a moment in the future which has now passed. Given the light and mirrors I've seen myself from a number of different angles over the past few days, and I've changed; a bit older, a bit wiser, a bit busier. Technology is addictive, and it's a pleasure to spend my professional life helping build it. But away from the screen the seasons are changing, and whether we like it or not, so are we. Look around.James Stanierhttp://www.blogger.com/profile/05102706474455020954noreply@blogger.com0tag:blogger.com,1999:blog-3709761046040818119.post-85603005570397486772014-11-09T13:41:00.000-08:002014-11-09T13:41:00.608-08:00You can build a lot in a dayOn Saturday at <a href="http://www.brandwatch.com/">Brandwatch</a> HQ, we hosted our inaugural student event, which we called <a href="http://www.brandwatch.com/bestinclass/">BestIn.class</a>. When I was an undergraduate student, I was often intimidated by industry, especially when reading and hearing about interview horror stories; they gave me the same feeling I used to get worrying about exams. We thought it would be a good idea to open up the office to university students in the area (Sussex undergraduates were the biggest attendees) for a day of coding with a show and tell session at the end. The premise was that the next generation of programmers could come and see that we aren't so scary after all, and that much of the code that they are writing from day to day isn't all that different from what we're writing from day to day.<br />
<br />
I was hesitant going into the event, as I was primarily concerned that people wouldn't turn up. I was wrong there. I was also concerned that 6 hours wasn't enough time for people to build anything cool. I was wrong there also. We gave the attendees a number of code samples ahead of time so that they could prepare. These ranged from code to connect to and parse the Twitter stream with <a href="https://github.com/twitter/hbc">Hosebird</a> to a <a href="http://spring.io/guides/gs/accessing-facebook/">Spring Boot application</a> that could login and access your friends on Facebook. However, we kept the datasets secret until the day to test everyone's creativity. We chose a dataset of 2500+ news articles on the day of the World Cup 2014 final, another with 5000 posts from the last week about ebola, and another with 5000 posts around the week of Robin Williams' death.<br />
<br />
What our attendees came up with was extremely impressive:<br />
<br />
<ul>
<li>An attempt to automatically classify tweet locations when no geo-coordinates are given (a problem we struggle with ourselves)</li>
<li>An implementation of a Caesar cypher for tweets</li>
<li>A map plot of the most talked about African countries in the ebola dataset</li>
<li>A map plot of where the ebola articles were posted from</li>
<li>A live Twitter stream with interactive cat GIFs driven by the emotion of the posts (a rules-based classifier)</li>
<li>A search engine for sentiment in different countries for World Cup players using <a href="http://lucene.apache.org/core/">Apache Lucene</a></li>
<li>A sentiment graph over time for the articles on the World Cup finals day</li>
<li>A 3D word mesh of the common words and their links in the posts around Robin Williams' death</li>
</ul>
<div>
Not only were the projects impressive; the enthusiasm for the event was extremely high, with attendees fighting to get everything finished before the presentations, and a real urge to show off what had been worked on. In short, I was extremely impressed, and I'm really looking forward to hosting another event like this in the future. Thanks folks!</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://pbs.twimg.com/media/B16xOG4IUAAvj9p.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="https://pbs.twimg.com/media/B16xOG4IUAAvj9p.jpg" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://pbs.twimg.com/media/B16xOFmIgAAITRO.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="https://pbs.twimg.com/media/B16xOFmIgAAITRO.jpg" width="320" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
James Stanierhttp://www.blogger.com/profile/05102706474455020954noreply@blogger.com0tag:blogger.com,1999:blog-3709761046040818119.post-5565773550198848762014-10-30T16:04:00.000-07:002014-10-30T16:16:06.898-07:00Some thoughts on microservicesOne of the themes that reoccurred through many of the talks at <a href="http://jaxlondon.com/2014/">JAX London</a> this year was microservices. Now, I'll be the first to admit that I really don't like the term microservices because it's very ambiguous: what's a service? And why is it "micro"? In comparison to what exactly? What does it do? However, regardless of arguments over the nomenclature, it's clear that service-oriented architectures are widely discussed. One may even say they're a bit of a buzzword right now.<br />
<div>
<br /></div>
<div>
If your organisation is structured similarly to <a href="http://blog.crisp.se/2012/11/14/henrikkniberg/scaling-agile-at-spotify">Spotify's matrix</a> (and we find it works well), then there will be few cross-team dependencies. Each team will able to get on and build stuff without anything or anybody getting in their way. Some teams will build new features, some will improve various parts of the platform, and others may focus on scaling the architecture. The general gist is that your engineering teams are all going to be building very different parts of the system with little to no overlap. Naturally, your team, when faced with the task of building a new large architectural feature, will jump at the opportunity to stop contributing to the <a href="http://en.wikipedia.org/wiki/Big_ball_of_mud">big ball of mud</a> that they usually commit to. "Let's have our own repo!" they cry.</div>
<div>
<br /></div>
<div>
<a href="http://en.wikipedia.org/wiki/Conway's_law">Conway's Law</a> conjectures that the architecture of the software that is produced in an organisation reflects the communication paths and team structures, and to an extent I think that's true. In an ideal world, each team would like to have their own codebase and process. This pushes the inter-team complexity down and keeps the need to look at "that" code at a minimum. The team may want to control their release cycle, to do their own continuous deployment, and generally feel like they have complete ownership over what they are producing and when they are producing it. This is great for team morale. A new feature can be built as a standalone application. Some months down the line, another team start building another new feature that would get out of the door really quickly if they were able to reuse the work that the previous team did, but neither of them want to work in each others' repositories; it just feels wrong. How can an interface be provided for other internal teams to work with? Thinking about features as services can act as guidance here.</div>
<div>
<br />
You may find yourself faced with a monolithic codebase already, and you want to reuse a part of it from another application in your architecture. This would be a good opportunity to pull that code out into a separate repository and run it as a standalone application, as you can decrease the complexity of the monolithic code at the same time. Just decide on an interface for the other parts of your architecture that are calling it, and away you go. It may even be a good opportunity to write that code from scratch if it's sufficiently small enough, knowing what you do now. Why not open source it while you're at it?<br />
<br />
Dealing with scale can naturally steer you towards a service-oriented architecture. Perhaps there is a part of your data collection process that is becoming a bottleneck. It could be split out into a service. This brings additional benefits such as allowing multiple instances to be run for load balancing purposes, allowing you to horizontally scale that part of the code without horizontally scaling the rest of it at the same time.</div>
<div>
<br /></div>
<div>
There's no right and wrong answer here, and there's no silver bullet for all situations. It may help to categorise what sort of contract your service will have with the rest of the system. Here are some examples.</div>
<div>
<br />
<h3>
RESTful services</h3>
If your service provides timely responses to requests, then a REST API may be a good approach. For example, part of your architecture may compute a set of recommended products based on a given product. Using a REST API also gives the added benefit of considering allowing external access in the future, either for free or for a price. <a href="http://projects.spring.io/spring-boot/">Spring Boot</a> allows you to get webapps up and running extremely quickly, and I'd recommend looking at it for a new project. They also have <a href="https://spring.io/guides/gs/consuming-rest/">examples of how to write REST consumer applications</a> so that your services can talk to each other with minimal effort.<br />
<br />
<h3>
Pipeline services</h3>
You may be splitting up a data collection pipeline because a certain part is a bottleneck. Since this area of the system will never have an external facing API, using a message broker to pass intermediate data is a good idea. We've been using <a href="http://kafka.apache.org/">Apache Kafka </a>for this at <a href="http://www.brandwatch.com/">Brandwatch</a> with great success. You can decide how to distribute the load between your various instances with a lot of flexibility. <a href="https://drive.google.com/open?id=0BzTJIn--JbQQUHJNQ0JvQmRZZWs&authuser=1">I gave a talk</a> on how we're using leader election to do that in our event-detection pipeline.<br />
<br /></div>
<div>
<h3>
Slow services</h3>
Some services can take a long time to provide a final response. For example, you may be firing off a batch job like updating a large portion of a search index, or performing a large MapReduce task. A REST API could work well here, with the request returning the location that the output is expected to be stored ahead of time. Polling can wait for it to appear, and this kind of task can be delegated to a background thread in your application.<br />
<br />
It's worth bearing in mind that while splitting your architecture into smaller services can reduce the complexity of the code and make it easier to understand various parts of the system in isolation, it pushes out the complexity into managing how the services communicate with each other. If you change the REST responses of a service, or alter the Java class that you're serialising to send down the wire, which other parts of the system will it break? It's hard to know at compile time. A team might not know that they are subtly breaking the interface with another team's service. Communication, monitoring and regression testing are extremely important here.<br />
<br />
So, services. I like them. Maybe you will too, but be careful and apply them gently.</div>
James Stanierhttp://www.blogger.com/profile/05102706474455020954noreply@blogger.com0tag:blogger.com,1999:blog-3709761046040818119.post-17244641641165287852014-10-26T13:36:00.001-07:002014-10-26T13:36:39.805-07:00Three years laterI would ordinarily scoff when I looked at a blog and saw that it hadn't been updated for a long period of time. I am now scoffing at myself. The last post that I made was just before I started working at <a href="http://www.brandwatch.com/">Brandwatch</a>, which was a few months after I finished my Ph.D., which was, more precisely, 3 years and 1 month ago. I've now been working here for longer than I was undertaking my doctoral studies and time has passed very quickly indeed. Work and life are very different now.<br />
<br />
There's a lot to be said about teams of engineers tackling big problems. My doctoral years were very solitary in terms of my work. You toil on a deep problem in a niche so narrow that at the end of it all you are the world expert on it. This is an empowering proposition, however, it does mean that nobody truly shares your burden and successes along the way apart from yourself. I thrived in this environment because I can command a substantial amount of self-determination, and to an extent, stubbornness. This is what got me out of bed in the morning and kept me pushing through until late in the night, day in and day out. It paid off. But it's not sustainable. Research in academia is being overtaken by the pure creative power of industry. I'd make the conjecture that industry is solving the most important and interesting problems in computer science right now, and that's where I want to be.<br />
<br />
The true joy of engineering is not sitting in an ivory tower and becoming a specialist on a particularly arcane area that few people know about. The true joy is building things with people, for people. Fundamentally, software engineers are not that different from traditional engineers. When Brunel built bridges or railways, he did so to solve people's problems; to connect people. Software engineering is the same, and connecting people resonates both inside and outside of the workshop.<br />
<br />
Inside the workshop, engineers connect with each other to dream, design and build things that help enrich the lives of others. Teams of engineers, when they work together well and have proper guidance, can do incredible things. Working closely with exceptionally smart people has been extremely rewarding. My programming skills have improved vastly. My ability to transfer this knowledge on to others has also improved, and I feel the audience has been infinitely more attentive than those that I taught at the university (intriguingly those that were most receptive in my seminars now work for the same company). Through pull requests, pair programming and technical talks the interactions where I learn something new can be counted daily.<br />
<br />
Outside of the workshop, engineers are connecting people together. When I read and hear feedback from our users that compliments our work, I feel genuinely happy to have made a difference. Even if the contribution was something small, it doesn't matter - we've made someone's day better: perhaps we fixed a common frustration, maybe we saved them a few minutes each morning, or we've delivered some great new functionality. When hundreds of clients turn up to one of our events to learn more about the platform and products we are building, I feel part of something much greater than myself. When I spoke at <a href="http://jaxlondon.com/2014/timetable">JAX London</a> earlier this month, a whole room of people were really listening to what I had to say and I felt that I'd given something useful back to the community. That was the biggest struggle I had with academic work; collaboration seemed fueled by the need to publish more papers for one's own citation count, rather than tackling a problem the world really needed solved.<br />
<br />
If you're an engineer and you want to make a difference, then know full well that you can. There's never been a more exciting time to be alive with the skills that you have. Join a start-up and help it grow into a world-class organisation. Join a world-class organisation and make it better. Work for yourself while traveling from place to place with a laptop and a minimal set of clothing - it's all possible. There's so much to build for everyone out there - there's just not enough time to do it all.James Stanierhttp://www.blogger.com/profile/05102706474455020954noreply@blogger.com0tag:blogger.com,1999:blog-3709761046040818119.post-7820329685959378382011-09-23T13:15:00.000-07:002011-09-23T13:15:49.845-07:00New thingsIt's been quite a busy few months. After handing in my PhD thesis at the end of June, I did manage to have some time off, but as is always the case, I ended up involved with some new projects and the inevitable job search that followed afterwards.<div>
<br /></div>
<div>
Some smaller, nice things first. I had one of my pictures exhibited at the Duke of York's Picturehouse during August as I've previously mentioned. Back then, at the time of writing, I hadn't actually seen the finished framed result. I had a custom frame made for me by the nice folks down at Spectrum Photo in Hove, and they did a great job. For a pretty reasonable price I got an archival dibond mount and frame. Here's a particularly rubbish picture of it because it's hard to not have anything reflecting in it when you take the shot.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGsqb9lHVTtTp6U7SiQqXXGC4nLU3z3_XhlbEaRJx1-ERpjzyRvMN2TeLFgptZi08KXst0SAoTrurIKj4E86sduX2SnNvO95V3EFN03eWaOb_KLsVVf6WCDS0M69MiavuursrtrStmc1A/s1600/slider1.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="224" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGsqb9lHVTtTp6U7SiQqXXGC4nLU3z3_XhlbEaRJx1-ERpjzyRvMN2TeLFgptZi08KXst0SAoTrurIKj4E86sduX2SnNvO95V3EFN03eWaOb_KLsVVf6WCDS0M69MiavuursrtrStmc1A/s400/slider1.jpeg" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div>
Then, at the beginning of September, thanks to the invitation from Ollie Glass, I installed a modified version of my <a href="http://stanierj.blogspot.com/2011/07/fun-pixelworms.html">PixelWorms simulation</a> at the after-party of <a href="http://updateconf.com/">Update 2011</a>, which was hosted in Brighton Museum at night (which was very cool). Rather than the simulation being based on the weather as before, I made it into a hands-off game where visitors could pop their name in via a website, and then the worms would roam, breed and fight on the display. Judging by the number of people hanging out near it on the night, it seemed to go down pretty well. The log file that PixelWorms generated that night is available <a href="http://www.informatics.sussex.ac.uk/users/js231/projects/pixelworms/wormslog.html">here</a> if you fancy having a look through it. There was one brilliant benefit to having made something; the Royal Banquet at Brighton Pavilion on Sunday night, of which I got two complimentary tickets. It was just incredible, and this <a href="http://360.io/CprvGt">360 degree panorama</a> doesn't need too many words.</div>
<div>
<br /></div>
<div>
Shortly after, I got another year older. Rebecca made me an awesome cake.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieybhm0niow9U0K5iMlCnk6aUicxgvrmFmuOkdeds0A6izcNoDxkcCyOdN_5N9MlFGFxHHFmD1VVgjxxSchWFzCAZmbluA7FRiXKg6QYvgT_Nmo28hrP2y1Yfayy-JOH7-JYHscAMsyGI/s1600/313256_10150775973625526_671115525_20409357_3615724_n.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="265" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieybhm0niow9U0K5iMlCnk6aUicxgvrmFmuOkdeds0A6izcNoDxkcCyOdN_5N9MlFGFxHHFmD1VVgjxxSchWFzCAZmbluA7FRiXKg6QYvgT_Nmo28hrP2y1Yfayy-JOH7-JYHscAMsyGI/s400/313256_10150775973625526_671115525_20409357_3615724_n.jpeg" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
I'm also happy to say that after a fairly gruelling few months of job interviews, considering commutes (and some were really epic commutes), companies, academia and industry, I chose to take up a position at <a href="http://www.brandwatch.com/">Brandwatch</a>. Out of all of the places I went, and there were a few, I couldn't find any that gave me the same "this is right" feeling. Plus, I walk to work and it takes me about 15 minutes. And there's pastry Mondays. And Fridays. And 20% personal project time. And private healthcare. And free food and drinks. And there's a bunch of smart folk there. All in all, bon oeufs.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
My PhD viva is lined up for the 14th October, so that's the next thing in the pipeline. The Winter issue of ACM XRDS has wrapped up and the <a href="http://xrds.acm.org/current-issue.cfm">Fall issue has just been published</a>. Here's to a good rest of the year, because it <i>All Worked Out Fine In The End</i>.</div>
James Stanierhttp://www.blogger.com/profile/05102706474455020954noreply@blogger.com0tag:blogger.com,1999:blog-3709761046040818119.post-56051459701438395502011-07-15T05:49:00.000-07:002011-07-15T10:39:49.982-07:00Fun: PixelWormsI've had the luxury of having a bit of spare time to do some fun programming on the side since I submitted my thesis the other week. The last couple of days I was having a play with the <a href="http://developer.yahoo.com/weather/">Yahoo! weather feeds</a>, which let you get weather information based on a geographic location.<br />
<br />
I made a little project called PixelWorms, which is a model of a patch of grass in Brighton, England. It uses the aforementioned weather feed to work in real time with the current weather conditions. The pixel earthworms are more likely to appear and reproduce if it is cloudy, night time, or cool outside. Worms are less likely to appear and reproduce if it is overly cold, dry, or hot. If it's raining outside, then the worms show a lot of activity. If it's night time outside, the visuals are meant to (sort of) look like you're viewing the ground through a pair of night vision goggles.<br />
<br />
Thanks to <a href="http://www.learningprocessing.com/">Daniel Shiffman</a> for providing an example of how to use the Yahoo! weather feeds.<br />
<br />
<br />
<center><iframe src="http://player.vimeo.com/video/26480746?title=0&byline=0&portrait=0" width="400" height="300" frameborder="0"></iframe><p><a href="http://vimeo.com/26480746">PixelWorms</a> from <a href="http://vimeo.com/user7777706">James Stanier</a> on <a href="http://vimeo.com">Vimeo</a>.</p></center>James Stanierhttp://www.blogger.com/profile/05102706474455020954noreply@blogger.com0tag:blogger.com,1999:blog-3709761046040818119.post-5234316631872607762011-07-11T15:23:00.000-07:002011-07-11T15:23:31.117-07:00Art: slidersI had an idea for a painting today, however it dawned on me that I cannot paint. So, instead, I programmed it. Here's five of an infinite selection of random number distribution visualisations of which I am being pretentious in calling the "Slider" series. Blogger appears to have some size limitations, so I'll post more of these on my actual website soon.<div><br />
</div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9lLBdZyyiqJDn6113DKoVgFWavSgOQbTDTi16NPV27Blk8IXUSd1xs0tBGlVVu0VCcuslfBwoiB2bFsaRj2KAxeC8Fb2dZ2kNZnLS5EF_nj_hpKgSbwS7-ZWAOBgnS42hJUlWg71oXV4/s1600/rects1.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9lLBdZyyiqJDn6113DKoVgFWavSgOQbTDTi16NPV27Blk8IXUSd1xs0tBGlVVu0VCcuslfBwoiB2bFsaRj2KAxeC8Fb2dZ2kNZnLS5EF_nj_hpKgSbwS7-ZWAOBgnS42hJUlWg71oXV4/s400/rects1.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Slider (1)</td></tr>
</tbody></table><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipe8wYBE3Da6zjq4mJluMHXSjYecnskwqJKrtySuG5PvgJETVyJwkFrIbp7nzgSK4Q1uLbRZwPcwFhQo6ZDh8KaHCrEUAV50fWhMHimEH7GP14QEsNcsoYB3_x6hbyDWi3WMT_UkMwmC4/s1600/rects2.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipe8wYBE3Da6zjq4mJluMHXSjYecnskwqJKrtySuG5PvgJETVyJwkFrIbp7nzgSK4Q1uLbRZwPcwFhQo6ZDh8KaHCrEUAV50fWhMHimEH7GP14QEsNcsoYB3_x6hbyDWi3WMT_UkMwmC4/s400/rects2.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Slider (2)</td></tr>
</tbody></table><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoXn5JCxwChiJWGdE1sbGi4iTsd6pBk1U4kL-_dXqA9YF0aN31kTmAU7VrpOcMRfSDiryrQR1DdTNbn3VJNoEkynOquafntLJBVy4MJy0Hd0GMDsOYyq8SLWpI-gF4jtKo7tsiO21qPYQ/s1600/rects3.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoXn5JCxwChiJWGdE1sbGi4iTsd6pBk1U4kL-_dXqA9YF0aN31kTmAU7VrpOcMRfSDiryrQR1DdTNbn3VJNoEkynOquafntLJBVy4MJy0Hd0GMDsOYyq8SLWpI-gF4jtKo7tsiO21qPYQ/s400/rects3.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Slider (3)</td></tr>
</tbody></table><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEif05wJTkHjbztagEHItfzl14gzo2bPECuC4DzsxM1_Z24pfoldwV9E9PCFN7w_NH72xfsN0TwFUxZF1arpqzwBm9919U7jeRAhT0W8KFmjBkfWMNN4YaUr6wCBbrjb-MDNpCVG15b53Nc/s1600/rects4.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEif05wJTkHjbztagEHItfzl14gzo2bPECuC4DzsxM1_Z24pfoldwV9E9PCFN7w_NH72xfsN0TwFUxZF1arpqzwBm9919U7jeRAhT0W8KFmjBkfWMNN4YaUr6wCBbrjb-MDNpCVG15b53Nc/s400/rects4.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Slider (4)</td></tr>
</tbody></table><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHy-1vITArL-w_birr0ZKJSml7MXQYmo-zqb825xmoDWgLUZHCoouy3u0NClp90FXz7ZFXi_fnzcFhG1OZXmUmjOp9Kz2rD8JUSuid9G7TxQ5i2ywTOoMTLwBNjl2PIjjVkGfT-6eLVK0/s1600/rects5.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHy-1vITArL-w_birr0ZKJSml7MXQYmo-zqb825xmoDWgLUZHCoouy3u0NClp90FXz7ZFXi_fnzcFhG1OZXmUmjOp9Kz2rD8JUSuid9G7TxQ5i2ywTOoMTLwBNjl2PIjjVkGfT-6eLVK0/s400/rects5.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Slider (5)</td></tr>
</tbody></table><div><br />
</div>James Stanierhttp://www.blogger.com/profile/05102706474455020954noreply@blogger.com0tag:blogger.com,1999:blog-3709761046040818119.post-12081615884609279962011-07-06T13:02:00.000-07:002011-07-06T13:02:14.189-07:00Advice to PhD students #4: write, write, writeThis piece of advice might seem really silly, but it's incredibly important. The primary goal of your PhD is to produce your thesis, and that's a lot of words that need to be written. More importantly, it's a lot of <b>good</b> words that need to be written. So, it's essential that during your research you practice writing as much as possible. I've noticed that the standard of writing in computer science is very varied, and quite often, it's very poor. This is a terrible thing, because you may have some of the best ideas in the world, but if you can't communicate them correctly then nobody is going to bother to read your work. This is a tragedy if you want to continue in academia.<br />
<br />
So, how can you improve your writing? The only way is to write, and to keep writing. Writing is very much like programming in that you learn by experience. One can be extremely well-versed in technique and grammar but completely incapable of putting together a sentence that reads well. Style tends to be learned through osmosis, by reading broadly from scientific papers, to fiction, to newspapers and magazines. <br />
<br />
The more you write, the better you get, and the easier that subsequent writing becomes. I've met various PhD students who absolutely hate writing. They tend not to publish very much. As a PhD student, you can incorporate writing into your daily routine in a number of ways:<br />
<br />
<ol><li>When reading a paper, write a short summary once you've finished.</li>
<li>Keep a diary of your work every day. This can include notes, reminders and other ideas that you had alongside what you were currently doing.</li>
<li>Consistently write a weekly or monthly report of your work. You can use this report in your supervisor meetings too, as a way of tracking your progress and reflecting on how much more you know now compared to before.</li>
</ol><div>One excellent side effect of performing these tasks is that <b>nothing is wasted</b>. Summaries of papers that you write will feed into your literature review chapter. You will have also gotten better at writing abstracts to your own papers by doing this. Your work diary will ensure that your ideas are not forgotten, and most importantly, will always remind you that you are making progress, even when you're having a terrible week. Your weekly reports, assuming that they are detailed, can be reused to show your methodology in your thesis.</div><div><br />
</div><div>Another way of improving your writing and also helping others is by offering to proofread your colleagues' papers and chapters. By doing so, you get to read a writing style that is different to your own. You can then ask some questions as you read. What's good about it? What's bad about it? How would you change it to make it better? What good points can you take away in order to improve your own writing?</div><div><br />
</div><div>As I consumed an ever greater number of scientific papers, I noticed a definite trend in the papers that I liked. That trend was a simple, yet engaging writing style with <b>no jargon. </b>Only use big words and flowery language if you absolutely have to use big words and flowery language. If there's a simple way of saying something, say it in a simple way. Often, the ideas in papers are hard enough to get your head around, so it's useful if the reader doesn't have to fight with the communication medium as well!</div><div><br />
</div><div>One final point about writing, and this may be a very unscientific thing I'm about to say. You have to learn to <b>let go</b> of what you've written. That way, if people criticise your writing you won't take it personally. If your supervisor tells you that you need to completely restructure one of your thesis chapters, you don't want to get upset about having to scratch out large portions and start them again. Your writing is merely a way of communicating with others. The words are not you, and much like you frequently kick around and change your ideas, you need to get comfortable with kicking around and changing your writing to make it better.</div><div><br />
</div><div>Keep writing, keep writing and keep writing. That way you'll always be better than the previous week, and constant weekly improvement will give you a vastly better thesis and larger collection of published papers at the end of the tunnel.</div><div><br />
</div><div>I'll leave you with some sage advice from Simon Peyton-Jones: <a href="http://research.microsoft.com/en-us/um/people/simonpj/papers/giving-a-talk/giving-a-talk.htm">how to write a good research paper and give a good research talk</a>.</div>James Stanierhttp://www.blogger.com/profile/05102706474455020954noreply@blogger.com1tag:blogger.com,1999:blog-3709761046040818119.post-5776422157243515812011-07-04T07:54:00.000-07:002011-07-04T07:55:48.333-07:00Advice to PhD students #3: deadlines and time managementMy third piece of advice: <b>set your own deadlines, make sure you meet them, and manage your time.</b><br />
<br />
When you embark upon a PhD, especially if you're doing one earlier rather than later in life, you may feel that the process is extremely open-ended compared to what you've previously encountered. In fact, depending on how demanding your supervisor is, you may even feel like you're not being told what to do at all. During your undergraduate and masters degrees, and in the workplace, you would have mostly been used to other people telling you when something has to be completed by, whether that involves handing in a piece of coursework or getting something shipped to a customer. The PhD process is very different.<br />
<br />
At the most abstract level, the PhD process looks something like this if you're in the UK:<br />
<br />
<ol><li>Start PhD.</li>
<li>Some amount of time passes between 2-<i>n </i>years.</li>
<li>Hand in thesis.</li>
</ol>Some informal discussion with people who have succeeded say that the process goes something like this instead (in an ideal world):<br />
<br />
<ol><li>Start PhD.</li>
<li>Spend time reviewing the literature in your area of interest to see what has been done, and more importantly, what hasn't been done. Produce literature review chapter. (1 year)</li>
<li>Decide on what your project will be, start planning out ideas, get them verified by supervisor, start initial experiments and data collection. (1 year)</li>
<li>Finish experimenting, coding or theorising and start getting some results together. (1 year)</li>
<li>Write up the thesis. (1 year)</li>
<li>Hand in thesis.</li>
</ol><div>My experience was a little different to this. Firstly, my literature review period took a lot less than 1 year, but this was mostly because I joined a project where there was a very clear portion of work to be done, so I didn't spend ages finding a niche in the market. Also, I wrote everything up as I went along (more on that in a later post) so that final "write up thesis" stage was more like "copy and paste a whole load of things together".</div><div><br />
</div><div>Still, deadlines 1 year apart are not that easy to get motivated about. You need to set yourself deadlines that are short-term enough so that you get fired up to meet them, but also have just <i>enough</i> work so that 1) you don't get bored and 2) you don't burn out quickly. It's a marathon rather than a sprint, after all. I found it best to work in weekly deadlines that ended the day before I met with my supervisor for our progress meeting. This way I was motivated to have something good to show which I could receive some praise for, and then the feedback from the meeting could drive next week's deadlines.</div><div><br />
</div><div>For example, in your first year when you start out, you might want to set yourself 3-5 papers to read a week, and your goal is to write a 500 word summary of each one, of which you talk through with your supervisor. You could continue this until you start to see where there is a gap in the market in which your own research could be done. This deadline is also nice because it's easy to chunk that work into daily portions as well, such as one paper and 500 word segment a day. If you're in your second year and deep into programming, then it might be worthwhile to dedicate four days a week to coding and debugging, and one day a week for writing up what you've been doing.</div><div><br />
</div><div>The most important thing is making your deadlines matter. I touched upon one way of doing this in the previous post, and that is using your supervisor. At every meeting you have, give your supervisor the list of deadlines that you want to have completed by next time, and ask them to hold you to these. Then, you're letting them down as well as yourself if you don't get them done. I used to print all of mine out once a week and stick them on the wall above my desk. This way everyone else in the lab could see them as well as myself, and it was also very satisfying scribbling things out in red pen when I'd got one of them done. Another great way of making deadlines matter is finding out when the relevant conferences and workshops are in your field, and then planning your work around these, trying your best to get something done and written up for submission. Even if papers get rejected, you'll get great feedback from the research leaders in your area of which you can then incorporate into your own work. I'll talk about this more next time.</div><div><br />
</div><div>Research is irritating in that you often have a lot of different things going on at once: programming to do, bugs to fix, papers to write, papers to read and so on. I experimented with lots of different ways of keeping a todo list, and the one I found the most successful was the one that Randy Pausch talked about, which is from Steven R Covey's <a href="https://www.stephencovey.com/7habits/7habits.php">7 Habits of Highly Effective People</a>.</div><div><br />
</div><div class="separator" style="clear: both; text-align: center;"><iframe allowfullscreen='allowfullscreen' webkitallowfullscreen='webkitallowfullscreen' mozallowfullscreen='mozallowfullscreen' width='320' height='266' src='https://www.youtube.com/embed/oTugjssqOT0?feature=player_embedded' frameborder='0'></iframe></div><div><br />
</div><div>The entire video is long and touches on many other things, but it's well worth watching. If you don't have time to watch it (but you should watch it), then you organise the things you have on your todo list into four sections. This is what Randy talks about at 20:54.</div><div><ol><li>Important things due soon.</li>
<li>Important things not due soon.</li>
<li>Unimportant things due soon.</li>
<li>Unimportant things not due soon.</li>
</ol><div>At a given time, you keep working on 1) the important things due soon. Now, the key point is this: once you've finished with 1), where do you go? A lot of people say that the next thing to work on is the unimportant things due soon. However, that's wrong. Once you've done all of your important things due soon, you do the <b>important things not due soon</b>. Hopefully, all of the unimportant things just get ignored, and just drop off into the bottomless pit where they don't bother you ever again. You buy back time by spending less time doing unimportant things, and more time doing the important things sooner. This works for me really well.</div></div><div><br />
</div><div>What are you going to achieve this year in your PhD? What do you need to do each quarter to get this done? What does this quarter need done each month? Each week? Each day? Break up your tasks, get them written down in order of <b>importance</b>, tell other people about them, make yourself accountable and then get stuff done. Have fun crossing off the things that aren't important as time goes on. It's a great feeling. </div><div><br />
</div><div>How are you managing your time right now? </div>James Stanierhttp://www.blogger.com/profile/05102706474455020954noreply@blogger.com0tag:blogger.com,1999:blog-3709761046040818119.post-4584098830221638772011-07-01T11:39:00.000-07:002011-07-01T11:39:50.023-07:00Advice to PhD students #2: it's your problem<div>Having a good working relationship with your supervisor is essential to getting your PhD done. I'm hoping that any current students would have chosen their supervisor due to a number of reasons such as liking their work and respecting them as a person. However, getting the most out of this relationship can be difficult if you don't know how to manage them. Yes, manage <i>them</i>.</div><div><br />
</div>Here's my second piece of advice: <b>it's your problem.</b><br />
<div><b><br />
</b></div><div>Out of this list, which do you think are your responsibility, and which do you think are your supervisor's responsibility?</div><div><ol><li>Making sure you are putting in full time or part time hours (depending on your arrangement) for your work.</li>
<li>Arranging weekly meetings to check-in and discuss what you have been doing.</li>
<li>Making sure you have the equipment and materials that you need to do your work.</li>
<li>Keeping up to date on the important research in your area.</li>
<li>Writing posters, workshop papers, conference papers and journal papers.</li>
<li>Going to conferences and networking with other academics.</li>
<li>Setting deadlines and getting things done.</li>
</ol><div>The unfortunate truth is that all of these things are your responsibility. Now, I'm not saying that, for example, if you need a specific piece of equipment in order to do your research that you should go out and buy it yourself, but it is your responsibility to initiate the acquisition of this through your supervisor, or IT support, or likewise. But, the key message is that ultimately, you are responsible for your own fate.</div></div><div><br />
</div><div>Ideally, a PhD student should not only be pushing themselves in terms of the knowledge they are gaining and the results they are finding, but they should also be pushing their supervisor's understanding also. When this relationship works well, it has has mutual benefits for both parties: publications, knowledge and growth. When it doesn't work well, your supervisor still has a job and plenty of other research projects to focus on, but you'll end up not getting your PhD.</div><div><br />
</div><div>I remember when I was an undergraduate it took me a long time to realise that the first priority of (some of) my lecturers wasn't actually teaching. In fact, they had a wide variety of different priorities which I now understand. In no particular order:</div><div><ul><li>Doing their own research</li>
<li>Teaching and grading</li>
<li>Sitting in on school and departmental meetings</li>
<li>Supervising 3rd year dissertations</li>
<li>Supervising Master's dissertations</li>
<li>Supervising PhD students</li>
<li>Answering email (never underestimate the amount of email some people get)</li>
<li>Writing grant proposals and securing funding</li>
</ul><div>Supervising PhD students ranks more highly than some of the other activities, but as a PhD student, you have to realise that you're not at the top of the stack. This isn't necessarily because you're not worth anything, but it's because you're one of a number of different things that your supervisor is dealing with. However, there are techniques for remaining near the top of the stack.</div></div><div><br />
</div><div>The trick is to be visible, but not annoying. Some of the things that worked for me are as follows:</div><div><ol><li><b>Being there in person. </b>I know that a PhD does technically allow you to work wherever and whenever you want, however, from experience, turning up every day to the lab does wonders for your (lack of) social life, and it lets you be around the few people who actually understand what you're going through. The other major plus point is that quick 5 minute questions that may get ignored by email can be answered by your supervisor instantly when you knock on their door.</li>
<li><b>Regular meetings.</b> At the beginning of your research, speak with your supervisor about how you like to work. Discuss whether it would be best if you should have a short weekly meeting or a long monthly meeting, and whether you'd like to do this orally or by preparing a written report every time of what you've been up to. Different people work in different ways, but it's important to keep touching base, otherwise it's surprising how quickly you can slip away from their mind.</li>
<li><b>Be honest about your intentions. </b>Talk to your supervisor about why you are doing a PhD. Is it because you want to become an academic, or do you want to do clever work in industry? Depending on your goal, your output could be different, and you should make your supervisor press you for this output. Hard. If you want to become an academic, they should regularly push you to produce posters and workshop papers (earlier on) and conference papers and journal papers (later on). However, it's up to you to find the venues and journals. If you're aiming more towards industry, then you might focus more on producing software tools that could be made open source, making a contribution to the programming community as well as completing your thesis. These may be extensions to existing compilers or simulators, or a brand new tool you can put out there. They should push you to get these done too.</li>
<li><b>Share your deadlines. </b>You should be setting yourself regular deadlines (more on that soon), but you should give these to your supervisor and make sure they know about them. That way there is someone other than yourself accountable for getting work done. And when you're potentially letting someone else down, the motivation to get things done increases.</li>
</ol><div>The relationship with a supervisor is strange in the sense that they are more like a mirror than a traditional manager. The more you keep your supervisor informed, and the more targets you set yourself, the more supervision that you'll get. You're training yourself to do research. Imagine that your research is a huge boulder than you can't see over. Their job isn't to pull you and your research along. After all, it's research: they don't know the answer. Your supervisor should be standing beside you as you push it, making sure it's going in the right direction.</div></div>James Stanierhttp://www.blogger.com/profile/05102706474455020954noreply@blogger.com0tag:blogger.com,1999:blog-3709761046040818119.post-21051862786030997102011-06-30T12:16:00.000-07:002011-06-30T12:16:32.093-07:00Advice to PhD students #1: know your customerOne week ago, on Thursday 23rd of June, I submitted my PhD thesis. By now, any current PhD students would have thoroughly scoured the Internet in search of comforting advice about the long and hard road to a doctoral degree, so my own advice may not be entirely original. However, I felt that there were a number of things that I had learned. I'll roll them out over a series of posts. This advice is primarily aimed at computer science doctoral students, but it could possibly be applied to other fields as well.<br />
<br />
My first piece of advice is this: <b>know your customer</b>.<br />
<br />
So, as a doctoral student, who is your customer? Fundamentally, your customer is your supervisor and thesis committee for the duration of your research, and your internal and external examiners afterwards. By the same logic, your customers are also the reviewers of any conferences and journals that you submit your work to. Institutions vary, but there is an important point that comes from this observation: your achievements and progress will be judged by words on paper, not by what is on your computer screen.<br />
<br />
This can come as a surprise, especially to doctoral students who have spent a lot of time in industry beforehand. As a doctoral student, you are not creating a software product. There is no software deliverable. Instead, the software you write is merely a tool for performing experiments and getting results. This way of thinking is not always apparent, and I have seen many doctoral students spend a lot of their time trying to make an implementation perfect, trying to make it run more and more efficiently, and continuing to add new tweaks and features that would certainly impress an end user. But there is no end user.<br />
<br />
Most, if not all, computer science doctoral students will have to write code while working on their thesis. However, how much code you write can be controlled if you have your priorities straight before starting. The following points should be considered:<br />
<ol><li>How will this piece of software help my research? Is creating this software essential for me to obtain the results I want?</li>
<li>How much of this code has been written before? Are there existing libraries or software tools that do a lot of the grunt work for me? </li>
<li>Am I tied to any particular language? If not, which language will get the job done most quickly? </li>
<li>What is the bare-minimum interface to this software? Can I get away with command-line input and output?</li>
<li>How long am I willing to spend attempting this implementation before I reconsider other avenues?</li>
</ol><div>During my own thesis, I had to write a lot of code. Most of it was in C++. I don't really enjoy writing in C++ all that much. Like everyone else, I get tired as the day goes on, and the sloppier my coding gets, the more segmentation faults I have to deal with. With the complexity of the algorithms that I was implementing, debugging often became a nightmare. Often I would wake up in the morning in a stink knowing that I had an entire day of complicated debugging ahead. </div><div><br />
</div><div>Now, why was I writing in C++? Originally, I was writing in C++ because I thought that I was going to eventually incorporate my code as optimisation passes inside the <a href="http://llvm.org/">LLVM compiler</a>, and this is also written in C++. However, by the end of my thesis, I was generating LLVM bitcode (a text representation) as output, and then using this in order to generate code. The short story is that I could have used any programming language at all in order to do this. I didn't need C++ at all. The slightly longer story is that my stubbornness prevented me from rectifying this much earlier on, and then starting again using Java or Python, or something similarly more programmer-friendly than C++. I wasted a lot of time debugging memory errors that I could have avoided completely if I had used a different language. My stubbornness also made me create my own graph libraries from scratch, rather than using the excellent graph libraries that come with <a href="http://www.boost.org/">Boost</a>, or similar. </div><div><br />
</div><div>From my customer's perspective, they don't care how I created my software, or how long it took. They probably won't ever use the software at all. Their interest is in the results that came from my work, and these are written on paper in my thesis. So don't be like me. Save yourself as much time as possible when it comes to implementation. Write good quality software that does as little as is required to demonstrate your ideas and the subsequent results, and make sure it can be expanded upon in the future if needs be. But, if you're sitting there wondering what the best colour is for your GUI, or whether the command line output could be spaced more nicely, then you're not actually doing research - which is what you're meant to be doing. </div>James Stanierhttp://www.blogger.com/profile/05102706474455020954noreply@blogger.com3